data. Asking a yes/no question eliminates the qualitative aspect of the study to a point where you’ll have nothing actionable to do when you see the results. If you ask, “Did you update your website this morning?” and 78% of the answers are yes, then what are you going to do next? What did you learn about user needs? Which pain points did you uncover? Here are some more bad examples:• Did you buy milk today?• Was your bus late this morning?• Do you like your boss?
5 Quantitative questions: Similar to yes/no questions, quantitative ones are also not going to be very helpful. A number, an average, or a percentage tells you nothing about unmet needs, missing features, painful problems, or joyful delights. It’s just a number. Here are some examples:• How many emails did you receive in the past hour?• What time did you wake up this morning?• How many items did you purchase the last time you went grocery shopping?
As mentioned earlier, phrasing an experience sampling question is a critical factor in the success of your research. Before you launch your experience sampling study, refer to the following imaginary scenario and then read sample questions and explanations about why these questions are right or wrong in Table 1.1.
Scenario: Imagine that Stop & Shop, a grocery shopping retailer in the northeastern U.S., came to you with this challenge: they want to identify ways to improve their customers’ in-store grocery shopping experience with technology. In other words, they want to uncover user needs, or answer the infamous question “What do people need?”
To sum it up, after you phrase your question, make sure that it:
• Asks about repeated behavior.
• Does not ask about opinions.
• Does not ask to “average” a behavior.
• Is not too general but very specific.
• Is not a yes/no question.
• Is not a quantitative question in which the answer is a number.
TABLE 1.1 SAMPLE EXPERIENCE SAMPLING QUESTIONS AND EXPLANATIONS
| Question | Good? | Why? | Better Question | |
| 1 | How did you create your shopping list the last time you went to the grocery store? | Yes | Asks about repeated behavior and is open-ended enough for getting valuable responses. | - |
| 2 | What is the reason you use your mobile phone while grocery shopping? | No | Phrased as if it is asked once, not repeatedly. Also, it does not meet a most likely situation where a phone is used multiple times for different purposes during grocery shopping. | The last time you went grocery shopping, what was the reason you used your phone to help you shop? |
| 3 | How often did you check the time when you recently went grocery shopping? | No | Asks for a number. Numbers will not help in identifying needs. Also, people would be making a guess because they are not aware of the number of times they use their phone. It might seem to you that asking this type of question is helpful for validating a need (e.g., if it happened zero times, it’s not a real need; if it happened many times, it is a need), but you will need to make a big leap from a number to a need. Numbers will never tell you why something is happening. | |
| 4 | Would you prefer to do your grocery shopping online to save time? | No | Asks for an opinion rather than a behavior. Also, this is a Yes/No question, which is useless for identifying needs. | What was the reason you recently grocery shopped online? |
| 5 | What did you do when you were last in the grocery store? | No | Too vague and general. People do many things while grocery shopping. Pick one thing and ask about it. | What was the biggest challenge for you the last time you went to the grocery store? |
| 6 | What part of the shopping experience is most difficult for you? | No | While asking about repeated behavior, this question is asking respondents to average their recent store visits. | What was most difficult for you the last time you went to the grocery store? |
| 7 | How do you decide what to buy? | No | Too general, asks respondents to average multiple experiences. | What was the most important consideration for deciding what to buy the last time you went to the grocery store? |
| 8 | What did you buy today? | No | Not a question for experience sampling. Can be answered by looking at store purchasing logs and cashier data. | |
| 9 | How often do you come to Stop & Shop? | No | Asks for a number. Numbers will not help in identifying needs. | |
| 10 | What would make finding items easier? | No | Asks for an opinion and predictions of the future, which are not helpful in identifying human needs. |
STEP 2: Find research participants.
Begin looking for research participants immediately. Recruiting participants is the greatest bottleneck of user research. Start as soon as you can.
Experience sampling generates huge amounts of data that affect your choice for the number of participants you can include in the study. The number of participants should be a trade-off between having enough participants who contribute enough answers to the question you ask over and over again and having a number that is too much to handle. For example, 5 participants is a very small number that will not get you enough data and verity. If these participants give you 5 answers each day for 5 days, you’ll have 125 answers. That’s not enough. On the other hand, 1,000 participants are probably too many for you to handle. Imagine if each one of them contributes 5 answers each day for 5 days. That’s 25,000 answers that need to be read, classified, and analyzed. Can you handle that?
Depending on how many answers you want, make sure that the number of participants is relatively low and digestible. Almost any number between 25 and 200 participants is something that probably makes sense.
To find the right participants for your experience sampling study that both qualify to participate in your research and are willing and available to do so, craft a screening questionnaire that will screen people into or out of your study:
1 List your assumptions