group used our program. Because both groups were spending roughly the same amount of time each week thinking about and logging their sleep, it was easier to say that improvements were because of our program and not because of the heightened attention to their sleep habits.
Finally, trial just means that the study takes place over a contained period of time. Usually, the researchers who run an RCT will recruit people to participate for a defined length of time, chosen because it’s long enough for the outcomes of interest to be detected. At the start of the study, the researchers will take baseline measurements from all participants, regardless of whether they’re in the experimental or control condition. Over the course of the study, additional data might come from the product (or the control equivalent), as well as from additional surveys or observations from the researchers. Finally, at the conclusion, the researchers redo the measurements from the baseline in order to see what changes have taken place.
TIP WORK WITH THE PROS
Designing, running, and analyzing a real RCT is a specialized skill that you don’t find in most product organizations. If you want this kind of research, consider hiring a contract research organization (CRO) or partnering with researchers from a university.
RCTs are the closest thing in the behavior change world to prove that something works. They can be expensive and time-consuming, but when they’re successful, they can set a product apart from its competition.
Real-World Comparisons
Real-world comparisons use an existing comparison group instead of a randomized comparison of people using a product to people using something else. Unlike an RCT, you don’t control what programs, if any, the comparison group uses. You do try to establish similarities between the comparison and test groups; for example, you might compare people using your new walking program to people who have a Fitbit. Often, a real-world comparison is much more feasible for companies to do than an RCT. They provide strong evidence for a product as long as the research team takes steps to rule out alternative explanations for results.
Here’s an example from my own work. The members of a large health plan had access to digital health-coaching programs to target a handful of chronic conditions like high blood pressure and diabetes. Some of the members used the programs; others did not. The health plan wanted to figure out if the programs made any difference. So they were able to identify a couple thousand program users in their database and separate them out. Then they identified an equal number of nonusers in their database who were pretty similar to the users at the outset; a similar mix of men and women, average age, and level of disease symptoms. Finally, they compared the data for the two groups at different time points to see if their outcomes progressed the same way or if one group looked different from the other. They found that people who used the programs ended up having less severe disease symptoms (and fewer healthcare expenses) in the years after the baseline data was collected. Because the comparison group was selected to be as similar as possible to the user group, the health plan had confidence that using the programs made a difference in people’s outcomes.
TIP USE A THIRD PARTY
In the example of the health plan study, my company made use of third-party data supplied by our client: their claims database. Third parties can be an excellent source of outcomes data, depending on what your product does. If you don’t have access to data through a client or partner, many companies sell access to third-party databases. This data can be especially useful for filling in some of the gaps about people’s behavior: Pharmacy data can show who is filling prescriptions, consumer data can show who is buying supplies, and financial data can show who is saving money. Just be careful to honor your users’ trust in using third-party data; it can be easy to cross the line from inquisitive to creepy. If you use an IRB for your research (more on them in “Mind Your Research P’s and Q’s”), they’ll help you define that line.
You don’t need to have such robust data to do a real-world comparison, although obviously richer data means a better story. You can still make a case for your product using publicly available information. How do your users look different from nonusers who are pretty similar to them in terms of the behavior being changed?
Case Studies
Case studies are specific stories about the use of a product. They can follow an individual user, or a group of users, as they interact with a product. Case studies don’t necessarily prove whether or not a product works, but they can provide an engaging glimpse into what the product might provide to users. The data in a case study often takes a before-and-after format looking at the same group of people over time that shows how the variables of interest changed after a product was introduced.
Although their scientific value is limited, case studies have some advantages. They offer teams an opportunity to highlight their products in a specific, contextualized way that may help potential customers envision themselves as users. They can also demonstrate aspects of the product, like how it gets implemented and rolled out, that might not be examined in an RCT or other purely outcomes-focused study. And, for many of the same reasons that normative feedback and social comparisons are compelling for users, case studies can be extremely effective in building your business. Clients want the shiny new toy they see in a well-done case study.
Surveys
As a user of technology, you’ve probably received invitations to complete surveys related to a product you’re using. These surveys are a way for companies to learn something about their users that isn’t detectable within the product itself. They can be a helpful tool because they’re inexpensive to develop and distribute, so you can potentially reach a large number of users quickly and cheaply. Surveys are also an opportunity to ask people about their behaviors outside of your product that may influence their results. Do resist the temptation to ask lots of questions that you don’t have a clear reason for wanting the answer to; overloading the survey makes it more likely people will not complete it, and you’ll be stuck with a bloated data set that’s not very useful.
The disadvantages of surveys include that response rates tend to be low (less than 20% on average), and you’re relying on respondents to report their data honestly and accurately. Even people with good intentions may misremember details about their behaviors,3 while others will be motivated to make themselves look good with their answers. Because of the potential for accuracy and honesty issues, one of the best uses of surveys is to assess user satisfaction. People usually have good insight into how much they enjoy a product, and they’re often willing to share.
Surveys are often incorporated into other research approaches such as RCTs. However, they can be used as a stand-alone research method. They’re a flexible tool that can be useful both for uncovering product efficacy and learning how to improve a product, which is covered in the next section.
Evaluating for Iteration
Measurement is not just for outcomes. It can also help you determine how to evolve your product over its lifetime. If your users aren’t achieving the results you’d hoped for, or they’re not doing the behaviors you expected them to do, it’s an opportunity to dig deeper and see how your product might be better able to meet their needs.
When your product is off track from the outcomes logic map you developed, it’s an opportunity for further research. You can run a survey to see what users think about their experience, or reach out to a handful of users for individual conversations. A focused review of your product from either internal team members or an external subject matter expert could also help identify areas that need improvement in order to achieve outcomes. To go back to the blood pressure example I’ve been using, if you tell people to take medication but don’t give them any information about how to get the medication, they may not be able to follow through. Identify the roadblocks that aren’t being overcome with your product and see what you can do about it.
It’s often a good idea to deliberately build test and learn cycles