Mind Your Research P’s and Q’s
The Upshot: Metrics Tell Your Story
Cynthia Castro Sweet and Pragmatic Scientific Rigor
How do you know if your product works? You measure! Measurement is critical for any sort of design, and even more so for behavior change design. The process of measurement is front-loaded, which is why I’m putting it at the front of this book. It requires detailed, thoughtful planning early in the product development process. It may feel like it’s taking a lot of time—perhaps too much time. But if you do the planning well, then you will avoid all sorts of missteps down the road. You’ll make better design decisions early in the process, saving expensive and time-consuming rework later. And you’ll save time when you’re actually collecting outcomes data; it’s a matter of executing to your plan. This chapter focuses on the preparatory work that goes into measuring and monitoring the performance of a digital product.
Behavior change designers must think not just about the typical metrics associated with digital products (like registrations and revenue), but also the long-term outcomes associated with getting people to do something differently over time. A corollary to how long it takes for behavior change to happen is that it also takes a long time—weeks, months, or even years—to measure the most meaningful outcomes and get them into shape to share. But never fear, there are plenty of indicators available earlier in the process that can help you gauge a product’s success.
Measurement isn’t just for proving the effectiveness of your product. It can also help you figure out how to improve it over time. By monitoring the right data, you can identify opportunities to build new features or content that helps users achieve their behavior change goals. You can also see if there are any clunkers in your product—features that are unloved, unused, or unneeded.
Writing Your Outcomes Story
The foundation of a strong outcomes story is to start at the end.1 Very early in the process, sit down and write the story you’d like to be able to tell once your product is on the market and thousands of people are using it. This could literally be a story—a helpful approach can be to write an imaginary press release or magazine article about your successful product and highlight what it has accomplished. If you’re more visual, create a storyboard or journey map that illustrates an ideal user experience with your product over time, including the indicators that the product is working. Invite others on your team, or your clients, to weigh in on the story so that it captures all of the most important success metrics.
At some point, the story can be formalized into an outcomes plan. One way to represent this plan is as an outcomes logic map, a document that shows the types of outcomes your product might produce over time. The outcomes logic map is a tool used in program evaluation research. One reason why it can be so powerful for digital products is that it considers how a product gets implemented and used, not just whether it’s effective in isolation. Data like click metrics, bounce rates, and drop-offs don’t just show usage; they can also help designers understand how their product is being used and make adjustments that improve outcomes.
Figure 2.1 shows a generic outcomes logic map without the information filled in. The word “logic” refers to the fact that each measurement should logically connect to the others in the sequence. There’s a concept in psychology called the mechanism of action. That refers to anything that facilitates an outcome happening. For example, if the outcome is lower body weight, the mechanisms of action might include physical exercise and changes to diet. You’ll want to identify mechanisms of action for the outcomes in your story. Measuring them will give you a way to test whether your product is doing what you expect it to do, and make adjustments if not. You’ll include mechanisms of action on your outcomes logic map as links between people using your product and the results they eventually achieve.
DIAGRAM BY AIDAN HUDSON-LAPORE.
FIGURE 2.1 At a high level, an outcomes logic map describes the data that should be available over the lifespan of a product to understand whether it will achieve its intended results.
This map gives a visual indicator of what types of things to be measuring at different periods of time. If your measurements at these points suggest that the product isn’t meeting your goals, it’s an opportunity to dig deeper and figure out why not. Plotting outcomes over a timeframe also helps set expectations about when to determine whether or not a product works. It doesn’t make sense to say a language learning program is unsuccessful if its users aren’t fluent in Greek after three weeks of use, but that same data from people who are three-year veterans might be more meaningful.
Your first attempt at an outcomes logic map will not be perfect. That’s okay. The critical thing is to go through the exercise so that your design process is organized and focused. You can and will update your outcomes logic map as you learn about your users and their needs, both during your design process and after your product is in market. But without a basic map in place at the outset, it will be difficult to know what questions to ask and what key performance indicators to build toward.
Here’s how to create an outcomes logic map.
Define the Long-Term Success Metrics
Look at the ending you’ve written in your product success story. What are the most critical metrics your product would achieve if it were successful at changing people’s behavior in a sustained way? Think about outcomes that might take months or years to accomplish, ones that would make someone look at your product and say “wow.” Write them down.
In behavior change, long-term success metrics generally fall into one of two buckets: domain-related or financial (i.e., return on investment). The domain-related outcomes vary widely, depending on what exactly the product is intended to do. If you were working on a health-related behavior change product, your long-term success metrics might include biometric data or anything that could be measured through medical tests, like someone’s weight, blood pressure, or the level of nicotine in someone’s blood. There are some health conditions, especially in mental health, that don’t have a physical test, so the outcomes there might be something like changes in people’s scores on a depression inventory. Health outcomes might also include performance indicators, like how fast someone runs a 5k or whether they’re able to complete a memory test successfully after brain training.
What if your intervention isn’t about health? If your intervention focuses on financial behaviors, long-term outcomes might include balances in someone’s savings account or the percentage of employees who have opened a 401(k) retirement account. For education, long-term outcomes might be achievement levels on a standardized test or the percentage of learners who could successfully complete an applied task. For sustainability, it could be the number of meatless meals eaten per week or the pounds of trash diverted into recycling or composting. You get the picture.
The financial outcomes often include money saved by using the medical system less or better; for example, a successful intervention might have people using the emergency room less often but the pharmacy more, because they’re taking their medications as prescribed. Sometimes the financial outcomes include increased revenue because more customers are buying the product as a result of its success. In digital projects, a common success metric is having customers choose to use the digital channel to complete tasks