You’ve been planning a family vacation for months, but you’ve been busy at work so you procrastinated a bit on the planning, and now it’s the morning of the trip and you’re trying to quickly print out directions between finishing your packing and getting your kids packed. Your coworker told you about this MapTool Web site you’ve never used before, so you decide to give it a shot, and it’s not so bad—that is, until you get stuck because you can’t find the freaking button to print out the directions, and you’re supposed to leave in an hour, but you can’t until you print these damn directions, but your kids are jumping up and down on their suitcases and asking you where everything is. Why can’t they just make this stupid crap easy to use? Isn’t it obvious what’s wrong with it? Haven’t they ever seen a real person use it before?
Circumstances matter a lot in user research, and someone who’s using an interface in real life, for real purposes, is going to behave a lot differently—and give more accurate feedback—than someone who’s just being told to accomplish some little task to be able to collect an incentive check. Time-awareness is an important concept, so we’ll bring it up again throughout this book to demonstrate how the concept relates to different aspects of the remote research process (recruiting, moderating, and so on).
Note
TAR Keeps You in the Right 1985
Remember that diagram in Back to The Future II? Doc argues that messing with time has sent the world crashing hopelessly toward an alternate reality where things are horrible—the “wrong 1985.” And that’s sort of what happens when you try to assign people a hypothetical task to do at a time when they may or may not actually want to do it: you’re meddling with their time, and it’ll create results that look like the real thing but are all wrong.
When you schedule participants in advance and then ask them to pretend to care, you’re sending your research into the wrong 1985. If you don’t want to create a time paradox—thereby ending the universe—you should do time-aware research.
Other Benefits of Remote Research
Here are some additional benefits of remote research.
Geographic diversity. Even if you do have a lab, the users you want to talk to may not be able to get to it. This is actually the most common scenario: your interface, like most, is designed to be accessed and used all around the world, and you want to talk to users from around the world to get a range of perspectives. Will Chinese players like my video game? Is my online map widget intuitive even for users outside Silicon Valley? Big companies like Nokia and Microsoft are often able to conduct huge, ambitious research projects to address these questions, coordinating research projects in different labs around the world, flying researchers around in first-class. If you don’t have the cash for an international Gorillas-in-the-Mist project, then remote research is a
no-brainer solution. If you can’t get to where your users are, test them remotely.
Ability to test almost anywhere. Remote research has comparatively minimal setup requirements and can reach anywhere that computers and the Internet can go: you can be anywhere; your participants can be anywhere. Lone-wolf consultants and start-up teams working out of cafés can have trouble finding the quiet office space they need to do in-person testing. If it’s too much bother to set up a proper lab, go remote; all you’ll need is a desk.
Some reduced costs. Beyond travel expenses, other costs associated with lab testing may be reduced or eliminated when you test remotely. With live recruiting methods, you can get around third-party recruiting costs, and because the recruiting pool is larger, you may not have to offer as much in the way of incentives as you might otherwise to attract enough participants. Because sessions are conducted through the computer, you can use relatively inexpensive software to replace costly testing accessories, such as video cameras, observation monitors, and screen recording devices. (Note, however, that the overall cost of a remote research study is often comparable to an in-person study for many reasons; see Chapter 10 for reasons why.)
Quicker setup. Closely related to the issue of money, as always, is time. Nearly all existing recruiting methods take many weeks. Recruiting agencies usually require a couple of weeks to gather recruits, and writing out precise recruiting requirements and explaining the study to them can eat up a lot of time. Getting users from your own mailing list can be faster and moderately effective, but what if you don’t have one? Or what if you’ve overfished the list from previous studies, or you don’t want to spam your customers, or you’re looking to test people who’ve never used your interface or heard of your company before? In any of these cases, recruiting your users online makes a lot of sense, since it allows you to do your recruiting as research sessions are ongoing. (We teach you how to do all this in Chapter 3.)
Context-dependent interfaces. Some interfaces just don’t make any sense to test outside their intended usage environment. If you need users to have their own photos and videos to use in a video editing tool, having them bring their laptop or media to a lab will be a tremendous hassle. Or, let’s say you’re testing a recipe Web site that guides users step-by-step through preparing a meal; it wouldn’t make much sense to take people out of their kitchen, where they’re unable to perform the task of interest. When this is the case, remote research is usually the most practical solution, unless the users also lack the necessary equipment.
When to Go Remote
If you have the gumption, you can test almost anything remotely. There are ways to get around nearly any obstacle, but the approach you take is all about what’s most practical and accurate. If it’s significantly cheaper, faster, or less of a hassle for you to just bring people into a lab, then by all means bring ’em in. Sometimes this decision can be a tough call; users in the developing world may have limited access to the Internet, for instance, so you’d have to decide whether it’s worthwhile to fly over and talk to users in person, or to find people from that demographic in your area, or to arrange for the users to be at a workable Internet kiosk to test them remotely.
For clarity’s sake, let’s talk about some clear-cut cases of things you should and shouldn’t test remotely.
Remote testing is a no-brainer for Web sites, software, or anything that runs on a desktop computer—this is the kind of stuff remote research was practically invented to test. The only hitch is that the participants need to be able to use their own computer to access whatever’s being tested. Other Web sites besides your own are a cinch: just tell your users during the session to point their Web browsers to any address you want. If you’re testing prototype software, there needs to be a secure way to digitally deliver it to them; if it’s a prototype Web site, give them temporary password-protected access. If the testing is just too confidential to give them direct access on their computer, you can host the prototype on your own computer and use remote access software like VNC or GoToMeeting to let them have control over the computer. There’s almost always a way to do it.
The stuff you test doesn’t even have to be strictly functional. Wireframes, design comps, and static images are all doable; we’ve even tested drawings on napkins (really). Just scan them in to a standard image format and put them on a Web site. Make sure the user’s browser doesn’t automatically resize it by using a plain HTML wrapper around each image. There are also plenty of software solutions (like Axure and Fireworks) that can help you convert your images to HTML.
Can you test programs that require users to enter personal information? Yes, but make sure to give your participants a way to enter “dummy” information wherever they’re required to enter sensitive or personally identifying information. (According to Rolf Molich, people act differently when using dummy information, so bear that in mind.) If you require the participant to use real personal information, be sure to obtain explicit consent right at the beginning of the testing session (an issue covered in Chapter 4); you don’t want to spend 20 minutes on the phone with a user only to have to terminate the study over privacy issues.
Most remote research tools (screen sharing, recording, chat, etc.) are suited for a computer desktop environment, so physical products are harder to test remotely. We’re just beginning to see mobile device and mobile interface research become feasible, and we’ve