3.3.1 What Factors Best Predict Desirable and Undesirable Outcomes?
Later in this chapter, we'll see that correlational studies rank relatively low on a research hierarchy for questions about effectiveness. We'll see that although they can have value in informing practice decisions about the selection of an intervention with the best chances of effectiveness, other designs rank higher. Experimental outcome studies, for example, rank much higher. But for questions about circumstances or attributes that best predict prognosis or risk, correlational studies are the most useful. With these studies, multivariate statistical procedures (statistics that account for multiple factors at once) can be employed to identify factors that best predict things we'd like to avoid or see happen.
Returning to the foster-care example discussed earlier, suppose you are a child welfare administrator or caseworker and want to minimize the odds of unsuccessful foster-care placements. One type of correlational study that you might find to be particularly useful would employ the case-control design. A study using this design to identify the factors that best predict whether foster-care placements will be successful or unsuccessful might proceed as follows:
1 It would define what case record information distinguishes successful from unsuccessful placements.
2 It would obtain a large and representative sample of foster-care placements depicted in case records.
3 It would then divide those cases into two groups: those in which the foster-care placement was successful and those in which it was unsuccessful.
4 It would enter all of the placement characteristics into a multivariate statistical analysis, seeking to identify which characteristics differed the most between the successful and unsuccessful placements (when all other factors are controlled) and thus best predicted success or failure.
If your previous research courses extolled the wonders of experiments, at this point you might exclaim, “Wait a minute! Why rank correlational studies above experiments here?” It's a good question, and we'll answer it with three others: Can you imagine the staff members of any child welfare agency permitting children to be assigned randomly to different types of foster placements? What would they say about the ethics and pragmatics of such an idea? What might they think of someone for even asking?
Correlational studies are not the only ones that can be useful in identifying factors that predict desirable or undesirable outcomes. Qualitative studies can be useful, too. For example, let's return to the question of why so many homeless people refuse to use shelter services. As is mentioned in Chapter 1, studies that employ in-depth, open-ended interviews of homeless people – or in which researchers themselves live on the streets among the homeless and experience what it's like to sleep in a shelter – can provide valuable insights as to what practitioners can do in designing a shelter program that might alleviate the resistance homeless people might have to utilizing the shelter.
3.3.2 What Can I Learn about Clients, Service Delivery, and Targets of Intervention from the Experiences of Others?
In Chapters 1 and 2, we can see that some studies suggest that one of the most important factors influencing service effectiveness is the quality of the practitioner-client relationship, and that factor might have more influence on treatment outcome than the choices practitioners make about what particular interventions to employ. We also know that one of the most important aspects of a practitioner's relationship skills is empathy. It seems reasonable to suppose that the better the practitioner's understanding of what it's like to have had the client's experiences – what it's like to have walked in the client's shoes, so to speak – the more empathy the practitioner is likely to convey in relating to the client. In other instances you may want to learn about the experiences of others – not just clients – to inform your practice decisions. For example, gaining insight into practitioners' experiences using a new caregiver support intervention or family members' experiences caring for an elderly client can help inform your practice decisions about implementing a caregiver support intervention in your own practice.
When we seek to describe and understand people's experiences – particularly when we want to develop a deep empathic understanding of what it's like to walk in their shoes or to learn about their experiences from their point of view – qualitative studies reside at the top of the research hierarchy. Qualitative research can provide rich and detailed information that is difficult, or even impossible, to capture accurately or fully in a quantitative study. Gambrill (2006) illustrated the superiority of qualitative studies for this EIP purpose via a study by Bourgois et al., (2003), which examined the kinds of risks taken by street addicts. Bourgois immersed himself in the “shooting galleries and homeless encampments of a network of heroin addicts living in the bushes of a public park in downtown San Francisco” (p. 260). Virtually all of the addicts reported that when they are surveyed with questionnaires, they distort their risky behavior. Often, they underreport it so that it takes less time to complete the questionnaire. Also, they may deceive themselves about the risks they take because they don't want to think about the risks. Consequently, quantitative methods like surveys would rank lower on a hierarchy for this type of EIP question.
3.3.3 What Assessment Tool Should Be Used?
As is discussed in Chapter 1, common questions to ask in selecting the best assessment instrument pertain to whether the instrument is reliable, valid, sensitive to small changes, feasible to administer, and culturally sensitive. Most of the studies that assess reliability, validity, and cultural sensitivity use correlational designs. For reliability, they might administer a scale twice in a short period to a large sample of people and assess test-retest reliability in terms of whether the two sets of scale scores are highly correlated. Or they might administer the scale once and see if subscale scores on subsets of similar items correlate to each other. For validity, they might administer the scale to two groups of people known to be markedly different regarding the concept being measured and then see if the average scores of the two groups differ significantly. For sensitivity, they might use a pretest-posttest design with no control group and administer the scale before and after treatment to see if the scale can detect small improvements. Although experiments and quasi-experiments are rarely the basis for assessing a scale's validity or sensitivity, it is not unheard of for an experiment or a quasi-experiment to provide new or additional evidence about those features of a scale. That is, if a treatment group's average scores improve significantly more than the control group's, that provides evidence that the scale is measuring what the treatment intends to affect and that the scale is sensitive enough to detect improvements. W return to these issues and cover them in greater depth in Chapter 11. That entire chapter is devoted to critically appraising and selecting assessment instruments.
3.3.4 What Intervention, Program, or Policy Has the Best Effects?
As we've already noted, tightly controlled experimental designs are the gold standard when we are seeking evidence about whether a particular intervention – and not some alternative explanation – is the real cause of a particular outcome. Suppose, for example, we are employing an innovative new therapy for treating survivors of a very recent traumatic event such as a natural disaster or a crime. Our aim would be to alleviate their acute trauma symptoms or to prevent the development of posttraumatic stress disorder (PTSD).
If all we know is that their symptoms improve after our treatment, we cannot rule out plausible alternative explanations for that improvement. Maybe our treatment had little or nothing to do with it. Instead, perhaps most of the improvement can be attributed to the support they received from relatives or other service providers. Perhaps the mere passage of time helped. We can determine whether we can rule out the plausibility