Psychotherapy Research 1 11. Unresolved Questions ...

Psychotherapy Research 1 11. Unresolved Questions and Future Directions in Psychotherapy Research Bruce E. Wampold, Steven D. Hollon, and Clara E. Hill Psychotherapy is a remarkably complex practice. Much has been learned over the many decades of research, but it is clear that there are many unanswered questions. Some of the unresolved issues are critical to understanding how best to deliver psychotherapy and how best to train psychotherapists. The path to knowledge, however, is not entirely clear. To some, randomized clinical trials (RCTs) are the most useful design because they involve experimental manipulations, giving rise to causal statements. Others hew to the idea that looking at quantitative indices of outcome masks the rich material generated in the complex therapeutic interaction and focus on the process of psychotherapy. Lately, there have been attempts to integrate process and outcome. As well, as neuroscience develops and we understand the neural substrate of much of human behavior, knowledge of psychotherapy will inevitably be informed by neuroscientific investigations. In this chapter, we review these four strands of research with an eye to what they can and cannot tell us about psychotherapy and what the future might hold for research in the field. Randomized Clinical Trials Clinical interventions are conducted in an effort to make things better, either to keep something bad from happening (prevention) or to remove the bad once it has occurred (treatment). RCTs are done in an effort to determine whether a particular intervention has made things better; that is, whether it has had the intended causal effect. The first question to ask about an intervention is, “Does it have the intended effect?” That is, whether the state of affairs observed after providing an intervention can be attributed to what was done. In short, the question is whether it caused things to be better than they otherwise would have been.

Psychotherapy Research 2 Clinical trials can take any of several forms, including within-subjects designs and between-groups comparisons, but they all involve some type of intentional manipulation(s) in a manner that allows a causal inference to be drawn. Someone or some group receives an intervention that they were not already receiving, that may subsequently be taken away or staggered across problems (in the case of single subject designs) or that is not provided to some other group (in the case of randomized controlled trials) and consequences are observed. Clinical trials are said to be “controlled” because of the quality of intentionality. Simply observing what happens to people going through psychotherapy in an uncontrolled fashion might provide useful information, but it does not allow for ruling out rival explanations as to what actually caused the outcomes observed. Anecdotal case reports can provide a wealth of useful information and often form the starting point for the process of discovery, but they cannot rule out the possibility that any changes observed would not have happened in the absence of the psychotherapy (Kazdin, 1981). When the pivotal question is whether some intentional act can produce the consequences intended, that is, if the goal is to draw a causal inference regarding the effects of psychotherapy, then most powerful way to answer that question is to do an experiment (Cronbach, 1957). Not all experiments involve intentional manipulations. Causal inferences can be drawn about the effects of genes and other “experiments of nature” when it is reasonable to conclude that preexistent factors could not have accounted for the consequences observed; but, when it comes to intentional acts, the controlled experiment is the most powerful design for ruling out rival plausible alternatives to the notion that the intervention had a causal effect. Psychotherapy had been a going concern for over half a century when Eysenck claimed that there was no good evidence that it actually had a causal effect (Eysenck, 1952). In his

Psychotherapy Research 3 critique, Eysenck cited evidence indicating that patients who pursued psychotherapy fared no better than people who did not. The problem with that critique is that people were not randomized to treatment or its absence; rather, people who chose to pursue treatment (often at considerable expense) were compared to people who did not make that choice. It was likely that those who pursued no therapy were not as troubled as those who did. The debate that was triggered by Eysenck led to a spate of RCTs, the vast majority of which suggested that any reasonable psychotherapy was better than its absence (Luborsky, Singer, & Luborsky, 1975). Eysenck was clearly wrong in his critique (because he allowed individual differences to determine whether someone received treatment, thereby confounding patients and procedures), but he did have the effect of spurring the field to actually test the efficacy of treatment in a systematic fashion. Eysenck was somewhat disingenuous in his critique; it was not just that he thought that the psychotherapies of the time (largely psychodynamic or humanistic) were ineffective, but rather that he preferred another treatment (behavior therapy) that he considered to be more scientific. Carl Rogers developed a theoretical orientation that is often labeled humanistic and believed that the role of the therapist was not to guide treatment in a deterministic fashion, but rather to create the conditions in therapy that would facilitate psychological growth within each client. At the same time, Rogers was firmly committed to the notion of testing his therapy in a scientific fashion; in fact, he and his colleagues produced some of the first RCTs that showed that psychotherapy had a causal effect (Rogers & Dymond, 1954). The simplest RCT compares a treatment to its absence. Patients are selected on a systematic basis and randomly assigned to either treatment or control, usually some form of wait list or assessment only condition. The results of literally hundreds of such experiments are clear;

Psychotherapy Research 4 most reasonable approaches to treatment work better than their absence. Not all types of treatments have been tested as extensively as others, and there are major gaps in the literature with respect to more severe and complex populations (the behavioral and cognitive behavioral therapies tend to have been more extensively explored than the other major schools). Nonetheless, there is good consensus across the field that on average most treatments work better than their absence (see chapter 10 for details). Scientific validity of clinical trials Clinical trials resemble classical experiments in that the research manipulates an independent variable to determine effects on a dependent variable. This type of design rules out several threats to the validity of the conclusions and therefore has reasonable validity compared to alternative designs. Internal validity refers to the extent to which one can be sure that the effects observed can be attributed to the intervention (Campbell & Stanley, 1963). A major source of variability in outcomes is the patients, given that they often change over time whether something is done to them or not (e.g., most people recover from the flu even without treatment). A control condition in the design enables spontaneous remission to be differentiated from the effect of treatment. Different patients are likely to change in different ways over time in response to the same psychotherapy. Allowing individual differences between patients to determine what type of intervention they receive can confound the process of interpretation; that is, any subsequent differences observed could be as much a consequence of differences between patients in maturational course as a consequence of treatment. Randomization provides a means of controlling for patient differences; ideally, in a RCT, different kinds of patients are distributed

Psychotherapy Research 5 equally across the groups and subsequent differences in outcome can be attributed to the effects of the intervention. Randomized clinical trials and their logical equivalents (single subject designs that provide the necessary internal controls to rule out rival causal interpretations) are the most powerful tools we have for drawing causal inferences about the consequences of clinical interventions. Nonetheless, they are far from perfect. Randomization does not always work, and groups can be unbalanced simply as a consequence of chance, especially when the number of patients sampled is small. That is why investigators test to determine whether the treatment groups differ on pretreatment characteristics (and control statistically for any such differences observed) and stratify prior to randomization on important variables likely to predict subsequent course. Similarly, attrition can undermine the balance produced by initial randomization. That is why the field has come to prefer analyses based on all patients initially randomized to treatment (“intent-to-treat”) rather than restricting analyses to only those patients who started treatment (“modified intent-to-treat”) or finished treatment (“completer”) samples. The basic principle recommended by statisticians is “once randomized always analyze”; investigators risk undermining the internal validity of the conclusions drawn when that principle is violated. External validity refers to the extent to which the findings observed in a study generalize to the conditions of actual interest (Campbell & Stanley, 1963). College students with elevated symptom scores may be more accessible to researchers in academic settings, but questions can be raised as to whether the results from experiments conducted on such participants generalize to bona fide clinical populations. The therapists selected for a given trial may not be adequately trained to implement the intervention of interest or restrictions may be imposed by the design or the setting that prevent the intervention from being implemented in a fully representative fashion.

Psychotherapy Research 6 Generalization can vary across multiple dimensions (including patients, therapists, and settings) and much of the existing literature has been criticized for screening out the complicated patients often seen by experienced therapists in actual clinical settings (Westen, Novonty, & ThompsonBrenner, 2004). A distinction has been drawn in recent years between efficacy studies that are often done in controlled research settings that seek to maximize internal validity versus effectiveness trials that are often done in applied clinical settings in an effort to maximize external validity (Nathan, Stuart, & Dolan, 2000). All efficacy trials are by definition randomized experiments but that is only true for some effectiveness designs (Seligman, 1995). Effectiveness designs that do not involve randomization can address important questions, but they cannot rule out rival plausible alternatives to the notion that treatment had a causal effect. Internal and external validity are not necessarily incompatible, and it is possible to do randomized trials in a clinically representative fashion. However, it proves difficult and good examples of studies that are high on both dimensions are hard to find in the literature. It is easier to detect an effect than it is to explain it. That is because any attempt to explain an effect necessarily requires identifying the causal mechanisms through which the intervention works and the links between those causal mechanisms; the outcomes observed are not themselves directly manipulated (Baron & Kenny, 1986). Nonetheless, explanation is an important part of science and knowing how an intervention works can facilitate efforts to maximize its effects. Construct validity is defined as the extent to which we understand how an intervention exerts its causal effects (Cook & Campbell, 1979). For as long as people have believed that psychotherapy is efficacious they have argued as to how it produces its effects. All psychotherapies are accompanied by a theory that attempts to explain how each

Psychotherapy Research 7 works. Sometimes the theory precedes the development of the intervention and sometimes follows the serendipitous discovery that an intervention works, but in either event refinements in theory can be used refine the intervention and vice versa in a synergistic fashion. Issues inherent in clinical trials of psychotherapy Although it is now well accepted that most psychotherapies work, it remains unclear precisely how they work. There are several major schools that each puts forth a different specific explanation for how treatment works, as described elsewhere in this book (chapters 3-8). All schools of therapy have given rise to numerous specific offshoots, and much literature has been devoted to distinguishing among them and singing the praises of each. There is yet another perspective that says what makes psychotherapy work is not what separates the competing schools, but the common elements they all share (Frank & Frank, 1991). This perspective emphasizes the role of nonspecific factors believed to be present in all therapies; the provision of a compelling rationale for treatment and the mobilization of hope that things can be made better. It is not the specific strategies and techniques that separate the schools, but the nonspecific aspects that they all share that are the causal mechanisms that drive change. To the extent that this is true, the field would be better served by investigating elements that the different treatments share rather than the specific techniques that differentiate them. Addressing the issue of specificity versus common factors is somewhat more complex, since it is not always possible to implement a nonspecific control with a compelling rationale in which therapists can believe. It is easier to implement a nonspecific control in the form of pill (e.g., placebos in pharmacotherapy trials) because patients and prescribers can be blinded as to whether the pills contain an active medication. Such dismantling studies are inherently difficult to conduct in psychotherapy, since both patient and provider know what is being done, and it is

Psychotherapy Research 8 harder to disguise the absence the theoretically specified active ingredient(s). Nonetheless, many studies exist that do control for therapeutic contact and attempt to equate the conditions on such nonspecific factors as treatment credibility and expectations for change. Some would argue that such efforts are doomed to fail and that any control condition that is not itself a bona fide treatment implemented by therapists who believe in its effects cannot possibly serve as an adequate nonspecific control (Baskin, Tierney, Minami, & Wampold, 2003). Comparisons between two or more bona fide treatments that do address this concern provide the additional benefit of addressing differential efficacy. If two or more treatments each implemented by competent adherents produce differential effects and one is ipso facto superior to the other it is likely so for nonspecific reasons. We qualify the latter since it is still possible that one treatment may just be better than another at mobilizing nonspecific factors and that accounts for its advantage, not the different strategies and techniques specified by theory. As we indicated earlier, it is easier to detect an effect (even a differential effect) than it is to explain it. Evidence for differential efficacy is sufficient to prefer one treatment over another but not sufficient to prove that the superior treatment works for the reasons specified. That would require a more extensive program of research that ideally would involve systematic assessment and manipulation of both specific and nonspecific factors in an effort to test the underlying theory. There is at this time no consensus in the field as to whether psychotherapy works for reasons specified by theory or even whether differential efficacy can be said to exist with respect to most disorders. There are more high-quality studies supporting the efficacy of the behavioral and cognitive behavioral therapies than the more traditional insight-oriented or humanistic therapies but the absence of evidence is not necessarily evidence of absence (of comparable or superior efficacy). Not that many studies have directly compared treatments across the different

Psychotherapy Research 9 schools and fewer still have implemented each of the interventions in a fully representative fashion. Box score and meta-analytic summaries that include all available studies regardless of quality typically find little evidence for differential efficacy and conclude that “all have won and all must have prizes” (Luborsky et al., 1975; Smith & Glass, 1977; Wampold et al., 1997), whereas summaries that require support from a minimum number of clinically representative studies across different research groups have tended to certify a smaller number of treatments as empirically supported for any given disorder (Chambless & Ollendick, 2002; Task Force on Promotion and Dissemination of Psychological Procedures, 1995). Critics of the more inclusive quantitative approach complain that including weak studies in the larger pool dilutes the effects of strong treatments, whereas critics of the less inclusive exemplar approach complain that it ignores negative findings and the “file drawer” problem. Independent governmental agencies like the National Institute for Health and Clinical Excellence (NICE) in England try to integrate the two approaches by restricting quantitative reviews to those studies that meet minimum quality criteria and then asking representatives from different theoretical perspectives to arrive at a Delphic consensus based on reasoned judgments regarding the findings and the studies underlying them. The result of this process has been to yield an outcome somewhat closer to the exemplar approach with only certain types of therapies (mostly cognitive or behavioral) certified for specific disorders, although it has been done with the clear recognition that often excluded therapies have not so much been shown not to work as well as those that are certified so much as it simply means that the excluded interventions have not been adequately tested. For example, NICE certifies three types of treatments for the treatment of post-traumatic stress disorder: prolonged exposure, cognitive behavior therapy, and eye-movement desensitization and reprocessing (Bisson et al., 2007); whereas a recent meta-

Psychotherapy Research 10 analysis suggested that any of several therapies (including more traditional dynamic and humanistic approaches) were all comparable in effect (Benish, Imel, & Wampold, 2008). What We Need to Know In summary, there is a broad consensus that any reasonable psychotherapy works better than its absence. There is considerably less consensus as to whether psychotherapy works for specific as opposed to nonspecific reasons, and whether there is any support for the notion of differential efficacy across the different therapies. Meta-analytic studies suggest that all are effective, whereas reviews that rely on a minimum number of exemplary studies or Delphic consensus methods typically certify a smaller number of interventions. Each approach has its adherents and the issue is not one that will be resolved soon. What remains unclear from the use of clinical trials in psychotherapy is whether different kinds of patients respond best to different therapies and the processes through which each intervention works. It has long been the goal of psychotherapy research to identify the best treatment for a given patient (Paul, 1967). Efforts to accomplish this goal rest upon the detection of moderation (patient characteristics that predict differential response to different treatments) and the best way to detect moderation is to randomly assign patients who differ in their characteristics to different treatments (Fournier et al., in press). Simply providing different patients with the same treatment and noting who does better provides prognostic information that can be useful, but it does not serve as a basis for treatment selection. In effect, such prognostic designs tell you what to expect as a consequence of treatment (and who to select to make a treatment look good), not which treatment to select to maximize response for a given patient. Given the crises in the cost of health care, future research must focus on how to improve the quality of mental health care in routine practice. There are emerging potential topics in this

Psychotherapy Research 11 area. First, research is increasing being focused on transporting evidence-based treatments to routine care (e.g., Addis et al., 2004; Weisz, Jensen-Doss, & Hawley, 2006). Second, the effect of providing therapists feedback on their progress appears to improve outcomes (Lambert, Harmon, Slade, Whipple, & Hawkins, 2005; Miller, Duncan, & Hubble, 2005). Although increasing the benefits of psychotherapy in routine care is absolutely critical, determining the costs of such quality improvement programs are necessary—how can the benefits of psychotherapy be improved most efficiently? Process Research Given that researchers have consistently demonstrated that psychotherapy is effective, the burning question for future research is: What makes it effective? From their review of the literature of the contributors of improvement in psychotherapy, Asay and Lambert (1999) suggested that the therapeutic relationship accounted for 30% of the variance, extra-therapeutic factors (client and environmental characteristics) accounted for 40% of the variance, therapist techniques accounted for 15% of the variance, and expectancy (client’s hope and therapist credibility) accounted for 15% of the treatment. Although the exact percentages can be questioned given our lack of sophistication about how to measure these factors and their contribution to outcome, these figures provide us with some beginning indication about the important process contributors. In this section, then, we discuss the future directions and unresolved questions related to the contributions of clients, therapists, the therapeutic relationship, and the interaction amongst these factors to psychotherapy process. Client Characteristics Several researchers have reviewed the multitude of variables (e.g., motivation, level of psychopathology, gender, and race) that have been assessed in the literature (e.g., Clarkin and

Psychotherapy Research 12 Levy, 2004). Here, we focus on two variables that seem very promising: client readiness for change and client involvement. Given that clients are the ones who come to psychotherapy seeking helping, who disclose their problems, who have to implement the changes suggested in psychotherapy, and who have to adjust to the changes in their lives, it is not surprising that client motivation (or readiness for change) for participating in the therapy process has been found to relate to the outcome of psychotherapy. Client readiness for change has been divided into several stages (Prochaska, Norcross, & DiClemente, 1994). In the precontemplation stage, clients have no desire to change and are unaware of their problems and impact on others. In the contemplation stage, clients are beginning to become aware of their problems, and they are thinking about changing. In the preparation stage, clients have decided they want to change and are taking small steps toward the change process. In the action stage, clients are in the process of actively modifying their behaviors or environments. In the maintenance stage, clients work to consolidate their changes and deal with lapses and problems as they arise. At the end of this process, clients are no longer troubled by the initial problem and have confidence that they can cope without relapse. Readiness to change has been assessed by a multitude of paper-and-pencil measures. Strong support has been shown for the importance of identifying which stage clients are in, and evidence indicates that therapists use different strategies when clients are in different stages (see Prochaska & Norcross, 2002). A related variable that seems promising is client involvement, which can be defined as openness, engagement, cooperation, and active participation in the psychotherapy process. Client involvement probably reflects client motivation for change, being in the action stage of readiness to change, and feeling good about working with the particular therapist (see more in the section

Psychotherapy Research 13 below on the working alliance). Client involvement can be assessed by having judges code the behavior of clients by listening to tapes of therapy sessions (e.g., Hill et al., 1992) or by having clients rate themselves or by therapists rating clients (e.g., Eugster & Wampold, 1996). Client involvement has been related to the therapeutic alliance (e.g., Karver et al. 2008) and treatment outcome (e.g., Eugster & Wampold, 1996; O’Malley, Suh, & Strupp, 1983). The opposite of client involvement is client resistance (e.g., Mahalik, 1994), also an important construct in need of further study. Of particular interest is how to increase client involvement and decrease client resistance, i.e., what can therapists do to help clients become more involved and less defended or resistant? Unfortunately, no consistent definition or measure of client involvement or resistance has been used, so while intuitively appealing, these constructs deserve further exploration. Therapist Characteristics The person of the psychotherapist is a huge part of the process of therapy. Empirically, we have strong evidence of therapist effects (Beutler et al., 2004; Wampold, 2001), although we have less evidence for what aspects of therapists contribute to these results. In this section, we look at the therapist techniques and personal reactions as variables that merit attention. We contend that what the therapist does within sessions has a huge impact on the immediate and long-term outcome of therapy, but, investigating the connection between therapist actions and outcomes is extremely complex. First, therapist actions can be conceptualized in many ways—nonverbal behaviors, verbal response modes, overall treatment approach, flexibility in applying techniques to the individual client, or ability to track the client Second, the context of the therapist action is critical, because what works differs depending on the moment in therapy. These comments are not meant to imply that techniques are not important, only that we have not figured out well how to study them.

Psychotherapy Research 14 The most typical way that therapist techniques have been assessed is through coding verbal response modes (VRM; e.g., reflection of feelings, interpretation) in transcripts or videotapes of therapy sessions. Much of the early research focused on correlating the frequency or proportion of VRMs with session or treatment outcome. Mixed and inconsistent results were found in these studies, probably because of the inappropriate use of a correlational strategy. In this regard, Stiles (1988) commented that it is not the amount of VRMs (i.e., more is not necessarily better) but rather the therapist’s responsiveness to giving the client what he or she needs at the moment in therapy that should be related to outcome. Given the concerns about correlational strategies, researchers moved to analyzing the immediate effects of therapist VRMs (e.g., how does the client respond in the next speaking turn when the therapist uses an interpretation). For example, one study (Hill et al., 1988) examined all the immediate effects of therapist VRMs in 127 sessions of psychotherapy (8 therapist/client dyads in brief therapy for 12 to 20 sessions). Therapist VRMs accounted for only 1% of the variance of immediate outcome (client and therapist helpfulness ratings, client experiencing in the subsequent speaking turn, and client self-reported reactions). When added into the equation, therapist intentions and previous client experiencing accounted for more of the variance (but not a lot) than did therapist VRMs. One suggestion for the lack of effects for VRMs was that the complexity of therapy was not accounted for. In other words, it makes little sense that a reflection of feelings would have a uniform impact on therapy. Rather, the helpfulness of the reflection is probably dependent on how it is presented (quality, accompanying nonverbals), when it is presented (e.g., stage of therapy, the problem being discussed, and what interventions have preceded the reflection), and client needs at the time. Thus, although this method of coding all the VRMs within several cases and examining the immediate impact is appealing, obvious

Psychotherapy Research 15 problems arise is trying to interpret the data. A more recent strategy for studying therapist techniques is to investigate specific techniques within a case study so that context can be taken into account. For example, Hill and colleagues recently conducted two case studies on therapist immediacy (Hill et al., 2008; Kasper, Hill, & Kivlighan, 2008), which they defined as working with the therapeutic relationship in the “here and now.” Immediacy involves such therapist actions as inquiring about reactions to the therapy relationship, drawing parallels between other relationships and the therapy relationship, processing ruptures or boundary crossings, and disclosing feelings of closeness to or lack of closeness from others. The positive effects across the two cases were very similar (facilitated negotiation of the therapeutic relationship, provided a corrective relational experience), although the Kasper et al. therapist more often used challenging forms of immediacy that helped break down the client’s defenses, and the Hill et al. therapist more often used supportive forms of immediacy that helped build the client’s fragile ego. Thus, these two cases illustrate the benefits of using a qualitative case-study methodology to study therapist techniques. Other therapist techniques that merit careful attention are interpretations, challenges, paradoxical interventions, self-disclosures, and reflections of feelings. Personal reactions of therapists (e.g., countertransference) also seem to exert influence on the process and outcome of psychotherapy (see Gelso & Hayes, 2007). It appears that it is important for therapists to be aware of these personal reactions and manage them in the therapy process to avoid harming clients (see Williams, Hurley, O’Brien, & DeGregorio, 2003). Finding appropriate strategies for assessing personal reactions and management strategies is a challenge. The Therapeutic Relationship The therapeutic relationship has been characterized as one of the most robust predictors

Psychotherapy Research 16 of outcome, although estimates are that the relationship accounts for only a modest amount of outcome variance (Horvath & Bedi, 2002). Thus, when clients perceives that they feel a bond with their therapists and agree with their therapists about the tasks and goals of therapy, the outcome is better than when clients do not feel such an alliance with their therapists. We know that the therapeutic relationship is related to outcome, although we do not know exactly what the mechanism is by which it leads to change. We propose that we need to know more about how the relationship is built and how it is fixed when broken. Given Bordin’s (1979, 1994) ideas about tear and repair, repairing ruptures probably builds stronger therapeutic relationships (see also Hill & Knox, 2009). Safran and colleagues (Safran et al., 2002) have done ground-breaking research on the repair of ruptures in therapy. Their research points to four stages in the resolution of ruptures. First, the therapist notices the rupture (e.g., client withdrawal or hostility). Second, the therapist helps the client explore feelings related to the rupture. Third, the therapist helps the client examine avoidance related to discussing the rupture. Finally, the client begins to be able to explore the precipitants of the rupture. Continued research is needed about which clients are most suited for this kind of work, as well as about when and how to do it. Interaction Amongst Client, Therapist, and Relationhip Variables in Leading to Outcome Beyond determining the effectiveness of the individual components, it is important for researchers to investigate the intertwining of client characteristics, therapy characteristics, and the relationship in terms of their impact on psychotherapy. Psychotherapists probably build the relationship through specific techniques with clients who are motivated, and then they use the relationship as a foundation for using other techniques to help clients change (Hill, 2005). But clients have to be involved and work outside of therapy, using the therapist as a consultant, and thus circumstances outside of therapy also facilitate or hinder the change process. Furthermore,

Psychotherapy Research 17 therapy is probably a mediated process, such that a certain amount of exploration is needed before the client can gain insight, which is needed before the client can take action; different therapist techniques undoubtedly facilitate these different steps. Although researching the interaction amongst the three components will be difficult, this is clearly a good direction for future exploration. Furthermore, new methods are needed for linking process to outcome. As mentioned earlier, correlational strategies (i.e., correlating the frequency or proportion of process events to outcome) seem to be inappropriate. Sequential analyses (i.e., relating process events to immediate outcomes within sessions) are slightly better but ignore important contextual variables. Event analyses are useful for examining what takes place over different episodes (e.g., ruptures, talking about the relationship) within therapy, but it is difficult to summarize across events and make linkages with treatment outcome. Qualitative strategies (i.e., interviewing therapists and clients retrospectively) are useful from a descriptive perspective but cannot provide causal evidence. Clearly, new methods are needed. What We Need to Know Clearly, we have learned much about the process of psychotherapy, yet there are a number of areas for which illuminating research is needed. As indicated, understanding is emerging with regard to client readiness for change and involvement in the therapeutic process, but we need to know more about how clients can be motivated to seek psychotherapy, what perceived barriers exist, and how help-seeking, motivation, and involvement can be increased. It may be that much of this process occurs prior to the commencement of therapy and sociocultural factors are involved.

Psychotherapy Research 18 At a microprocess level, personal reactions of clients to therapist techniques remains another area in which research can yield further knowledge. Determining how the therapeutic alliance is generated from complex interaction of therapist and client is particularly important, given the robust connection between alliance and outcome. Although it has been established that the alliance is critical to successful therapy, relatively little is known about optimal development of the alliance over time and role of the “tear and repair” of the alliance in psychotherapy. Predominantly, psychotherapy research has relied on a linear causation assumption: client and therapist characteristics therapeutic process outcome. However, most would agree that psychotherapy is a complex transaction between a therapist who has various characteristics, and clients who have various characteristics, in a particular context, with various concomitant external events, that unfolds over time, creating reciprocal causal relations with chaotic and emergent properties. Process research has great potential for understanding this level of complexity. Yet another important area for further process research involves examining the effects of various therapist interventions (e.g., interpretations, challenges) within a case study format. Once a number of case studies are done, meta-synthesis can be used to determine commonalities and differences across cases (see Iwakabe, in press). Finally, the recent research on measuring outcome to provide feedback to therapists (Lambert et al., 2005) presents another opportunity for process researchers. If feedback is effective, as it appears to be, then it would be informative to know what therapists do when they receive feedback to improve their outcomes. Integrating Process and Outcome

Psychotherapy Research 19 To this point, we have presented two very different research strategies to understanding psychotherapy. On the one hand are randomized clinical trials, focused on investigating whether a particular treatment affects outcomes vis-à-vis a control group. On the other hand are process studies that examine what happens within therapy. Although the paradigms might appear to be incommensurable, there have been increasing efforts to integrate the approaches to, in a sense, have “the best of both worlds.” Of course, the goals of the two paradigms are, in some ways, comparable. Clinical researchers are not solely on a mission to find the most efficacious treatment; they are keenly interested in how treatments work and ways to improve those treatments. A variant of the randomized clinical trial involves the dismantling a design, in which a full treatment is compared to the treatment without a purported critical ingredient (Borkovec, 1990), to test whether the ingredient is important for therapeutic benefits. Process researchers are not content to describe what happens in therapy but want to tie the various process variables to outcome. Clearly, the future ideal is an integration of process and outcome research. One perspicuous difficulty in melding the two paradigms is that most process variables cannot be experimentally manipulated. The logic of randomized clinical trials depends on an experimental manipulation in order to establish causality; for the most part, process variables, such as the alliance, client readiness for change, and, for the most part, therapist responses to clients (except in a general theoretic manner, as specified in manuals) cannot be manipulated. Nevertheless, these process variables may well be causal variables in the sense that they undoubtedly affect outcomes of psychotherapy. That is to say, the fact that practically or ethically a variable cannot be manipulated does not preclude causality, as in the example that

Psychotherapy Research 20 there is a strong consensus that smoking causes certain human disease despite the fact that no one has experimentally manipulated nicotine inhalation in humans. Transactional nature of process variables Most of the critical process variables are created in the transaction between the therapist and the client. Consider the therapeutic alliance, which cannot exist independent of the transaction. The alliance, which is conceptualized as the bond as well agreement on the tasks and goals of therapy (Bordin, 1979, 1994), refers to a collaboration between client and therapist (Hatcher & Barends, 2006). Surely, levels of the alliance cannot be experimentally manipulated; more cumbersome for understanding the alliance is the fact that it is not solely under the control of the therapist. Some clients come to therapy with reasonable set of social skills, have a decent attachment history, and are motivated to work in therapy; these clients will likely form a productive working alliance with most therapists. Other clients however, will have a different profile—poor social skills, a disturbed attachment history, and a resistance to change—and they will have difficulty forming a bond with the therapist and may well resist setting goals and fail to comply with the tasks of therapy (Mallinckrodt, 1991). On the therapist side, some therapists will form alliances across a variety of clients; that is, a given client will report a better alliance with some therapists than the client would with other therapists. The therapist and the client brings to therapy certain propensities, but nevertheless the alliance is built in the context of the therapy, changes over the course of therapy, and might well depend on the match of the therapist and client (i.e., the “chemistry” of this particular client/therapist pair). Moreover, the alliance may also be, to a large extent, the result of early benefit of therapy. So, the association of alliance an outcome could be due to (a) the client’s contribution to the alliance, (b) the therapist’s contribution to the alliance, (c) the match of the therapist and the client, or (d) early improvement

Psychotherapy Research 21 in therapy (DeRubeis, Brotman, & Gibbons, 2005). Unfortunately, neither the typical RCT nor the typical process research can discriminate among the four explanations, limiting knowledge about psychotherapy process and outcome. However, as discussed below, there are statistical methods coming on line that are suitable to investigating such questions, but first an ignored aspect of therapy should be included. Therapist Effects At the origins of the randomized design in agriculture, medicine, and education, the provider of the practice (viz., the farmer, the physician, and the teacher) was not considered as an important factor to consider in the design and analysis of experiments that focused on outcomes (Danziger, 1990; Serlin, Wampold, & Levin, 2003; Wampold, 2001). This omission occurred despite the fact that the statistical machinery necessary for estimating the importance of the provider was well known since the origins of the analysis of variance (Serlin et al., 2003). Psychotherapy research has been no different (Wampold, 2001), despite evidence that psychotherapists, within treatments, in clinical trials and in practice, vary considerably in their outcomes (Huppert et al., 2001; Kim, Wampold, & Bolt, 2006; Wampold & Brown, 2005). Although the therapist is often a focus for process researchers, its presence in outcome research has largely been ignored. Ignoring therapist effects in clinical trials has three pernicious effects. First, it is important to know whether some therapists consistently attain better outcomes than others. If this is the case, as it appears that it is, then future research should be addressed toward investigating the characteristics and actions of the more effective therapists. Second, ignoring therapist effects creates inferential problems for RCTs. Failing to consider that clients are nested within therapists (i.e., each therapist treats multiple clients) inflates treatment effects (Wampold

Psychotherapy Research 22 & Serlin, 2000). As well, ignoring therapists limits the generalizability of evidence produced by RCTs as the results are specific to the therapists who conducted the treatment (Serlin et al., 2003); this is exacerbated by the fact that the number of therapists in RCTs is typically small and the therapists are selected for their expertise and are provided training, supervision, and feedback. Third, ignoring therapists precludes investigating how process variables are related to outcome. That is, client, therapist, and match contributions to a phenomenon cannot be disentangled if the therapist is not considered (Baldwin, Wampold, & Imel, 2007). Multilevel Models Multilevel statistical models can be applied in psychotherapy research to account for the fact that clients are nested within therapists, (Hox, 2002; Snijders & Bosker, 1999). Multilevel models estimate the proportion of variability in outcomes due to variables at various levels (e.g., client level, therapist level, and treatment level) so that the importance of the variables for outcome can be assessed. Thus, multilevel models address questions such as, “How important are therapist variables vis-à-vis treatment variables?” Multilevel models can then be used to disentangle therapist and client contributions to various process variables. For example, Baldwin and colleagues (2007) used multilevel models and found that the therapist’s contribution to the alliance was related to outcome whereas the client’s contribution was not. That is, therapists who generally had higher alliances with their clients also had generally better outcomes, but clients who had better alliances with a given therapist did not have better outcomes than other clients seeing the same therapist. The same method can also be used examine other process variables. Adherence to a treatment protocol is an important variable in psychotherapy research (Waltz, Addis, Koerner, & Jacobson, 1993). However, adherence to the protocol is not simply a therapist variable, as

Psychotherapy Research 23 cooperative and motivated clients permit their therapists to adhere to the typical sequence of a treatment whereas and resistant and difficult clients can create havoc for adherence. Thus, multilevel models could illuminate the effects of adherence on outcome. Multilevel models can also incorporate longitudinal data so that the process of psychotherapy can better be examined. These models are best applied when both process and outcome measures are measured regularly so that the cause and effect can be examined (Klein et al., 2003). Such examinations should always be nested within therapist so that both temporal and nested aspects of psychotherapy can be disentangled. What We Need to Know Clinical trials have focused on particular treatments whereas process research has mined the complexity of the interaction between therapist and client. Future research needs to integrate the two strands to explore the complexities of psychotherapy. There is little doubt that the therapist delivering a particular treatment is critical to success and consequently future investigations, including RCTs and process studies, will need to include therapists as a factor. This necessitates attention to power issues and an increase in the number of therapists in studies. A fundamental unanswered question in psychotherapy is: What are the characteristics and actions of effective therapists? Once that question is answered, attention can be focused on how to better train therapists to be effective. To understand how the process of psychotherapy unfolds over time, various outcome and process variables will need to be collected overtime. Recently there has been increased attention to studying the trajectories of change (e.g., Baldwin, Berkeljon, Atkins, Olsen, & Nielsen, 2009) and future research is need in this area. Attending to Biological Processes in Psychotherapy

Psychotherapy Research 24 Biological variables can be just as relevant understanding psychotherapy and selecting appropriate treatments as psychological ones. With the decoding of the human genome and the rapid advance in gene processing technologies, it has become possible to relate genetic variation to treatment response. Much of the recent efforts to personalize medicine have been focused on identifying those genes that predict differential response to different medications. The same logic can be as readily applied to different psychotherapies and medications for mental disorders. Ideally, blood or tissue samples could be subjected to genetic analysis and related to variability in subsequent treatment response. We know of no study that has done so to date, but the technologies currently exist and the possible benefits are evident. We know that genes are only part of the story (most non-psychotic disorders are less than half heritable) but any strategy that helps account for even part of the variability in outcome will facilitate the detection of the remaining components. For example, detecting the presence of a gene-environment interaction facilitated specification of the causal role of negative life events in the etiology of depression (Caspi et al., 2003). The same kind of logic can be used to isolate the extent to which psychotherapy plays a causal role in the amelioration of psychological distress. Even in disorders as heritable as autism, insights gleaned from cognitive and developmental neuroscience have been used to develop psychosocial strategies that slow the expression of the disorder (Dawson, 2008). Similarly, recent advances in neural imaging provide a means of refining what we know about how psychotherapy works and developing novel psychosocial interventions. Any change in affect or behavior has underlying neural correlates and any encoding of memories or learning of new information involves physical changes in the brain. Specifying the neural correlates of the changes induced by psychotherapy will contribute to a greater understanding of the mechanisms

Psychotherapy Research 25 through which those treatments work and the patient process that contribute to the etiology and maintenance of the disorder. This will be true whether those neural mechanisms are specific to the particular intervention or not. For example, both cognitive behavior therapy (CBT) and medication treatment for obsessive compulsive disorder were found to be associated with decreased caudate activity in treatment responders and reduced linkage between the orbitofrontal cortex, the caudate, and the thalamus (Baxter et al., 1992). This suggests that these two very different types of intervention either mobilize similar neural mechanisms in the process of change or that changes in those specific neural regions are epiphenomenal consequences of successful treatment and do not play a causal role in the production of change. Conversely, there are indications that CBT and medications may work through different neural mechanisms in the amelioration of depression (Goldapple et al., 2004). Depression is marked by hyperactive limbic and brain stem processes that contribute to emotional reactivity and hypoactive cortical process that typically serve to regulate those lower processes. Antidepressant medications appear to work from the “bottom up” by turning down those hyperactive limbic and brain stem processes whereas CBT appears to work from the “top down” by strengthening higher cortical regulatory processes (DeRubeis, Siegle, & Hollon, 2008). These differences in patterns of neural changes are particularly intriguing in that CBT appears to have an enduring effect not found for medications after treatment termination (Hollon, Stewart, & Struck, 2006). Although it has yet to be tested explicitly, it would be interesting to relate neural changes specific to CBT to its enduring effects. We do not yet know what neural processes mediate CBT’s enduring effects, but we do know that something is different in the brains of patients who respond to CBT versus patients who respond to medications alone.

Psychotherapy Research 26 Neural imagining can not only help us refine our understanding of how different treatments work, it also can be used to guide the development of novel treatments. For example, training patients to recognize bird calls serves to strengthen the anterior cingulate, a region of the brain that plays a role in regulating lower limbic processes, and leads to a reduction in affective distress (Siegle, Ghinassi, & Thase, 2007). The idea was suggested by an understanding of the ways that different neural systems interact and led to the adoption of a strategy that proved to be successful despite the complete lack of face validity. Adding to the intriguing possibilities is research on the neurochemical effects of placebos and expectations in real and sham treatments (Benedetti, 2009). Our growing understanding of the neural substrates of behavior may even deepen our understanding of the mechanisms underlying preventive interventions. Maier and colleagues have shown that rats exposed to controllable stress are inoculated against the deleterious effects of subsequent exposure to uncontrollable stress (Maier, Amat, Baratta, Paul, & Watkins, 2006). Moreover, they have been able to identify the neural pathways that mediate this learned resilience. In brief, exposure to uncontrollable stress activates the raphe nucleus in the brainstem that in turns sends serotonergic projections to the amygdale and other limbic structures that activate the stress response. However, exposure to controllable stress activates the ventral medial prefrontal cortex (mPFCv), a higher cortical center that has direct inhibitory connections to the raphe nucleus. Pharmacological inhibition of mPFCv firing during exposure to controllable stress blocks the acquisition of this acquired resilience (the rats act as if they had been exposed to uncontrollable stress), whereas pharmacological facilitation of mPFCv firing during exposure to uncontrollable stress leads to resilience in the face of subsequent stress (the rats act as if they had been exposed to controllable stress). We have long known that a sense of control provides a

Psychotherapy Research 27 psychological buffer against stress, but we have not understood the neural mechanisms that underlie that effect. We now have efficacious preventive interventions for disorders like depression (Garber et al., 2009). It would be of interest to see if those interventions mobilize the same neural processes that have been shown to facilitate resilience in animal experimentation. There is no necessary incompatibility between biological and psychological explanations. Everything we do rests on a biological basis and a better understanding of the neural substrates underlying human behavior will only facilitate our understanding of the processes that drive therapeutic change. That is not to say that all psychological processes must be explained in biological terms or that somatic or pharmacological interventions are necessarily superior to psychological ones; just that we enrich our understanding of what goes on in psychotherapy to the extent that we understand the neural processes that underlie those changes. With the advent of modern tools for exploring genetic variation and imaging neural processes, we move closer to realizing Freud’s dream of understanding the neural processes underlying human behavior. Conclusions In this chapter, we have discussed two paradigms -- clinical trials (outcome research) and process research -- as well as means to integrate the two. As well, we have speculated about how neuroscience might inform the field about the process and outcome of psychotherapy. There are those who see another distinction, namely the quantitative versus qualitative approaches. This distinction, alluded to in this chapter, is grounded in a philosophy of science argument, a full discussion of which is beyond the scope of this chapter (see Hunsberger & Slife, 2007; Schneider, 2008). An argument can be made that quantifying discrete aspects of psychotherapy places the procedure in a positivist framework when the phenomenon belongs in the humanistic and intersubjective realm, within which only phenomenological-based investigation can capture. The

Psychotherapy Research 28 exciting aspect of anticipating the future is the realization that research methods that we now cannot envisage will be used in innovative and productive ways.

Psychotherapy Research 29 References Addis, M. E., Hatgis, C., Krasnow, A. D., Jacob, K., Bourne, L., & Mansfield, A. (2004). Effectiveness of cognitive-behavioral treatment for panic disorder versus treatment as usual in a managed care setting. Journal of Consulting and Clinical Psychology, 72, 625635. Asay, T. P., & Lambert, M. J. (1999). The empirical case for the common factors in therapy: Quantitative findings. In M. A. Hubble, B. L. Duncan, & S. D. Miller (Eds.), The heart and soul of change: What works in therapy. Washington DC: American Psychological Association. Baldwin, S. A., Berkeljon, A., Atkins, D. C., Olsen, J. A., & Nielsen, S. L. (2009). Rates of change in naturalistic psychotherapy: Contrasting dose effect and good-enough level models of change. Journal of Consulting and Clinical Psychology, 77, 203-211. Baldwin, S. A., Wampold, B. E., & Imel, Z. E. (2007). Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75, 842-852. Baron, R.M., & Kenny, D.A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51, 1173-1182. Baskin, T. W., Tierney, S. C., Minami, T., & Wampold, B. E. (2003). Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71, 973-979. Baxter, L. R., Jr., Schwartz, J. M., Bergman, K. S., Szuba, M. P., Guze, B. H., Mazziotta, J. C., Alazraki, A., Selin, C. E., Ferng, H. K., Munford, P., & Phelps, M. E. (1992). Caudate

Psychotherapy Research 30 glucose metabolic rate changes with both drug and behavior therapy for obsessivecompulsive disorder. Archives of General Psychiatry, 49, 681-689. Benedetti, F. (2009). Placebo effects: Understanding the mechanisms in health and disease. New York: Oxford University Press. Benish, S. G., Imel, Z. E., & Wampold, B. E. (2008). The relative efficacy of bona fide psychotherapies for treating post-traumatic stress disorder: A meta-analysis of direct comparisons. Clinical Psychology Review, 28, 746-758. Bisson, J. I., Ehlers, A., Matthews, R., Pilling, S., Richards, D., & Turner, S. (2007). Psychological treatments for chronic post-traumatic stress disorder: Systematic review and meta-analysis. British Journal of Psychiatry, 190, 97-104. Bordin, E. S. (1979). The generalizability of the psychoanalytic concept of the working alliance. Psychotherapy: Theory, Research, and Practice, 16, 252-260. Bordin, E. S. (1994). Theory and research on the therapeutic working alliance: New directions. In A. O. Horvath & L. S. Greenberg (Eds.), The working alliance: Theory, research, and practice (pp. 13-37). New York: Wiley. Borkovec, T. D. (1990). Control groups and comparison groups in psychotherapy outcome research. National Institute on Drug Abuse Research Monograph, 104, 50-65. Beutler, L. E., Malik, M., Alimohamed, S., Harwood, T. M., Talebi, H., Noble, S., Wong, E. (2004). Therapist variables. In M. J. Lambert (Ed.), Handbook of psychotherapy and behavior change (5th ed., 194-226.). New York: Wiley. Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research and teaching. Chicago: Rand McNally.

Psychotherapy Research 31 Caspi, A., Sugden, K., Moffitt, T. E., Taylor, A., Craig, I. W., Harrington, H. L., McClay, J., Mill, J., Martin, J., Braithwaite, A., & Poulton, R. (2003). Influence of life stress on depression: Moderation by a polymorphism in the 5-HTT gene. Science, 301, 386-389. Chambless, D. L., & Ollendick, T. H. (2002). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685-716. Clarkin, J. F., & Levy, K. N. (2004). The influence of client variables on psychotherapy. In M. J. Lambert (Ed.), Handbook of psychotherapy and behavior change (5th ed., 194-226.). New York: Wiley. Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Cronbach, L. J. (1957). Two disciplines of scientific psychology. American Psychologist, 12, 671-684. Danziger, K. (1990). Constructing the subject: Historical origins of psychological research. Cambridge, UK: Cambridge University Press. Dawson, G. (2008). Early behavioral intervention, brain plasticity, and the prevention of autism spectrum disorder. Development and Psychopathology, 20, 775-803. DeRubeis, R. J., Brotman, M. A., & Gibbons, C. J. (2005). A Conceptual and Methodological Analysis of the Nonspecifics Argument. Clinical Psychology: Science and Practice, 12, 174-183. DeRubeis, R. J., Siegle, G. J., & Hollon, S. D. (2008). Cognitive therapy versus medication for depression: Treatment outcomes and neural mechanisms. Nature Reviews Neuroscience, 9, 788-796.

Psychotherapy Research 32 Eugster, S., & Wampold, B. (1996). Systematic effects of participant role on evaluation of the psychotherapy session. Journal of Consulting and Clinical Psychology, 63, 268-278. Eysenck, H. J. (1952). The effects of psychotherapy: An evaluation. Journal of Consulting Psychology, 16, 319-324. Frank, J. D., & Frank, J. B. (1991). Persuasion and healing: A comparative study of psychotherapy (3rd ed.). Baltimore: Johns Hopkins University Press. Fournier, J. C., DeRubeis, R. J., Shelton, R. C., Hollon, S. D., Amsterdam, J. D., & Gallop, R. (in press). Prediction of response to medication and cognitive therapy in the treatment of moderate to severe depression. Journal of Consulting and Clinical Psychology. Garber, J., Clarke, G. N., Weersing, V. R., Beardslee, W. R., Brent, D. A., Gladstone, T. R. G., DeBar, L. L., Lynch, F. L., D'Angelo, E., Hollon, S. D., Shamseddeen, W., & Iyengar, S. (2009). Prevention of depression in at-risk adolescents: A randomized controlled trial. JAMA, 301, 2215-2224. Gelso, C. J., & Hayes, J. A. (2007). Countertransference and the therapist’s inner experience.: Perils and possibilities. Mahwah, NJ: Lawrence Erlbaum. Goldapple, K., Segal, Z., Garson, C., Lau, M., Bieling, P., Kennedy, S., & Mayberg, H. (2004). Modulation of cortical-limbic pathways in major depression: Treatment-specific effects of cognitive behavior therapy. Archives of General Psychiatry, 61, 34-41. Hatcher, R. L., & Barends, A. W. (2006). How a Return to Theory Could Help Alliance Research. Psychotherapy: Theory, Research, Practice, Training, 43(3), 292-299. Hill, C. E. (2005). Therapist techniques, client involvement, and the therapeutic relationship: Inextricably intertwined in the therapy process. Invited article in Psychotherapy: Theory, Research, Practice, Training, 431-442.

Psychotherapy Research 33 Hill, C. E., Corbett, M. M., Kanitz, B., Rios, P., Lightsey, R., & Gomez, M. (1992). Client behavior in counseling and therapy sessions: Development of a pantheoretical measure. Journal of Counseling Psychology, 39, 539-549. Hill, C. E., Helms, J. E., Tichenor, V., Spiegel, S. B., O'Grady, K. E., & Perry, E. S. (1988). The effects of therapist response modes in brief psychotherapy. Journal of Counseling Psychology, 35, 222-233. Hill, C. E., & Knox, S. (2009). Processing the therapeutic relationship. Psychotherapy Research, 19, 13-29. Hill, C. E., Sim, W., Spangler, P., Stahl, J., Sullivan, C., & Teyber, E. (2008). Therapist immediacy in brief psychotherapy therapy: Case study II. Psychotherapy: Theory, Research, Practice, Training. Hill, C. E., Stahl, J., & Roffman, M. (2007). Training novice therapists: Helping skills and beyond. Invited paper in Psychotherapy: Theory, Research, Practice, Training, 44, 364370. Hollon, S. D., Stewart, M. O., & Strunk, D. (2006). Cognitive behavior therapy has enduring effects in the treatment of depression and anxiety. Annual Review of Psychology, 57, 285315. Horvath, A. O., & Bedi, R. P. (2002). The alliance. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 3770). New York: Oxford University Press. Hox, J. (2002). Multilevel analysis: Techniques and applications. Mahwah NJ: Lawrence Erlbaum. Huppert, J. D., Bufka, L. F., Barlow, D. H., Gorman, J. M., Shear, M. K., & Woods, S. W.

Psychotherapy Research 34 (2001). Therapists, therapist variables, and cognitive behavioral therapy outcomes in a multicenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747-755. Hunsberger, P. H., (2007). Reestablishing clinical psychology’s subjective core. American Psychologist, 62, 614–615. Iwakabe, S. (in press, 2009). From Single Case Studies to Practice-Based Knowledge: Aggregating and Synthesizing Case Studies. Psychotherapy Research, Karver, M., Shirk, S., Handelsman, J., Fields, S., Crisp, H., Gudmundsen, G., & McMakin, D. (2008). Relationship processes in youth psychotherapy: Measuring alliance, alliancebuilding behaviors, and client involvement. Journal of Emotional and Behavioral Disorders, 16, 15-28. Kazdin, A. E. (1981). Drawing valid inferences from case studies. Journal of Consulting and Clinical Psychology, 49, 183-192. Kim, D. M., Wampold, B. E., & Bolt, D. M. (2006). Therapist effects in psychotherapy: A random effects modeling of the NIMH TDCRP data. Psychotherapy Research, 16, 161172. Kasper, L., Hill, C. E., & Kivlighan, D. (2008). Therapist immediacy in brief psychotherapy therapy: Case study I. Psychotherapy: Theory, Research, Practice, Training. Klein, D. N., Schwartz, J. E., Santiago, N. J., Vivian, D., Vocisano, C., Castonguay, L. G., et al. (2003). Therapeutic alliance in depression treatment: Controlling for prior change and patient characteristics. Journal of Consulting and Clinical Psychology, 71(6), 997-1006. Lambert, M. J., Harmon, C., Slade, K., Whipple, J. L., & Hawkins, E. J. (2005). Providing feedback to psychotherapists on their patients' progress: Clinical results and practice

Psychotherapy Research 35 suggestions. Journal of Clinical Psychology, 61, 165-174. Luborsky, L., Singer, B., & Luborsky, L. (1975). Comparative studies of psychotherapies: Is it true that “Everyone has won and all must have prizes”? Archives of General Psychiatry, 32, 995-1008. Mahalik, J. R. (1994). Development of the Client Resistance Scale. Journal of Counseling Psychology, 41, 58-68. Maier, S. F., Amat, J., Baratta, M. V., Paul, E., Watkins, L. R. (2006). Behavioral control, the medial prefrontal cortex, and resilience. Dialogues in Clinical Neuroscience, 8, 353-356. Mallinckrodt, B. (1991). Clients' representations of childhood emotional bonds with parents, social support, and formation of the working alliance. Journal of Counseling Psychology, 38(4), 401-409. Miller, S. D., Duncan, B. L., & Hubble, M. A. (2005). Outcome-Informed Clinical Work. In J. C. Norcross & M. R. Goldfried (Eds.), Handbook of psychotherapy integration (2nd ed.). (pp. 84-102). New York, NY, US: Oxford University Press. Nathan, P., Stuart, S. P., & Dolan, S. L. (2000). Research on psychotherapy efficacy and effectiveness: Between Scylla and Charybdis? Psychological Bulletin, 126, 964-981. O’Malley, S., Suh, C., & Strupp, H. (1983). The Vanderbilt Psychotherapy Process Scale: A report on the scale development and a process-outcome study. Journal of Consulting and Clinical Psychology, 51, 581-586. Paul, G. L. (1967). Strategy of outcome research in psychotherapy. Journal of Consulting Psychology, 31, 109-118. Prochaska, J. O., Norcross, J. C., & DiClemente, C. C. (1994). Changing for good. New York: Guilford.

Psychotherapy Research 36 Prochaska, J. O., & Norcross, J. C. (2002). Stages of change. In J. C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 37-70). New York: Oxford University Press. Rogers, C. R., & Dymond, R. F. (1954). Psychotherapy and personality change. Chicago: University of Chicago Press. Schneider, K. (2008). Existential-integrative psychotherapy: Guideposts to the core of practice. New York: Routledge. Siegle, G. J., Ghinassi, F., & Thase, M. E. (2007). Neurobehavioral therapies in the 21st century: Summary of an emerging field and an extended example of Cognitive Control Training for depression. Cognitive Therapy and Research, 31, 235-262. Seligman, M. E. P. (1995). The effectiveness of psychotherapy: The Consumer Reports Study. American Psychologist, 50, 965-974. Serlin, R. C., Wampold, B. E., & Levin, J. R. (2003). Should providers of treatment be regarded as a random factor? If it ain't broke, don't "Fix" it: A comment on Siemer and Joorman (2003). Psychological Methods, 8, 524-534. Stiles, W. B. (1988). Psychotherapy process-outcome correlations may be misleading. Psychotherapy, 25, 27-35. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32, 752-760. Snijders, T., & Bosker, R. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London, UK: Sage. Task Force on Promotion and Dissemination of Psychological Procedures (1995). Training in and dissemination of empirically-validated psychological treatments: Report and

Psychotherapy Research 37 recommendations. Clinical Psychologist, 48, 3-23. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting and Clinical Psychology, 61, 620-630. Wampold, B. E. (2001). Contextualizing psychotherapy as a healing practice: Culture, history, and methods. Applied and Preventive Psychology, 10, 69-86. Wampold. B. E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, NJ: Lawrence Erlbaum Associates. Wampold, B. E., & Brown, G. S. (2005). Estimating therapist variability: A naturalistic study of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73, 914-923. Wampold, B. E., Mondin, G. W., Moody, M., Stich, F., Benson, K., & Ahn, H. (1997). A metaanalysis of outcome studies comparing bona fide psychotherapies: Empirically, “All must have prizes.” Psychological Bulletin, 122, 203-215. Wampold, B. E., & Serlin, R. C. (2000). The consequences of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425-433. Westen, D., Novonty, C. M., & Thompson-Brenner, H. (2004). Empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130, 631-663. Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671-689. Wendt, D. C., Jr., & Slife, B. D. (2007). Is evidence-based practice diverse enough? Philosophy of science considerations. American Psychologist, 62, 613–614.

Psychotherapy Research 38 Williams, E. N., Hurley, K., O’Brien, K., & DeGregorio, A. (2003). Development and validation of the Self-Awareness and Management Strategies (SAMS) Scale for therapists. Psychotherapy: Theory, Research, Practice, Training, 40, 278-288.