The World Mental Health Survey Initiative just published its report concluding, “The overall picture for the world’s mental health care is bleak” (The Nation’s Health, 2007). The overwhelming need for psychotherapy is indisputable. Yet, insurance companies have managed to devalue this need. In response, we have been struggling to show that psychotherapy is of value. Since research consistently shows that it is undeniable that psychotherapy is of value (Nathan & Gorman, 2007; Roth & Fonagy, 2005) why are we, as a profession, stuck in this situation?

Evidence Supported Treatments

Evidence supported treatments (EST’s), or Evidence Based Practice (EBP), have been touted as a way to show the value of psychotherapy. Yet, in the Fall 2007 issue of The Independent Practitioner, Dr. Ronald Fox wrote, “The discussion…quickly moved to arguments over definitions of science, what qualifies as research, and the nature of evidence” (p.181). Most disturbing, this “led to vitriolic attacks of colleagues by their opponents on both sides of the issue” (Ibid). Further, this “leads to the rejection of the very idiographic approaches that clinicians are forced to pursue by the very nature of their work” (Ibid).

It has long disturbed me how psychology is so fragmented: psychologist vs. psychologist, researcher vs. psychotherapist; cognitive behaviorist vs. psychoanalyst; Ph.D. vs. Psy.D.; even women vs. men. As paradoxical as it seems and as counter-productive as it can be, teamwork is not our greatest strength. Unfortunately, there will always be factions and politics in any group of people. Yet, our own psychotherapy work shows that “dividing and conquering” ultimately leads to defeat. Everyone loses. Surely this was not the intent of the Boulder model. It was certainly not the intent of the APA Task Force on Evidence Based Practice (2006). Skinner quoted Freud more than most of us realize (Overskeid, 2007). The trauma approach and Schema Therapy combine and use strengths of different orientations within psychology.

As a profession, we have unused strengths that can show the value of psychotherapy. Dr. Fox (Ibid) wrote that we need to “create a real profession of psychology”. He talks about the importance of visibility, relevance, and training. Building on the work of Dr. James Bray (2002; 2006), I wrote about the importance of collaborating through teamwork. (Rosenberg & Rothbaum, 2003; Rothbaum, 2006). As a psychotherapist trained first as a researcher (including earning an Ed.M. in Statistics and Measurement), I have long believed in the need to show the value of psychotherapy. If we extend the idea of teamwork to the EST/EBP issue, we may better address it. This could benefit each of us in our psychotherapy work and research, our patients and clients, our profession, policy makers and the general public. I would like to make some practical suggestions about how we might address the EST/EBP issue in a more collaborative and constructive way.

Meta-analysis

Most psychotherapists do not share my interest in statistical analysis, so a brief review might be helpful.
The usual analysis used in EST studies is meta-analysis. It is “the statistical analysis of the summary findings of many empirical studies”. Meta-analysis is a “secondary analysis” which is “the reanalysis of data for the purpose of answering the original research question with better statistical questions or answering new questions with old data” (Glass, McGaw & Smith, 1981, p.21).

Meta-analysis is a multivariate analysis, looking at treatments (X) and outcome (Y) variables to make comparisons among studies, as opposed to just within a particular study. The goal is to draw summary conclusions that can be generalized.

It looks at the average effect size, or standardized differences, of treatments (including control groups). It shows both similarities and differences among studies. It can be used to reconcile or emphasize differences in findings. The differences being analyzed have to be important enough to look at, but not so large that it doesn’t make any sense to try and combine them. For the analysis to have any meaning it must answer the question for whom and under what conditions a particular treatment works.

Meta-analysis has a long history of use by researchers to show the value of psychotherapy. Its use has been fraught with controversy since its inception (see Glass, 2000 for a review). Recently, it has been used increasingly to attack the value of psychotherapy. We have responded accordingly. That is, we have responded by counter-attacking. Our analyses are used against us from both within and outside of our profession. We have created more and more studies, books, and conferences to either show or criticize the value of certain meta-analysis studies. To a certain extent it is helpful for our profession to create well designed studies using meta-analysis and use them to show value and to devalue criticisms of psychotherapy. However, we have missed an important opportunity. We have not shown in a constructive way that there are conditions where meta-analysis may not be the best choice of analysis or that other analyses may either be used instead or to compliment it. “Statistical precision cannot substitute for conceptual clarity” (Light & Pillemer, 1984, p.10).

Use of Meta-analysis

One of the most common criticisms of meta-analysis is that it gives unequal weight to studies as it averages effect sizes (see Blanton & Jaccard, 2006). This can be understood intuitively. We remember, from statistics classes, that using an average loses information. One or more extreme values can inflate or deflate an average. Criticisms also include the importance of the error term (random variation in individual measurements which cannot be known), used in calculating effect size. One way to understand this is to remember how we learned that multiple t tests should be abandoned in favor of analysis of variance, which looks at both within group and between group variation and their respective measurement errors. This reduces the probability that too many tests will produce significant findings by chance alone and inflate the error term.

Measurement error is tricky. Large studies are more likely to reveal statistically significant findings. Yet significant findings in smaller studies may also be meaningful, albeit not as powerful statistically. Similar findings from studies that are constructed differently can be as important as similar findings from studies constructed similarly.

Many criticisms of meta-analysis studies of psychotherapy have to do with which variables should and should not be included in the analysis. Some studies build questionable variables into the design as additional variables. Other studies argue that such variables are too complex and cannot be reduced to a statistic, but are important enough to influence outcome, so they should not be ignored. Does statistically accounting for differences by building them into a design as variability or error, rather than further investigating them, adequately capture what each has to offer? Or is there rich variability that cannot be captured by so doing?
Some other problems in meta-analysis studies are: study selection (published/non-published); type of sampling (stratified/random); participant characteristics (gender, ethnicity, personal history), differences in therapist characteristics, change in the variables over time or in different locations, studies with multiple outcome measures, differences in quality of studies, “the apples and oranges problem” (treatments are too different to be analyzed together) etc. Operationalization, or definition of terms, is not always consistent. Is anxiety defined the same way in all studies in a particular meta-analysis? How about PTSD? Are rape, war, trauma, and chronic illness, all of which may receive the diagnosis of PTSD, similar enough to be summarized in one analysis? How are they measured? Which cognitive-behavioral interventions are similar enough to be analyzed together? How do longitudinal studies compare with more short-term studies? These and other questions are addressed by Glass, McGaw & Smith (1981), Light & Pillemer (1984) and Wampold (2001). In addition there are important issues with integrating research and practice (APA Task Force, 2006)

Other summary analyses

I think that psychoanalysts are particularly disturbed by the very idea, never mind the problems, with meta-analysis. Perhaps this is true of psychotherapy research in general. Some of this concern may be unwarranted. The feeling seems to be that most cognitive-behavioral and short-term variables are easier to measure, or have been measured more often. It is true that there are some important variables not included in many meta-analyses, partially because we believe that they cannot be measured. However, this does not mean that longer term psychoanalytic variables cannot be measured. Important psychoanalytic variables and processes such as attachment (Mahler, Pine, & Bergman, 1975) and separation (Spitz, 1946) have been captured quite well in classic studies.

Further, if we look at the work of some of our fellow APA colleagues, there are many fine studies showing the value of variables of interest to psychoanalysts in single study and multiple psychotherapy interventions. For example, well designed research has clearly shown the therapeutic importance of transference (Berenson & Andersen, 2006; Miranda & Andersen, 2007), countertransference (Gelso & Hayes, 2002), resistance (Beutler, Moleiro, & Talebi, 2002), and the repair of therapeutic alliances (Safran, Muran, Samstag, & Stevens, 2002). Many of the studies reviewed in these chapters show links to therapeutic outcomes. Even the popular press shows the importance of the unconscious (deBecker, 1997; Gladwell, 2005). Thus, we have the building blocks for collaborative research, and perhaps even meta-analysis studies.

There are many problems and challenges with research in general and meta-analysis in particular. I don’t believe that the best strategy for us is to use those problems to criticize each other’s work with such frequency and strength rather than building on it. To use a practical example, I am a member of a research team looking at relationships among psychological and metabolic variables in youngsters with diabetes. Our sample spans 15 years and includes almost 300 youngsters and their parents. Our motivation for conducting this research was that the literature is fraught with methodological and theoretical problems. Conclusions about the relationships among variables are contradictory, thus hard to interpret. Different instruments for psychological variables and different metrics for measuring diabetic control are seen throughout the literature. Other research teams, including some psychologists, are resolving these issues by conducting multi-site collaborative research, some of which is linked to treatment outcomes (Anderson, 2001; Anderson, Brackett, Ho, et. al, 1999).

Studies such as those of Anderson and colleagues may be at least as useful as any meta-analysis study. They are multi-site, have common measurement instruments and homogenous samples. It would also be interesting to compare the findings of these studies with a meta-analysis study of the literature on youngsters with diabetes and with longitudinal studies and cross sectional studies such as those of my team. How would this research compare with similar research with youngsters with cancer and their parents? Research using these designs can be linked to outcomes. They can be used to create studies using various analyses.
Additionally, a time-honored way of summarizing research findings is a literature review of individual studies. Two fine books (Nathan & Gorman, 2007; Roth & Fonagy, 2005) review a variety of studies, including those using meta-analysis as well as single studies. Our profession has a long tradition of using other summary analyses. Do we really want to ignore this tradition in favor of only using meta-analysis? Much more information can be shown by looking at different types of analyses and what each analysis can contribute to research on the value of psychotherapy. (Also, see APA Task Force, 2006, p.274).

What SHOULD our future hold for EST/EBT?

We certainly should not abandon using meta-analysis to show the value of psychotherapy, although we may want to be more aware of its problems (Blanchard & Jaccard, 2006; Light & Pillemer, 1984). Policy makers show growing interest in EST/EBT. There is so much work to be done (APA Task Force, 2006), that there is enough for everyone. We can show strengths of meta-analysis while simultaneously showing its problems (Reed & Eisman, 2006) and suggesting improvements. Many criticisms of meta-analysis can be managed with properly designed studies. It would be useful to show which studies have overcome criticisms and how they have done so. If we would focus on the strengths of the analysis and the studies where meta-analysis is well used, this would be an improvement over attacking each other’s work. How can we expect to show the value of psychotherapy to others where there is so much “vitriolic” behavior within our own profession?

Instead of trying to show what each research team is doing wrong, we could focus on what different teams are doing right. We could show the strengths of various studies. No study is perfect. It is true that many studies can be attacked and devalued based on criticisms of meta-analysis particular to that study. However, most studies actually do add something about the value of psychotherapy. We could show that one team’s research has some problems, but has important promise because of the way it addresses others. We could show that one research team has outcomes that support the work of another team. There could be suggestions about how studies using meta-analysis can add to other summary analyses. Further, each of us might gain more if we use what already exists within our profession to build on our own work.

Almost 40 years ago, Kuhn (1970) wrote about resistance to change in science. Psychotherapists are also familiar with resistance to change in individuals. In relation to the EST/EBT debate, it would be helpful for each of us to examine our resistances. What gets in the way of collaborative constructive teamwork? We are so critical of each other’s work that we are failing to emphasize some important facts such as “effect sizes for psychological interventions…rival or exceed those of widely accepted medical treatments” (APA Task Force, 2006, p. 274). I wonder how many policy makers are aware of this fact.

As a profession, we need to expand our focus and use other strengths by collaborating more with our APA colleagues. We need more teamwork within our profession. For example, we have a number of very talented statistician colleagues. They work in academia, at institutes, for the government, in think-tanks, and in industry to name a few places. Some of them do impressive clinical research which can show the value of psychotherapy while also showing the strengths of meta-analysis and other summary analyses. Imagine, if you will, APA psychoanalysts, cognitive-behaviorists, statisticians, psychotherapists and researchers collaborating to review and create studies, books, conferences, and task forces to show the value of psychotherapy.

Psychologists, of all mental health professionals, have more training in scientific methodology and research. If insurance companies try to make decisions based on meta-analysis studies do they know what they are looking at? I doubt it. Who is in a better position to explain the design and analysis of psychotherapy studies than psychologists? Who besides psychologists can better explain to policy makers which studies are well constructed, which are not and why, and how they add to each other?

We are in a unique position to become unified in addressing the EST/EBP debate. We need to put into action the conclusions of the APA Task Force (2006) and Fox (2007). I understand that everyone needs to earn a living, have the respect of colleagues, a healthy self esteem, and pursue individual professional agendas. I do too. However, I think that we would each benefit more by building on the work of our colleagues, rather than attacking each other’s work. Rather than a fragmented and contentious summary of the value of psychotherapy, we could show that, like a jigsaw puzzle, many pieces are in place. The sum of the parts is greater than the whole. As with teamwork in healthcare, when addressing the EST/EBP debate we need to “put aside unilateral needs for control” and change our focus to one of “functional interdependency” (McDaniel & Hepworth, 2004). Certainly this would be more helpful to our psychotherapy work, our research, our patients and clients, our profession, policy makers and the general public.

References

American Public Health Association (November, 2007). The Nation’s Health, 36, 15.

Anderson, B.J. (2001). Children with diabetes mellitus and family functioning: Translating research into practice. Journal of Pediatric Endocrinology and Metabolism, 14, 645-652

Anderson B. J., Ho, J., Brackett, J., Finkelstein, D. & Laffel, L. (1999). An office-based intervention to maintain parent-adolescent teamwork in diabetes management: Impact on parent involvement, family conflict, and subsequent glycemic control. Diabetes Care, 22, 713-721.

APA Presidential Task Force on Evidence-Based Practice (2006). Evidence-based practice in psychology. American Psychologist, 61, 271-283.

Berenson, K. R., & Andersen, S.M. (2006). Childhood physical and emotional abuse by a parent: Transference effects in adult interpersonal relationships.  Personality and Social Psychology Bulletin, 33, 1509-1522.

Beutler, L.E., Moleiro, C.M. & Talebi, H. (2002). Resistance. In (Ed.) J.C. Norcross, Psychotherapy Relationships that Work: Therapist contributions and responsiveness to patients. New York: Oxford University Press.

Blanton, H, & Jaccard, J. (2006). Arbitrary metrics in psychology. American Psychologist, 61, 27-41.

Bray, J.H. (2002). Collaborative practice between psychologists and primary care physicians: Marketing your practice. The Independent Practitioner: Bulletin of psychologists in independent practice, 22, 194.

Bray, J.H. (2006). Expanding your practice into primary care. The Independent Practitioner: Bulletin of psychologists in independent practice, 26, 84.

deBecker, G. (1997). The Gift of Fear. New York: Random House.

Fox, R. E. (2007). Towards creating a real profession of psychology, The Independent Practitioner: Bulletin of psychologists in independent practice, 27, 180-186.

Gelso, C.J. & Hayes, J.A. The management of countertransference. In (Ed.) J.C. Norcross, Psychotherapy Relationships that Work: Therapist contributions and responsiveness to patients. New York: Oxford University Press.

Gladwell, M. (2005). Blink. New York: Little Brown and Company.

Glass, G.V. (2000). Meta-analysis at 25. Unpublished paper, Arizona State University.

Glass, G.V., McGaw, B. & Smith, M.L. (1981). Meta-analysis in Social Research. Beverly Hills: Sage Publications.

Kuhn, T.S. (1970). The structure of scientific revolutions. Chicago: The University of Chicago Press.

Light, R.J. & Pillemer, D.B. (1984). Summing Up: The science of reviewing research. Cambridge MA: Harvard University Press.

Mahler, M.S., Pine, F., & Bergman, A. (1975). The Psychological Birth of the Human Infant: symbiosis and individuation. New York: Basic Books.

McDaniel, S.H., & Hepworth, J. (2004). Family psychology in primary care: Managing power and dependency through collaboration.In R.G. Frank, S.H. McDaniel, J.H. Bray, &. Heldring (Eds.), Primary Care Psychology. Washington, DC: American Psychological Association.

Miranda, R., & Andersen, S.M. (2007). The therapeutic relationship: Implications from the social-cognitive process of transference. In P. Gilbert & R. Leahy (Eds.), The Therapeutic Relationship in the Cognitive Behavioral Psychotherapies (pp. 63-89). London: Routledge.

Nathan, P. E. & Gorman, J.M. (2007). A Guide to Treatments that Work. New York: Oxford University Press.

Overskeid, G. (2007). Looking for Skinner and finding Freud. American Psychologist, 62, 590-595.

Reed, G.M. & Eisman, E. J. (2006). Uses and Misuses of Evidence: Managed care, treatment guidelines, and outcomes measurement in professional practice. In C.D. Goodheart, A.E. Kazdin, and R.J. Sternberg (Eds.) Evidence-Based Psychotherapy: Where practice and research meet. Washington DC: American Psychological Association.

Rosenberg, A.R. & Rothbaum, P.A. (2003). Physician-psychologist teamwork: Twelve years of practice. The Independent Practitioner: Bulletin of psychologists in Independent practice, 23, 6-9.

Roth, A. & Fonagy, P. (2005). What Works for Whom: A critical review of psychotherapy research. New York: The Guilford Press.
Rothbaum, P.A. (2006). Learning to be a healthcare team member: Building resilience for patients, colleagues, and ourselves. The Independent Practitioner: Bulletin of psychologists in independent practice, 26, 146-149.

Safran, J.D., Muran, J.C., Samstag, L.W., & Stevens, C. (2005). Repairing Alliance Ruptures. In (Ed.) J.C. Norcross, Psychotherapy Relationships that Work: Therapist contributions and responsiveness to patients. New York: Oxford University Press.

Spitz, R. (1946). Hospitalism: A follow-up report. Psychoanalytic Study of the Child, 1, 53-74.

Wampold, B.E. (2001). The Great Psychotherapy Debate: Models, methods, and findings. Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Peggy A. Rothbaum, Ph.D. is a psychologist, researcher, and writer. She can be contacted at 232 Saint Paul Street, Westfield, New Jersey 07090.