The Outcomes Assistant: A kinder philosophy to the management of outcomes
— David Kraus, Louis Castonguay, and Abe Wolf
We have been hearing about behavioral health outcome requirements for nearly fifteen years now, yet the standard practice patterns of most clinicians have still not been affected. With so many years of forewarning, additional cries that “the outcomes are coming,” are not likely to alarm our sympathetic nervous system. Rather than causing alarm, you could say that Peter has cried “Wolf” enough times that his story, and his pleas, are doing a better job of putting children and psychotherapists to sleep.
There are many complex reasons for the delay in outcomes management, but the following are the most important ones:
- outcomes management is far more complicated in the real world than any expert anticipated;
- first-generation outcome tools were too crude to show enough meaningful (clinically significant) change and typically measured only a narrow band of global issues we call ‘psychological distress,’ and ignored the multi-dimensional specificity of human psychological functioning;
- outcome reports did not provide enough assistance and advice to improve the therapeutic process and help clinicians feel that the effectiveness of their work was enhanced by the integration of outcomes management;
- and the infrastructure to process large volumes of data, generate real-time reports, track outcomes across multiple clinicians and different episodes of care, and to statistically aggregate standard analyses was not even on the drawing board.
We believe that the entire philosophy and approach to outcomes management has been off-target. We certainly believe that the pressure for accountability is here to stay; however, it should neither be the single, nor the most important use of outcome data. The entire process got off on the wrong foot when the major healthcare players gathered in the late 1980’s to discuss the use of health outcomes (Geigle & Stanley, 1990); their meeting had overwhelmingly punitive tones. For example, the consensus, number-one use of outcome data was to profile clinicians on outcomes and eliminate those with ‘documented poor quality.’ With such approaches, there is little reason to expect clinician buy-in.
We believe the principal focus of outcomes should be to guide and assist the psychotherapist in planning the treatment process. Such a tool should never prescribe a certain intervention but provide the clinician with information tailored to the patient’s assessment and condition about the relative success of various treatment options, and outline current advances in standard care by pointing to evidenced-based treatments. By properly guiding clinicians, a system of outcome management can facilitate communication between the patient and clinician while helping to identify budding problems before they become serious. Such a system is much more likely to be embraced by clinicians because it can inform and potentially improve the therapeutic process, rather than just evaluating and judging it.
“Outcomes management is far more complicated in the real world than any expert expected.”
The Treatment Outcome Package (TOP, Kraus, Jordan, & Seligman, 2005), and its supporting infrastructure, is designed to move the field of outcomes management in this more friendly direction, and bring to the forefront the positive and beneficial aspects of outcomes management. The goal of this paper is to describe the TOP as a way to highlight how the business of outcome management is evolving to meet clinician needs.
The TOP was designed to meet the recommendations of the 1994 Core Battery Conference which was organized by the Society for Psychotherapy Research and the American Psychological Association (Horowitz, Lambert, & Strupp, 1997). As a Universal Core Battery, the TOP is not tied to any specific theoretical orientation and measures many categories within symptom, functional, and quality-of-life domains. The current version of the TOP is in its fourth incarnation with 48-58 questions, depending upon the age version (child, adolescent, and adult). TOP data is processed by Behavioral Health Laboratories (BHL), which has created a centralized data warehouse that currently holds de-identified assessment data on more than 600,000 behavioral health consumers. Such massive archived data sets allow clinicians to learn by comparing their results to other clinicians who are treating similar patients. By identifying our successes and failures, we can learn from this valuable feedback system.
Stat Lab Test Results
The major reason previous generation outcome projects failed is because of data processing. From Georgia to Washington State there are countless examples of massive amounts of data being dumped into a black hole with only the remnants of destroyed phantom particles spinning off at the fringes of the void’s reach. Needless to say, it is impossible to sustain a project that cannot deliver useful results to its key participants —the patient and the psychotherapist. To survive, the outcome assistant system needed to be inexpensive, fast, and user friendly.
Whether the data is processed electronically or on paper, the BHL TOP system is designed to return useful results with the priority of a stat blood test. Paper processing is obviously the most challenging obstacle, and BHL has been a leader in simplifying this process for more than a decade.
- After the patient completes a TOP, the form is faxed to BHL’s central computer system. There, it never touches paper again. A TIF file image (the computer graphic file generated by your fax machine) is transferred to three data processing engines that translate the images into data.
- A human verifier looks over every form and makes sure the computers have correctly processed the information. The data is then transferred to the data warehouse where it is scored, compared to general population norms and any historical patient records, and a report generated.
- These reports are returned via fax or e-mail to the clinician with an average return time (from hitting send on your fax machine) of 14 minutes.
- As an alternative to a fax-based system, BHL also has an electronic/web system where the results are returned within three seconds.
BHL also provides toll-free customer service, a training video, and extensive documentation, making startup simple. By offloading the time-consuming process of warehousing and scoring reports, clinicians can stay focused on what they do best—treatment.
Patient Reports that Inform
TOP questions have high face validity to patients and psychotherapists alike. Questions are easy to read (5th grade level) and are related to key DSM symptoms when conducting an initial interview (e.g. “felt little or no interest in most things”). Years of exploratory and confirmatory factor analytic work on the TOP items reduced the number of questions to the three-to-five most powerful questions in a broad array of clinically useful domains. For the adult version, TOP domains include: Depression, Panic, Mania, Psychosis, Sleep, Sex, Work, Quality of Life, Substance Abuse, Suicide, and Violence. In contrast with outcome tools that address only one or a few dimensions of functioning, the TOP patient reports provide a wealth of clinically useful assessment data that can be easily integrated into treatment planning. Results are reported as normalized Z-scores that represent their deviation from population norms. This includes variables like life stress (Axis IV) as well as the clinical domains discussed earlier. Diagnostic considerations are reported for Axes I, III, and IV.
BHL is also finalizing a pre-filled, yet modifiable treatment plan (based on TOP responses) that is returned along with the standard TOP report, helping the therapist save time in developing an individualized course of treatment.
With assessment of dimensions like medical utilization, prior treatments, life stress, and co-morbid medical conditions, the TOP also helps paint a full picture of the patient. Clinicians can give a new patient an access code to go on-line and complete the TOP before the appointment. The clinician then has an excellent picture of the patient’s perspective of their troubles before they actually conduct the initial interview.
Links to the Research
In conjunction with Leslie Wilson and Louis Castonguay at Penn State University, each of the Adult TOP domains have been linked to a library of evidence-based practices, guidelines, and research findings that should help clinicians find the most effective treatments for patients with different TOP profiles. For example, if a patient scores very high on the Depression Scale, this TOP library integrates findings compiled by Castonguay & Beutler (2005) and other sources into an easy-to-read summary of state-of-the-art treatments.
Building on the work of Michael Lambert—who has single handedly shown that outcomes management makes us all more effective clinicians—the TOP provides early warnings if treatment appears to be heading in an unhelpful direction. Whether it might be the therapeutic alliance, or the need to incorporate adjunctive interventions like medication or family therapy, the checklist of resources to consider will help clinicians drastically reduce the number of patients categorized as “negative responders”.
Reviewing the report with the patient enhances the therapeutic alliance. Self-report of clinical symptoms can be unreliable, and having something concrete to review with patients helps to further cement the trust that you have already worked hard to create. Imagine having a report that shows your patients how much progress they have already made (from their own self-report) and how far they are from achieving their goals. The TOP results demonstrate to the patient powerful evidence that their work is heading in the right direction.
Reviewing initial reports with patients provides an excellent platform for an informed discussion of the priorities and challenges of their treatment. Six controlled studies have shown that patients are more honest about shame-based issues on questionnaires than they are in face-to-face initial evaluations (Carr & Ghosh, 1983; Erdman, Klein, & Greist, 1985; Hile & Adkins, 1997; Lucas, 1977; Searles, Perrine, Mundt, & Helzer, 1995; Turner et al., 1998). Therefore, integrating an outcome questionnaire opens exciting new channels of communication.
The rich database of TOP results is also providing opportunities to study new ways of administering items to patients. Recent developments in item response theory and computerized adaptive testing indicate that clinically reliable and meaningful results can be obtained from responses to only a few items. The BHL database of TOP results is being analyzed to identify those sets of items that have the optimal specificity and clinical “bandwidth” to evaluate symptoms and change.
Enlightening Aggregate Data
Every month, BHL sends an aggregate report that summarizes the changes of a psychotherapist’s average patient from intake and plots the changes their patients report over the course of treatment. Since more than 92% of patients report clinically and statistically significant change in at least one dimension of functioning, the TOP can provide very rewarding statistics to help psychotherapists guide their work.
In addition, BHL provides psychotherapists with unlimited access to its enormous benchmarking database. Psychotherapists can profile the types of patients with whom they work best and those patients with whom they need to improve their clinical skills. We have used this database to identify the proverbial “super shrink,” the ideal psychotherapist who is well above average on everything. The data suggests that there is no such psychotherapist—we all have our strengths and weaknesses. A more realistic goal is for all clinicians to monitor their personal strengths and weaknesses by comparing their clinical outcomes with other professionals using a standardized instrument. BHL’s database of TOP results provides just that.
The Clinical Report Card
There is a dark side to the outcome management—report cards. It is not wise for the business of outcome assessment to pursue profiling clinicians. It is premature to evaluate clinicians on the basis of one instrument. Psychotherapy is not like baseball where we can evaluate the hitters on the basis of statistics like RBIs. (And even RBIs are not that great an indicator of performance!)
Nevertheless, there are increasing pressures for accountability in our field. We cannot stop this oncoming train. We can accommodate to these pressures by using a state-of-the-art system that guides our clinical work and helps our patients. Clinical accountability may lead to the unfair use of outcome measures to profile clinicians. If we are to change our practices to incorporate measures of clinical outcome, then let us find a way to meaningfully use these instruments to guide and not just monitor treatment.
References
Carr AC, Ghosh A. Response of phobic patients to direct computer assessment. Frit J Psychiat. 1983; 142:60–65.
Erdman HP, Klein M, Greist JH. Direct patient computer interviewing. J Consult Clin Psychol. 1985; 53(6):760–773.
Castonguay LG, Beutler LE (Eds.) (2006) Principles of Therapeutic Change That Work (New York: NY: Oxford University Press)
Geigle R & Jones SB (1990) Outcomes Measurement: A Report from the Front Inquiry, Vol 27,7-13.
Hile MG, Adkins RE. Do substance abuse and mental health clients prefer automated assessments? Behavior Res Methods, Instruments, Computers. 1997; 29(2):146–150.
Horowitz LM, Lambert, MJ & Strupp HH (Eds.) (1997). Measuring patient change in mood, anxiety, and personality disorders: Toward a core battery. (Washington, D. C.: American Psychological Association Press)
Lucas RW. A study of patients’ attitudes to computer interrogation. Int J Man-Machine Studies. 1977;9: 69–86.
Kraus DR, Seligman D, Jordan JR, (2005). Validation Of A Behavioral Health Treatment Outcome And Assessment Tool Designed for Naturalistic Settings: The Treatment Outcome Package. Journal of Clinical Psychology, Vol 61(3), 285-314.
Searles JS, Perrine MW, Mundt JC, Helzer JE. Self-report of drinking using touch-tone telephone: Extending the limits of reliable daily contact. J Studies Alcohol. 1995;56(4):375–382.
Turner CF, Ku L, Rogers SM, Lindberg LD, Pleck JH, Sonenstein FL. Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science. 1998;280(May 8):867–873.
