Cochrane Expert Testimony

Psychology and the Law: What is Credible; What is Questionable and What is Junk?

By Dr. Gordon Cochrane

Published in the November, 2002 Advocate

Lawyers and judges utilize testimony by experts from a wide range of professions including the psychology profession. One of the reasons that psychologists and psychiatrists are frequently called upon to provide testimony and consultation is because psychological factors play a role in such a wide range of legal cases. For example, the nature and reliability of memory is a factor in many legal proceedings. As Daniel Schacter of Harvard points out in his paper, The Seven Sins of Memory 1, published in the American Psychologist, people assume that they remember events accurately and when their memories are emotional-laden, they feel even more certain about the accuracy of their recollections. However, memory is subjective and malleable rather than objective and it is not consistent over time. Kassin, et al., in their article, On the "General Acceptance" of Eyewitness Testimony Research 2, published in the American Psychologist, found that most experts now agree that memory without corroboration is not sufficiently reliable for presentation in court.

Sometimes psychologists and psychiatrists confuse their strongly felt and emphatically expressed opinions with evidence-based opinions and unless judges and lawyers know what questions to ask and unless they have a sound basis for assessing the answers given, they may reach decisions that are based on misinformation or insufficient information.

The United States Supreme Court in Daubert v Merrell Dow Pharmaceuticals, 113s.Ct. 2786 (1993) placed responsibility for determining the scientific admissibility of expert testimony in the hands of judges. The admissibility criteria are scientific in nature and quite specific. Canadian courts have not adopted these specific rules though there has been a general adherence to the principles referenced in Daubert. The Supreme Court of Canada's decision in R. v. Mohan, 2 S.C.R.9, at 25, 114 D.L.R. (4th) 419, set out the following four preconditions for admissibility of expert evidence. There must be relevance to an issue at trial, necessity in assisting the trier of fact, an absence of any exclusionary rule and the expert must be properly qualified. The requirements of a "properly qualified" expert are not defined and subsequently, Canadian judges have the task of determining the criteria for expertness in psychology on a case-to-case basis.

Most judges and lawyers understandably have little interest in becoming experts in research design, statistical analysis or ongoing research on treatment outcomes. However, by learning a few basic concepts in research design and a few basic principles of statistical analysis, they can recognize when the "show me" questions need to be asked, in what way they need to be asked and on what basis the answers need to be assessed. The purpose of this paper is to provide judges and lawyers with practical information concerning key research concepts from the field of psychology and to offer practical responses to some of their most frequently asked questions in this realm.

What constitutes an expert in psychology?

An expert in psychology should be a psychologist or psychiatrist who has training and clinical experience in the area of psychology for which his or her expertise is being sought. In addition, the expert needs to have a comprehensive and current awareness of the research literature that addresses the area of psychology in question and a conceptual understanding of the research design and statistical analysis that was used in the research reported upon in this literature.

Unfortunately, there are still some psychologists and psychiatrists who feel that the authority residing in their position is sufficient to give credibility to their opinions. Surprisingly, there are also a few psychologists and psychiatrists who are woefully lacking in their understanding of research design and the need for research validation of their opinions. For example, in a recent case the author reviewed a report in which a psychiatrist offered, "everybody knows; it is a well-established phenomenon; it is common knowledge; it is well-documented and Dr. X says so" as support for this person's "expert" opinions. Expertise rests on credibility and an expert's credibility rests on his or her awareness and understanding of the professional research in question.

What is meant by research design and how will lawyers benefit from understanding its role in the field of psychology?

Research design refers to psychology research that uses the scientific model to determine whether a specific concept, instrument or treatment is valid, reliable and effective. Much of the research published in peer reviewed psychology journals is about studies designed to assess the effectiveness of selected psychological treatments for accurately diagnosed human problems. This research is designed to minimize the influence of as many potentially confounding variables as possible. Control of extraneous variables allows the researchers to focus on the specific treatment variables and statistically assess their influence on outcome. Cochrane, in his paper, Psychology, Expert Testimony and the Law 3, used the simple metaphor of a rainmaker and a rain dance to illustrate the need for science-based research in psychology. The rainmaker-therapist may be charismatic and skilled and the rain dance may be elaborate and mysterious, but, the credibility of both the dancer and the dance rests on measurable evidence of whether it rained following the dance and, if it did rain, whether it rained because of the rainmaker's dance.

For example, Eye Movement Desensitization and Reprocessing (EMDR) has been enthusiastically embraced by many health professionals as the new and amazingly brief treatment of choice for trauma victims. Many health professionals, in spite of their inability to explain the mechanism of change that allegedly resides in specific eye movements, have expressed strong confidence in the efficacy of this treatment. However, in 2000, Macklin et al published the results of a five-year follow-up on EMDR treatment and found no differences between EMDR with eye movement and EMDR without eye movement 4. If the central variable of a treatment model can be removed from the treatment and there is no measurable difference in outcomes, where is the credibility for EMDR? Also in 2000, Goldstein et al found that EMDR was not effective for treating panic disorder 5. It seems that eye movement or the more recent finger tapping routine, which were touted as fundamental and unique to this form of treatment, are not statistically significant treatment variables. If a psychologist or psychiatrist is providing testimony concerning some important aspect of a case where the use of EMDR is a factor, lawyers need not engage the witness in an exchange about the real or imagined merits of EMDR, or of any other treatment method. Lawyers need only ask the witness to cite mainstream research evidence that shows the efficacy of the treatment in question.

How can you tell whether the references cited are credible?

Credible references cited as validation for a treatment, treatment tool, assessment instrument or psychology concept will refer to studies that were scientifically designed, were peer reviewed and were published in a credible APA journal. Books written by proponents of the treatment, assessment tool or concept in question are not research sources. Authors of these books often cross-cite other author-advocates as if validation is reached by popular vote rather than by science. If doubt remains, lawyers should ask for the specific reference(s) so the papers can be reviewed by another expert.

To avoid confusion and unintended validation for pseudoscientific claims, lawyers should not accept the use of the terms anecdotal and evidence in the same sentence. Anecdotes are not evidence. Anecdotes are unverified or unverifiable stories and testimonials that are frequently laden with bias, speculation, selective emphasis, and erroneous assumptions about cause and effect.

Concepts and treatments that have been around for a long time should not be assumed to have credible research validation simply because they have been around for a long time. Hypnosis, for example, has been around for a long time but many of the most fundamental assumptions about hypnosis lack credible research validation.

Are there limitations to the reliability of psychology research?

There are a number of limitations to scientific research in psychology. Human beings are creative, active, information-processing organisms for whom reality is unique. People are interactive rather than mechanistically reactive. Therefore, when conducting psychology research with human beings, we must do the best we can to apply the scientific principles designed to minimize the influence of as many extraneous variables as possible. Unlike plants and inanimate objects, many people make assumptions about treatment efficacy based on enthusiastic anticipation and scientific illiteracy. Researchers attempt to minimize human bias, erroneous attributions of meaning, confusion between correlation and cause-effect relationships and any factor that could lead to inaccurate conclusions.

Most psychology research is conducted by comparing the treatment outcomes of one group to those of another group. Dismantling research is also used to assess the viability of specific treatment components. Every effort is made to randomize assignment to the groups and to control for as many variables as possible but the outcomes are still group outcomes. The statistical analysis used in these studies provides probabilities rather than statements of certainty. A study might show, for example, that there is a 95% or greater probability that exposure therapy is effective in treating posttraumatic stress disorder. Because the study was conducted with groups, we cannot say for certain that a given individual will respond to exposure treatment. A small percentage of research is conducted with single cases but it is very time consuming and requires numerous carefully controlled cases before credible generalizations about treatment outcome is possible.

The application of the scientific method to humans is obviously challenging. However, L.E. Beutler, editor of the Journal of Consulting and Clinical Psychology, crystallized this issue in his 1998 article, Identifying Empirically Supported Treatments: What If We Didn't? 6 Without empirical support, testimony in psychology and psychiatry is at best questionable and at worst, it is junk.

Alan Gold repeatedly states in his paper, When "Experts" Speak Should Judges Listen? Some Basic Science Lessons, republished in The Advocate 7, that justice is poorly served if testimony in psychology or psychiatry is accepted without verification of the scientific research upon which the testimony rests. If credible research is not cited and upon questioning, cannot be cited, how can such testimony be admissible in a court of law? As Will Rogers is credited with saying, often it's not what we don't know that causes problems, it's what we know that isn't so that causes problems.

Which psychology research claims are credible, which ones are questionable and which ones constitute junk psychology?

Credible research in psychology is that which uses scientific research design. An excellent resource for lawyers who want to enhance their awareness of research design, common psychology concepts and a range of recent treatment outcomes is the Handbook of Psychological Change: Psychotherapy Processes & Practices for the 21st Century, 2000, edited by C.R. Snyder & R.E. Ingram, published by John Wiley & Sons, New York. A second source of research information that many lawyers will find helpful is Truth in Memory, 1998, edited by S.J. Lynn & K. M. McConkey, published by Guilford Press, New York. Lawyers will find, as they peruse these resources, that some advocates of empirically based treatments view psychological treatments as analogous to medical treatments in that efficacy is attributed to specific ingredients while others contend that factors such as the working alliance between the client and therapist are important aspects of outcome. These are not exclusive positions and there is research support for both views. Researchers Hyun-nie and Wampold reported in their 2001 paper, Where Oh Where Are the Specific Ingredients? A Meta-Analysis of Component Studies in Counseling and Psychotherapy, published in the Journal of Counseling Psychology, 8 that efficacy is evident for a number of treatment models but specific components of these models have not been statistically verified.

On the other hand, DeRubeis and Crits-Christoph, 1998, report in their paper, Empirically Supported Individual and Group Psychological Treatments for Adult Mental Disorders, published in the Journal of Consulting and Clinical Psychology, 9 that manual guided cognitive behavioral therapy, CBT, has been found effective for disorders such as depression and generalized anxiety. Systematic exposure therapy, which involves the development of emotional discipline, has been shown to be effective for posttraumatic stress disorder, social phobia, obsessive compulsive disorder and agoraphobia. Credibility of expert testimony in psychology is assessed upon whether the treatment in question, with or without verification of the component parts, has empirical evidence of efficacy.

Questionable claims of efficacy are sometimes made for new and unproven therapies or techniques. Until sufficient research is conducted to confirm or disconfirm the efficacy of new treatments, they must be considered unproven. Supportive anecdotes, testimonials and speculations about variables not included in the specific studies are not evidence. A qualified expert will acknowledge that unproven claims, as promising as the new model may seem, are not acceptable as credible evidence.

Junk psychology is void of research verification and it is cloaked in pseudoscientific terms or valid but inappropriately applied scientific terms. Real science starts with an untested hypothesis; junk science starts with an ordained truth. Junk psychology claim quick, simple and miraculous solutions to life's more difficult problems. These ill-defined solutions involve new and special promise-laden techniques to rid the client of his or her problem. The power to change resides in the charismatic therapist-rainmaker and the mysterious therapy-rain dance rather than in the initiative and problem-solving skills of the client. Junk psychology and justice are clearly incompatible concepts.

CONCLUSIONS:

Lawyers and judges did not go to university to become psychologists or psychiatrists; they chose law. Therefore, they can be comfortable not knowing the specifics of psychology. When in doubt, ask. When a term or concept is unclear, ask that it be explained in everyday terms. Then ask for research evidence to support the claim being made. Be most skeptical of quick fixes and terminology that sounds impressive but cannot be expressed in understandable terms.

When assessments are involved in a case, ask for evidence that the instruments used have research validation. Be particularly skeptical about projective assessment procedures. If an assessment is given on the basis of a single interview, ask for evidence to confirm the reliability of this surprisingly common subjective procedure.

In the therapy room, psychology combines science and art. In the court room, the science of psychology must take precedence over the art.

ENDNOTES

1. Schacter, D.L. (1999). The seven sins of memory: Insights from psychology and cognitive neuroscience1.. American Psychologist, Vol.54, No.3, 182-203.
2. Kassin, S.M., Tubb, V.A., Hosch, H.M., & Memon, A. (2001). On the "General Acceptance" of Eyewitness Testimony Research: A New Survey of the Experts. American Psychologist, Vol.56, No.5, 405-416.
3. Cochrane, G.J. (2000). Psychology, Expert Testimony and the Law. Washinton State Bar News, Vol. 55, No.3, 25-32.
4. Macklin, M.L., Metzger, L.J., Lasko, N.B., Berry, N.J., Orr, S.P., & Pitman, R.K. (2000). Five-year follow-up study of eye movement desensitization and reprocessing therapy for combat-related posttraumatic stress disorder. Comprehensive Psychiatry, 41(1), 24-27.
5. Goldstein, A.J., de Beurs, E., Chambless, D.L., & Wilson, K.A. (2000). EMDR for panic disorder with agoraphobia: Comparison with waiting list and credible attention-placebo control conditions. Journal of Consulting and Clinical Psychology, Vol.68, No.6, 947-956.
6. Beutler, L.E. (1998). Identifying Empirically Supported Treatments: What If We Didn't? Journal of Consulting and Clinical Psychology, Vol.66, No.1, 113-120.
7. Gold, A.D. (1999). When "Experts" Speak, Should Judges Listen? Some Basic Science Lessons. The Advocate, Vol.57, Part 6, 885-900.
8. Hyun-nie, A. & Wampold, B.E. (2001). Where Oh Where Are the Specific Ingredients? A Meta-Analysis of Component Studies in Counseling and Psychotherapy. Journal of Counseling Psychology. Vol. 48, No. 3, 251-257.
9. DeRubeis, R.J. & Crits-Christoph, P. (1998). Empirically Supported Individual and Group Psychological Treatments for Adult Mental Disorders. Journal of Consulting and Clinical Psychology. Vol. 66, No.1, 37-52.