Article Text

PDF

Why we do not abstract analogue studies of treatment outcome and scale development
  1. Shirley Reynolds, MSc,
  2. David Streiner, PhD

Statistics from Altmetric.com

One of the basic principles behind the publication of Evidence-Based Mental Health (EBMH) is that the research which we abstract in the journal should have immediate practical implications for mental health clinicians. The key question we ask ourselves in selecting studies to include in EBMH is will mental health clinicians be able to use the findings of this study in their practice? We do not believe that the results of analogue studies are useful in clinical practice, and in this note we outline our reasoning. By analogue studies, we are referring to studies in which the therapeutic setting is in some senses an “analogue” of routine clinical practice or the participants (usually students) are different from those to whom the treatment or scale will ultimately be applied.

There are various issues that determine how useful a study can be to practitioners, some of which are shared by research in other areas of health care. Thus, for example, all studies should be designed in a way that gives readers confidence in their findings. This is why EBMH and the other evidence-based journals have explicit methodological criteria for the selection of studies (detailed in our purpose and procedures section p 98-9). Other issues may be more specific to mental health. One of these is the use of analogue studies. Analogue studies are sometimes used, particularly in studies of psychotherapy, to examine the outcomes and processes of different methods of therapy. They are also used in scale development to look at reliability, validity, and factor structure.

The use of analogue studies in psychotherapy research was stimulated by experimental and social psychological research.1 Analogue research attempts to mimic real life while at the same time controlling as many extraneous variables as possible. Analogue studies therefore maximise the internal validity, or rigour, of research studies; the trade off for this rigour is reduced generalisability to real clinical settings.2 Thus, an analogue outcome study of psychotherapy might recruit university students with symptoms of depression to compare the efficacy of brief cognitive therapy from a clinical psychologist with treatment as usual from the university counselling centre. Recruitment to the study may be by advertisements put up around the university or directly from undergraduate classes. Students may be attracted by the offer of free counselling, or by the offer of course credits for completing the research project.

Analogue studies may also be used to examine questions such as treatment preferences among potential clients, or preferences for different types of therapists. Under experimental conditions, volunteer subjects can be shown standardised video taped extracts of different methods of psychotherapy, or extracts of therapists with different characteristics.

Because analogue therapy studies are relatively easy to conduct they can be extremely useful in the early stages of treatment research. Well conducted analogue studies can inform the design of subsequent clinical trials and may indicate if treatment methods are acceptable to clients, and practicable to deliver by therapists. They enable researchers to exercise control over key variables which may not be possible in clinical trials. Analogue studies can help researchers explore promising avenues of enquiry. However, even well conducted analogue studies do not provide reliable information about the effectiveness of treatment in clinical populations. Thus they are only of limited value to clinicians, whose main interest is in the effective care of patients.

In the area of scale development, analogue studies have the advantage of access to a large pool of often captive participants, who must be compliant to receive class credit. This, however, is counterbalanced by the fact that in many cases the intended targets for scales are people attending clinics or in hospital with real disorders or traits that are interfering with their lives. Assessing reliability or validity with students, and then assuming these results also apply to clinical populations is misguided. Reliability and validity are not invariant properties of the test; they are functions of the test score obtained under particular circumstances.3, 4 This means that a test may have high reliability and good internal consistency with one group of participants, for example, but not with another. Thus, psychometric studies with “analogue” patients rarely inform us of the utility of the test with real patients.

Analogue studies limit generalisability in several ways. Firstly, the target disorder in therapy itself may be an analogue for a clinical disorder. Coyne discusses the limits of self reported distress among college students as an analogue for clinical depression.5 Secondly, participants in analogue studies differ from clinical populations in important sociodemographic variables such as age, social class, and educational level. Less important differences between participants in analogue studies and participants in clinical trials include the method of recruitment, and the contexts in which treatment and assessment take place.

The distinction between analogue studies and clinical studies is not always clear. Important mental health services are provided to college students by college based counselling services, and college students do experience mental health problems of clinical severity. For some clinical populations, such as people with eating disorders, college students may constitute a high risk group, and treatment and psychometric studies with a high proportion of students would not necessarily be excluded from EBMH. Indeed, the first issue of EBMH in February 1998 included a study by Mintz et al in which a questionnaire to diagnose eating disorders was tested on a non-clinical sample of mainly college students, as well as on a clinical sample of women with eating disorders.6 The guiding principle for selection of studies in EBMH will continue to be the relevance of research findings to mental health clinicians.

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.