Article Text

PDF

Do we under utilise actuarial judgement and decision analysis?
  1. John D Hamilton, MD
  1. Permanente Medical Group of Northern California Sacramento, California, USA

Statistics from Altmetric.com

Introduction

Whereas in the clinical method we use whatever data we believe relevant, actuarial judgement exclusively uses empirically established statistical links between patient data and outcome, choosing the path the “numbers” recommend. The actuarial method may rely exclusively on a main effects linear regression model, but also may use a decision tree model allowing at least 2 choice points.1 Decision analysis expands actuarial judgement to include the patient's values in addition to empirically established data in weighing options.

Comparing clinical judgement and “what the numbers say” (actuarial judgement)

The idea that actuarial judgement is often superior to clinical judgement is not new and was championed vigorously by Meehl almost 50 years ago.2–,4 Later others joined in. For example, actuarial prediction based solely on the empirically determined relation between clearly specified input data and specified criteria for psychopathology was found to equal or exceed the accuracy of prediction by clinical decision in 13 of 14 comparison studies.5 In nearly 100 comparative studies between actuarial judgement and clinical judgement, the actuarial method has equalled or surpassed the clinical method, sometimes slightly and sometimes substantially in such diverse areas as predicting parole violation,6 or predicting psychosis from MMPI scores.7 Actuarial methods have been applied to such difficult decisions as violence prediction,8–,10 deciding which children should leave preschool for kindergarten,11 choosing psychiatric medication,12 placement in children's protective services,13 and evaluating child psychopathology.14

In predicting the outcome of psychotherapy and psychoanalysis in the Psychotherapy Research Project (PRP) of The Menninger Foundation, psychologists using projective psychological test batteries alone to predict treatment course and outcome were significantly more accurate than psychiatrists using data from interviews in addition to written summaries of the projective testing.15 In almost every instance the psychologist working from the test data alone predicted less change, whereas the interviewing psychiatrist predicted more extensive changes and was usually incorrect. The conclusion was “…major questions about these patients would have been better answered if the examinations had consisted solely of testing".16

In head to head testing in a variety of settings, the actuarial method is either equal or superior to clinical judgement, and meta-analysis of studies in the social sciences and medicine comparing clinical and actuarial judgement shows an advantage for the latter.17 Limited research shows higher accuracy is achieved when clinicians rely uniformly on actuarial conclusions and avoid discretionary judgements, a result similar to the one noted by Meehl.18

Barriers to actuarial methods

Disuse of the actuarial method in psychiatry suggests that substantial barriers exist to its use. One barrier may be psychiatry's emphasis on the individual as unique rather than as a member of a class. Another is almost certainly hubris. The anxiety of giving over control to the data in anxiety provoking situations (such as predicting suicidal behaviours) may create another barrier. Such anxiety laden situations invite regression to dependence upon a wise parent figure rather than a sheet of numbers, even though the empirical literature on the prediction of suicide is substantial.13–,19 Emotionally it is comparable to being among the first to experience a computer taking full responsibility for landing an aeroplane. And would a clinician relying on numbers be seen by a frightened patient as too detached, uncaring, or even lacking in authority and confidence? Would using an actuarial approach to assessing suicidal risk hold up in court in the event of a bad outcome? Would the effort to quantify subtle feelings miss the point for many patients? A medical student diagnosed with brain cancer saw 2 different surgeons, one of whom recommended an operation and chemotherapy, and the other chemotherapy alone. He made his decision to have the operation not because of numbers, but rather on the basis of looking in one surgeon's eyes and seeing caring.20

Psychiatry and the standard gamble: including the patient in decision analysis

If actuarial methods include patient values, the result is decision analysis.21 Decision analysis fits well with those aspects of psychiatric care where there is both something to be gained and something to be lost from a standardised intervention (eg, the standard gamble), outcome data to determine the consequences of choosing one path or the other, and a patient able to quantify key values. Decision analysis may be useful in psychiatric decisions when the net result (the benefits minus the negative consequences) for 2 or more choices are not obviously different. An example is the choice to initiate, change, or discontinue a psychotropic medication in a situation with an “upside” and “downside” to each possible choice. Should a 21 year old college student with a single episode of mania during a fluoxetine trial (and therefore not classified as Bipolar I in DSM-IV) and now stable for 3 years on a low concentration of valproate continue to take it indefinitely? Or would a trial of no medication and the potential benefit of not being assigned to a lifetime of the risks associated with valproate justify the increased risk of a recurrent episode? Should a 51 year old woman with a history of recurrent major depression and an excellent response to fluoxetine marred by anorgasmia receive a trial of bupropion alone? The potential benefit is losing the anorgasmia, the potential risk is the return of her depression. In each case, there is a gamble with potential gains and losses. Which choice to make is not obvious.

Using decision analysis,21 published data on outcome probabilities can be included at the chance nodes of the decision tree, those nodes where nature decides which way things go. Choice nodes of the tree are where the clinician chooses which way things go. Both chance and choice nodes can be “exploded” so that each possibility is carried out to the terminal branches, each representing an outcome. Finally, a utility for each outcome can be quantified using some combination of the patient's values and the literature. Decision analysis's ability to include what's important to the patient is a key advantage, allowing a clear view of how different people (ie, doctors, patients, and families) make different decisions in similar circumstances.

But psychiatry and decision analysis: how good a fit?

Many aspects of psychiatric care fit well with decision analysis. Firstly, there is usually adequate time to be thoughtful about decision making. Secondly, many common situations in psychiatric practice do involve uncertainty yet require decisiveness. The patient is either bipolar and begun on a mood stabiliser or not. He or she is either admitted to the day hospital programme or continues in outpatient treatment, either receives electroconvulsive therapy (ECT) or another antidepressant. Outcome data exist for each of these decisions.

Several aspects of psychiatric practice do not fit well with decision analysis, however. Firstly, outcome in psychotherapy, a central aspect of psychiatric care, varies with the personality of the therapist and the quality of the relationship with the therapist.22 Even with manuals, many aspects of psychiatric care are less uniform than the size, weight, and content of a 10 mg tablet of fluoxetine. And quantifying feelings that are sometimes elusive is challenging.

Secondly, many important aspects of psychiatric outcome involve relationships with several individuals, so that results may be difficult to assess in a simplistic one dimensional scale. A paediatric surgeon may find either an inflamed appendix or a normal one. A child psychiatrist may improve the confidence and autonomy of an early adolescent enmeshed with his mother only to find that the mother then becomes severely depressed.

Thirdly, clinicians often use DSM-IV in making psychiatric diagnoses rather than using instruments with well defined receiver operating characteristics relative to a standard instrument. A diagnostic interview structured only by DSM-IV lacks clearly defined probe questions and clearly defined thresholds and produces poor interrater agreement.23 If a child psychiatrist is to link with the research literature in child psychiatry on major depression, for example, he or she might consider using one of the Kiddie-Schedule for Affective Disorders and Schizophrenia (K-SADS) instruments. Most publications cited in a major consensus statement about major depression in youth were based on the set of youth with major depression defined by one of the K-SADS instruments.24

Fourthly, the decision to include the patient in assessing utilities may need to include the possible role of the disorder itself in affecting judgement. A mildly hypomanic patient may be lucid but overly optimistic and minimise risk, whereas a depressed patient may be too pessimistic and certain that no treatment will help. A patient with panic disorder may be reluctant to take an adequate dose of an antidepressant due to his intense anxiety and catastrophic thinking.

Fifthly, it is time consuming, especially if the therapist is diligent about unearthing all possible data on possible outcomes from multiple databases.

Finally, the current system is in flux. The duration of psychiatric inpatient treatment, for example, has dropped during the era of managed care.25,26 Using valid probabilities about outcome in the decision to admit a patient to hospital or to treat him or her as an outpatient may be difficult, because the intensity and duration of the researchers' treatment may be markedly different than what is realistically available currently. Local knowledge of the results from a particular facility may be crucial in the decision to refer or not.

Conclusion

Despite many barriers, actuarial judgement and decision analysis have probably been underused in psychiatry, especially in making decisions about medication management.

References

View Abstract

Request permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.