Clinical vs' Actuarial Judgment

1
Clinical vs. Actuarial Judgment

• Psych 600
• 10/29/08

2

• My way is to divide half a sheet of paper by a
  line into two columns writing over the one Pro
  and over the other Con. Then, during three or
  four days consideration, I put down under the
  different heads short hints of the different
  motives, that at different times occur to me for
  or against the measure. When I have thus got
  them all together in one view, I endeavor to
  estimate the respective weightsfind at length
  where the balance liesAnd, though the weight of
  reasons cannot be taken with the precision of
  algebraic quantities, yet, when each is thus
  considered, separately and comparatively, and the
  whole matter lies before me, I think I can judge
  better, and am less liable to make a rash step
  and in fact I have found great advantage for this
  kind of equation, in what may be called moral or
  prudential algebra.
• -Ben Franklin, advising someone how to decide

3

• Franklins prudential algebra is sort of a
  heuristic or an intuition in itself, but its
  probably superior to most other intuitions upon
  which people base their judgment. For questions
  about which there is a correct answer, this sort
  of approach is very successful.

4
Cues
Criterion
Crime severity Prior criminal history
Criminal recidivism
Undergraduate GPA GRE score Selectivity of
institution
Faculty ratings of students

• Based on available cue values, how well can we
  predict the criterion? What is the best way to
  make this prediction?

5

• The clinical method is making global, holistic,
  or expert judgments.
• The mechanical method (aka actuarial or
  statistical method) uses formulaic rules, often
  based explicitly in statistics on past data.

6

• Meehl (1954) made a lot of people hysterical
  when he reviewed 20 studies comparing the
  clinical and mechanical methods and found that in
  all but one, the mechanical method was equal or
  superior to expert judgment
• The one was later found to be a mistake
• Sawyer (1966) reviewed 45 studies without
  finding one favoring the clinician
• Grove et al. (2000) reviewed 136 studies with
  94 tied or favoring the mechanical method

7
Some examples

• Criminal recidivism
• Performance in training programs
• Prognosis following electroshock therapy
• Loan/credit risk
• Quality of wine vintage
• Diagnoses of brain damage
• Violence on psychiatric ward
• Results of American football games

8

• There is no controversy in social science which
  shows such a large body of qualitatively diverse
  studies coming out so uniformly in the same
  direction as this one. When you are pushing 90
  investigations, predicting everything from the
  outcomes of football games to the diagnosis of
  liver disease and when you can hardly come up
  with a half dozen studies showing even a weak
  tendency in favor of the clinician, it is time to
  draw a practical conclusion.
• -Meehl (1986)

9

• Model gt Clinician even when clinician has the
  models prediction (Goldberg 1967)
• Model gt Clinician even when clinician has more
  information (e.g. Grove et al 2000)
• Model of clinician gt clinician (a.k.a.
  bootstrapping) (e.g. Dawes and Corrigan, 1974)
• Improper model gt clinician (Dawes, 1979)

10
Interviews

• The personal interview is the most relied upon
  alternative to a formulaic rule for making
  selection decisions, even though its notorious
  invalidity (when unstructured) has been
  consistently noted (Kelly, 1953 Milstein et al.,
  1981 DeVaul 1987 Wiesner Cronshaw, 1988)

11
Texas Med School Interviews (Milstein et al.,
1981)

• A natural experiment presented itself when the
  Texas legislature forced a medical school to
  accept more Texas residents late in the process.
  The available candidates who were initially
  rejected had much poorer interview scores. These
  students went on to perform equally well on all
  academic measures as well as on clinical ratings
  in practicum.

12

• Common objections (not complete)
• -we use both methods
• -mechanical method favors one kind of data over
  another
• -broken leg argument (Meehl)
• -Hollerith machine argument
• -statistics dont apply to the unique individual
• -statistics are out of date/dont apply in my
  locale
• -wrong clinicians were used
• -ethical responsibility to make the decision
• -important judgments are complex and
  configural