The human forecaster: Components of judgmental skill

1
The human forecaster Components of
judgmental skill
Workshop Doing Something about the
Weather January 3-5, 2007, Naval Postgraduate
School, Monterey, California

• Thomas R. Stewart
• Center for Policy Research
• Rockefeller College of Public Affairs and Policy
• University at Albany
• State University of New York
• T.STEWART_at_ALBANY.EDU

2
Evaluating skill of forecasts

• Skill is multidimensional
• The relation between skill and forecast value
  may differ among users
• Roebber, P. J. and L. F. Bosart (1996). The
  complex relationship between forecast skill and
  forecast value A real world analysis. Weather
  and Forecasting 11(4) 544-559.
• No summary measure of skill is adequate for all
  purposes
• Brier skill score is one summary measure
• 1.0 for perfect forecast
• 0.0 for climatological forecast
• no lower bound

3
Alternative measures of performance

• Detection theory Area under ROC curve
• Correlation (rYO)
• O is the observation, or ground truth
• Y is the judgment
• Mean square error
• Skill score

4
Decomposition of skill score

• Murphy decomposition
• Lens model equation
• Extended lens model
• Result 7 components of skill

5
Decomposing accuracy--the skill score

• Murphy (1988)
• Skill Correlation - Conditional
  bias - Unconditional bias
• score (regression bias)
  (base rate bias)

Murphy, A. H. (1988). Skill scores based on the
mean square error and their relationships to the
correlation coefficient. Monthly Weather Review,
116, 2417-2424.
6
Intuition and analysis

• Analytical thought a step-by-step, conscious,
  logically defensible process.
• Intuition signifies the opposite a cognitive
  process that somehow produces an answer,
  solution, or idea without the use of a conscious,
  logically defensible, step-by-step process.

Based on Hammond, 1996
More
7
A model of quasirational cognition Brunswik's
lens model
Cues
Observation
Forecast
O
Y
X
8
Decomposing accuracy The Lens Model Equation
(Tucker 1964)
Observation
Forecast
more
9
Expanded lens model
True
Subjective
Cues
Descriptors
Cues
Forecast
Observation
X
T
U
10
Forecasting and Decision Making Under Uncertainty
10
11
Decomposition of skill score
Skill score
12
Components of skill and the lens model
True
Subjective
Cues
Descriptors
Cues
Forecast
Observation
Environmental predictability
Reliability of information processing
Fidelity of the information system
Reliability of information acquisition
Match between environment and judge
Forecasting and Decision Making Under Uncertainty
12
13
Forecasting and Decision Making Under Uncertainty
13
14
1. Environmental predictability
Components of skill

• Environmental predictability is conditional on
  current knowledge and information. It can be
  improved through research that results in
  improved information and improved understanding
  of environmental processes.
• Environmental predictability determines an upper
  bound on performance and therefore indicates how
  much improvement is possible through attention to
  other components.

15
Environmental predictability limits accuracy of
judgment
15
16
2. Fidelity of information system
Components of skill

• Judgmental skill may be degraded if the
  information system that brings data to the judge
  does not accurately represent actual conditions,
  i.e., if the cues do not accurately measure the
  true descriptors. Fidelity of the information
  system refers to the quality, not the quantity,
  of information about the cues that are currently
  being used.
• Fidelity is improved by developing better
  measures, e.g., though improved instrumentation
  or increased density in space or time.

17
3. Match between environment and judge
Components of skill

• The match between the model of the judge and the
  environmental model is an estimate of the
  potential skill that the judge's current strategy
  could achieve if the environment were perfectly
  predictable (given the cues) and the judgments
  were unbiased and perfectly reliable.
• This component might be called knowledge. It
  is addressed by training and experience. If the
  judge learns to rely on the most relevant
  information and ignore irrelevant information,
  this component will generally be good.

18
Reliability (Consistency)
Components of skill

• Reliability is high if identical conditions
  produce identical judgments.
• Humans are rarely perfectly reliable.
• There are two sources of unreliability
• Reliability of information acquisition
• Reliability of information processing

19
Reliability
Components of skill

• Reliability decreases as amount of information
  increases.

Theoretical relation between amount of
information and accuracy of judgment
20
Reliability decreases as environmental
predictability decreases
Components of skill
20
21
4. Reliability of information acquisition
Components of skill

• Reliability of information acquisition is the
  extent to which the judge can reliably interpret
  the objective cues.
• It is improved by organizing and presenting
  information in a form that clearly emphasizes
  relevant information.

Tufte examples
22
Visual display of information influences mode of
thought
Components of skill

• Use of images tends to induce intuition
• Use of numbers tends to induce analysis

23
5. Reliability of information processing
Components of skill

• Decreases with increasing information and with
  increasing environmental uncertainty
• Methods for improving reliability of information
  processing
• Limit the amount of information used in making
  judgments. Use a small number of very important
  cues.
• Use mechanical methods to process information.
• Combine several judgments.
• Require justification of judgments.

24
Theoretical relation between amount of
information and accuracy
Forecasting and Decision Making Under Uncertainty
24
25

• The relation between information and accuracy
  depends on environmental uncertainty

- - - - - Theoretical limit of accuracy
Actual accuracy
- - - - - Theoretical limit of accuracy
Actual accuracy
26
6 and 7. Bias -- Conditional (regression bias)
and unconditional (base rate bias)
Components of skill

• Together, the two bias terms measure judgment
  "calibration. (called reliability in
  meteorology)
• Reducing bias
• Experience
• Statistical training
• Feedback about nature of biases in judgment
• Search for discrepant information
• Statistical correction for bias

27
Calibration of judgments depends on the task
Calibration data for precipitation forecasts
(Murphy and Winkler, 1974)
Heideman (1989)
28
Conclusion

• Problem Improving judgmental accuracy
• Understanding and improving judgment requires
  understanding the task and the environment.
• Decomposing skill can aid in identifying the
  factors that limit judgmental accuracy.