Judgment Analysis

1
Judgment Analysis
Public Administration and Policy PAD634 Judgment
and Decision Making Behavior

• Thomas R. Stewart, Ph.D.
• Center for Policy Research
• Rockefeller College of Public Affairs and Policy
• University at Albany
• State University of New York
• T.STEWART_at_ALBANY.EDU

2
Process and structure

• Process studies are "concerned with how judgments
  are formed over time, i.e., what happens between
  the moment the cues are presented and the moment
  when a judgment is produced."
• A study of structure "focuses upon the judgmental
  output and tries to decompose this in terms of
  the input variables."
• --Brehmer and Brehmer (1988), p. 105

3
Problem Modeling the judgment process

• How is the information used in making judgments
  (i.e., the cues) related to the judgments?
• Judgment analysis
• First known application in 1920
• Modern use began in mid-50s
• Interactive software developed in 60s and 70s
• PC based software developed in 80s
• Web-based software has been used for research

4
Choosing a method for studying judgment

• Existing data
• Content analysis of existing documents
• Data on existing judgments
• Naturalistic methods
• Observation
• Interview or questionnaire
• Systematic estimation and self report (next
  slide)
• Design experiments/model judgments of scenarios
  (next slide)

5
Choosing a method for modeling judgment
Decision theory Prescriptive method Multiattribut
e utility theory analyze weights and utility
functions Can be used descriptively, but relies
on accuracy of self-report Protocol
analysis Descriptive, think-aloud during
judgment process Information integration
theory Descriptive, ANOVA, non-representative
design Conjoint measurement Descriptive, ANOVA,
non-representative design Policy
capturing Descriptive, regression,
non-representative design Judgment
analysis Descriptive, representative design
Estimation and self report
Modeling scenario judgments
6
Can people accurately describe their judgment
process?

• Protocol analysis answer Yes, if they do it as
  they are making judgments or decisions, not later
  when they are subject to errors of forgetting.
• Judgment analysis answer Yes for analytic
  processes, no for intuitive processes.
• Caveat Any method for studying judgment may
  change the process being studied.

7
Policy capturing vs. judgment analysis

• Policy capturing
• Developed by Bottenberg, Crystal, and Ward
• Regression based
• Not theoretical
• Judgment analysis
• Developed by Hammond and colleagues
• Based on Brunswiks lens model
• Requires representative design
• Not necessarily regression based

8
Judgment Analysis

• Analyze the task.
• Construct a number of scenarios by varying the
  values of multiple cues or obtain a set of real
  cases.
• Have subjects judge each of the scenarios.
• Conduct a statistical analysis of the judgments
  of each judge.

9
Example case Acute Otitis Media
10
Demonstration Task
Run demonstration
A male patient presents at a clinic with atypical
chest pain (not classic angina)
11
Judgment analysis

• Descriptive model, not prescriptive
• Does not rely on self-report, therefore useful
  when intuitive process is of interest.
• Representative design is necessary.
• Therefore requires detailed analysis of the task.

12
Analyze the task

• What is the judgment (or judgments) of interest?
• What are the cues?
• How are the cues presented (numbers, images,
  etc.)
• What are their ranges and distributions?
• What are the interrelations among the cues?
• Intercorrelations
• Other relations
• Is there a distal variable and are distal
  variable data or a distal variable model
  available?
• Examine the distribution of the distal variable.
• Weight sensitivity

13
Brunswiks lens model
Distal variable, a.k.a. Truth Correct
answer Outcome Actual event Observation Gold
standard Criterion
Judgment, a.k.a. Proximal variable Forecast Diagno
sis Prediction Evaluation Assessment
14

• Designing a Judgment Analysis Study

Distal variable
Cues
Judges
1
p
....
1
m
....
1
Derivation
Cases
n
Cases
1
Crossvalidation
Cases
n
2
Replicated
Cases
n
3
15
Cases

• Real cases
• Hypothetical cases

16

• Externalization of judgment policy

Function form Weight Organizing principle
17

• Function form
• The functional relation between a cue and the
  judgment

Positive linear
Negative linear
Negatively accelerated
Cue 1
Cue 2
Cue 3
Positively accelerated
Inverted U
Cue 4
Cue 5
18

• Weight
• The relative importance of a change from the
  "worst" value to the "best" value of a cue

19

• Two types of weights

• Measure-sensitive weights
• They apply to specific units and ranges of cues
• They convert units of cues to units of judgment
• They are required if we want to evaluate real
  cases
• Concept-sensitive weights
• They are sensitive only to relative importance of
  abstract concepts
• They cannot be applied to real cases

Denver air quality example
20

• Attributes of judgment analysis tasks determine
  what type of weights can be obtained

• Task familiarity
• Has the judge made these kinds of judgments
  before?
• Task congruence
• Information presented in concrete units of
  measurement
• Information presented in abstract units

21
Categories of judgment analysis research
Task congruence
Concrete
Abstract
Familiar
Measure-sensitive weights may be obtained
Only concept-sensitive weights can be obtained
Task familiarity
Unfamiliar
Measure-sensitive weights possible, but unlikely
Only concept-sensitive weights can be obtained
Adapted from Cooksey, R W. (1996). Judgment
Analysis, Theory, Methods, and Application, San
Diego, Academic Press
22

• Organizing Principle The principle by which
  cues are organized into a judgment

Additive
Multiplicative
Many others ...
23

• External validity of judgment analysis

• Available empirical evidence is inadequate to
  make a strong case for (or against) the external
  validity of judgment analysis.
• We need to develop a standard for designing and
  analyzing validation studies.
• Differences in external validity across
  individuals and tasks are substantial.
• Studies comparing the results (e.g., weights,
  means, standard deviations) of the analysis of
  judgments of paper and real cases find
  differences between paper and real cases.
  Studies computing correlations between paper and
  real cases, and particularly those relying on G,
  find a high degree of similarity.
• Higher correspondence between judgments of paper
  and real cases is likely to be found if the
  judgments are aggregated over individuals than if
  individual models are compared.
• External validity is likely to be greatest when
  the task is familiar and information is presented
  in a concrete format.

24
How do we evaluate methods?

• Theoretical basis
• Fit
• Lack of fit
• Crossvalidation (prediction in a similar set of
  cases)
• Generalizability (prediction in a natural
  setting)
• Cognitive simplicity (e.g., fast and frugal)
• Advancement of research goals
• Usefulness (learning, interpersonal
  communication, conflict management)
• Reliability of results
• Reactivity or obtrusiveness
• Others?

25

• Issues to consider in choosing a method

Purpose of study One-shot analysis vs. iterative,
multimethod intervention Prescriptive vs.
descriptive models Nomothetic vs. idiographic
analysis Elicitation vs. construction A priori
vs. a posteriori decomposition Correspondence vs.
coherence theories Repetitive vs. one-time
judgments Task familiarity Value judgment Self
insight Representative design vs. statistical
power
26
One-shot analysis vs. iterative, multimethod
intervention
Judgment analysis appropriate in the context of
an iterative, multi-method study when ...
Issue