Combining Expert Judgement: A Review for Decision Makers

1
Combining Expert Judgement A Review for
Decision Makers

• Simon French
• simon.french_at_mbs.ac.uk

2
Valencia 2Group Consensus Probability
Distributions
3
Valencia 2Group Consensus Probability
Distributions
Group of experts
Issues and undefined decisions
4
Different contexts ? different assumptions
appropriate

• Expert Problem
• Expert judgements are data to DM
• OK to calibrate judgements
• no assumption of equality
• Many to 1 communication
• Group Decision Problem
• two step process learn then vote
• learn from each other ? mutual communication
• wrong to calibrate at decision?
• equal voting power?
• Text book Problem
• Need to think of later unspecified decision
• Need to communicate to unspecified audiences

5
How do you question experts?
If the non-swimmer averages advice on depths he
drowns! If he were to ask the question, will I
drown if I wade across? he would get a unanimous
answer yes!
6
Approaches to the expert problem (1)

• Bayesian
• Expert judgement is data
• Difficulty in defining likelihood

DMs prior for quantities of interest in real
problem
7
Approaches to the expert problem (1)

• Bayesian
• Expert judgement is data, x
• Difficulty in defining likelihood

p(? x) ? p(x ?) ? p(?)
Posterior probability ? likelihood ? prior
probability
DMs probability for the experts
judgementsgiven actual quantity of
interestcorrelations? elicitation errors?
calibration?
8
Approaches to the expert problem (2)

• Opinion Pools
• Expert judgement are taken as probabilities
• Essentially a weighted mean
• arithmetic, geometric,
• Weights defined from
• DMs judgement
• Equal weights (Laplace, equal pay)
• Social networks
• Cookes Classical method
• Weights defined from calibration data
• Are there better scoring rules?
• Many applications
• Database of 45 studies
• Computationally easy
• Appears to discard poor assessors but actually
  finds spanning set

9
But all this is the easy bit .
Expert advice on what might happen
Expert input on models, parameters, probabilities

• cf, discussions of EDA then confirmatory
  statistics
• How do you elicit models and probabilities?
• Plausibility bias if it is the experts model?

10
Group decision problem
Many approaches

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.

11
Group decision problem
Many approaches

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.
• individuals rank using their own expected utility
  ordering then vote

12
Group decision problem
Many approaches

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.
• individuals rank using their own expected utility
  ordering then vote
• altruistic Supra Decision Maker

13
Group decision problem
Many approaches

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.
• individuals rank using their own expected utility
  ordering then vote
• altruistic Supra Decision Maker
• negotiation models

14
Group decision problem
Arrow Theorem and similar results ?

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.
• individuals rank using their own expected utility
  ordering then vote
• altruistic Supra Decision Maker
• negotiation models

Paradox and impossibility theorems abound in
group decision making theory
15
Group decision problem
Arrow and similar results ?

• Decision conferences
• Built around reference decision or negotiation
  models
• Decision analysis as much about communication as
  about supporting decision making
• Might vote or might leave the actual decision to
  unspoken political/social processes

• combine individual pi(.) and ui(.) into group
  pg(.) and ug(.) then form group expected utility
  ranking.
• individuals rank using their own expected utility
  ordering then vote
• altruistic Supra Decision Maker
• negotiation models
• social process which translates individual
  decisions into an implemented action

16
Group decision support systems

• The advent of the readily available computing
  means that algorithmic solutions to the Group
  Decision Problem are attractive.
• Few software developers know any of the theory in
  this area, and ignorance of Arrow is rife.

17
The textbook problem

• How to present results to help in future as yet
  unspecified decisions
• How does one report with that in mind?
• Public participation and the web means that many
  stakeholders to issues are seeking and using
  expert reports whether or not they understand
  them

18
Cookes Principles for scientific reporting of
expert judgement studies

• Empirical control Quantitative expert
  assessments are subjected to empirical quality
  controls.
• Neutrality The method for combining/evaluating
  expert opinion should encourage experts to state
  their true opinions, and must not bias results.
• Fairness Experts are not pre-judged, prior to
  processing the results of their assessments.
• Scrutability/accountability All data, including
  experts' names and assessments, and all
  processing tools are open to peer review and
  results must be reproducible by competent
  reviewers.

19
Cookes Principles for scientific reporting of
expert judgement studies

• Empirical control Quantitative expert
  assessments are subjected to empirical quality
  controls.
• Neutrality The method for combining/evaluating
  expert opinion should encourage experts to state
  their true opinions, and must not bias results.
• Fairness Experts are not pre-judged, prior to
  processing the results of their assessments.
• Scrutability/accountability All data, including
  experts' names and assessments, and all
  processing tools are open to peer review and
  results must be reproducible by competent
  reviewers.

Few reports satisfy this Chatham house
reporting
20
The Textbook Problem relates to

• Exploring issues, formulating decision problems,
  Developing prior distributions
• So report should anticipate meta-analyses and
  give calibration data, expert biographies,
  background information, etc.
• Since the precise decision problem is not known
  at the time of the expert studies, the reports
  will be used to build the prior distributions not
  update them
• Need meta-analytic approaches for expert
  judgement
• Little peer-review
• No publication bias
• self promotion of reports by pressure groups
• Cookes principles not even considered.

21
The textbook problem for public participation

• Public and stakeholders will need to develop
  their priors from information available
• But they will not always be sophisticated DMs nor
  will they be supported by an analyst
• Behavioural issues
• Probabilities versus frequencies (Gigerenzer)
• Risk communication
• celebrity
• Observables versus parametric constructs

22
Questions?