Review of graphic representation of uncertainty

1
Review of graphic representation of uncertainty
Suraje Dessai Tyndall Centre for Climate Change
Research, UK and School of Environmental
Sciences, University of East Anglia,
Norwich Expert meeting Uncertainty
Communication 10 December 2004, Utrecht
2
Jeroen asked me to give this presentation, but

• Disclaimer Im not an expert on graphics In
  fact I usually ask a colleague to do my graphs!
  You probably wont believe in this presentation
  anymore.
• I started my research by asking some colleagues
  if they new of any research in the area of
  graphic representation of uncertainty their
  answers

3
Pseudo expert elicitation of literature

• Jeroen Look at Tufte, E.R. (1983) The visual
  display of quantitative information. Graphics
  Press
• John Handmer, RMIT University has done some work
  in the context of communicating uncertainty in
  short-term weather forecasts, but has lacked
  funding to to look explicitly at the graphs is
  planning some focus groups on this though
• Tony Patt/Torsten Grothmann, PIK Look at Morgan
  and Herion (1999) and recent IPCC workshop report
  (2004)
• Richard Moss, CCSP After an examination of the
  literature Moss and Schneider (2000) decided
  Tukey box plots was a good way to represent
  uncertainty in graphs.
• Tony Leiserowitz, Decision Research he asked his
  physiologist friends but nothing

4
There is surprisingly little research done on the
evaluation of the effectiveness of different
graphical ways of communicating uncertainty!

• One exception is Ibrekk and Morgan (1987) so
  lets look at what they did.
• They used nine different graphs to display
  uncertainty and evaluated their ability to
  communicate to nontechnical people.

5
The nine graphical representation were

• Traditional point estimate with an error bar
  that spans a 95 confidence interval
• Bar chart (discretized version of the density
  function)
• Pie chart (discretized version of the density
  function)
• Conventional probability density function (PDF)
• Probability density function of half its regular
  height together with its mirror imagine
• Horizontal bars of constant width that had been
  shaded to display probability density using dots
• Horizontal bars of constant width that had been
  shaded to display probability density using
  vertical lines
• Tukey box modified to indicate the mean with a
  solid point
• Conventional cumulative distribution function
  (CDF), the integral of the PDF.

6
Respondents were asked to
Context Subjects were told to suppose that they
were going to a mountain cabin next Friday and
were concerned about snow

• Estimate the forecasters best estimate
• Estimate the chances that there will be more than
  2 inches of snow
• Estimate the chances that there will be between 2
  and 12 inches of snow
• Rate how sure they felt about their answer
• Tell us whether they had ever before seen
  uncertain information communicated with this sort
  of pictures

7

• This was done twice
• without having received any explanation of how to
  use or interpret the various displays.
• With a series of nontechnical explanations of the
  meaning and use of each of the displays in the
  context of water depth in a flood.
• At the end of both sections respondents were
  asked which of the pictures would you prefer to
  have the newspaper use?

8

• On best estimate, point estimate (1) and Tukey
  box (8), which explicitly marked the location of
  the mean, had the best results. For the other
  graphs most people were influenced by the
  location of the modes. Subjects were most sure
  about their responses for displays 1 and 2 and
  least sure about 5 and 9 (CDF). Explanations had
  a weak effect. CDFs alone are likely to mislead
  significantly
• For over threshold only pie chart and CDF
  produce correct responses, although performance
  is degraded for pie chart after explanation
• For between thresholds only pie chart before
  explanation and CDF after explanation
• The best performance for a 95 confidence
  interval was display 1, followed by CDF (9)
• Before explanation subjects preferred bar and pie
  charts, while after explanation their preferred
  pie charts and CDFs

9
Some insights from this study

• The performance of a display depends upon the
  information that a subject is trying to extract
  displays that explicitly contain the information
  that people need show the best performance
• In making judgements about the location of the
  best estimate, people show a tendency to select
  the mode rather than the mean unless the mean is
  explicitly marked.
• Used alone the CDF is not a reliable way to
  communicate the mean.
• The authors conclude that a CDF plotted directly
  above a PDF with the same horizontal scale and
  with the location of the mean clearly indicated
  on both curves is a good approach

10
The literature seems to imply that graphics are a
better way to communicate uncertainty than text

• The COMET Outreach Program undertook a study
  examining different methods of communicating
  hurricane risks and uncertainties to the general
  public. A large majority of the sample (202)
  preferred a graphical approach to information
  dissemination in contrast to the text-based
  approaches in a comparison of a text-based
  win advisory versus a graphical advisory, 74.75
  of the sample preferred the graphical advisory as
  opposed to 11.88 who preferred the text-based
  product.
• But preference is not understanding Over 62 of
  the sample answered correctly, from a map of the
  probabilities that there was a 20 chance of the
  centre of Hurricane Georges striking within 75
  miles of Mobile, Alabama within 72 hours. This
  result was in striking contrast to a correct
  answer of 24.75 on a similar question using a
  copy of a text-based warning issued by the NWS
  for the probability of Georges striking Panama
  City, Florida.
• The authors concluded that there is strong
  evidence that the public is better able to
  interpret graphical products than textual
  products.

11
Of course graphic representation of uncertainty
depends on context

• So for example, if one uses the Kandlikar et al.
  (2004) scale Ambiguous direction (sign) of change
  could be represented graphically like this

12
Ambiguous direction (sign) of change
13
Ranges upper and lower bounds or as the 5th and
9rth percentiles based on objective analysis or
expert judgment
14
(No Transcript)
15
Probability distribution determined for a range
of changes in the variable either objectively or
through use of a formal decision analytic survey
or protocol
16
Cumulative probabilities
17
Which are the most promising experiments to do in
the policy lab to test different graphical
representations?

• Considering there is hardly any empirical work in
  this area and the importance of effective risk
  communication we could do almost anything!
• Here are some suggestions
• Compare text-based versus graphic-based
  communication of uncertainty (to discover the
  added value of graphics, etc.)
• Compare different graphical ways to display
  uncertainty (things have moved since 1987,
  especially in terms of computer graphics perhaps
  a good way to do this would be to ask people
  first how they would like graphs to display
  uncertainty then test a mix of public and expert
  displays in laymen to see which work best)
• It seems that most of us work in areas of deep
  uncertainty so we resort to scenarios very often
  however, these seem to be the most difficult to
  communicate uncertainty about as show by this
  example

18
Whats the best estimate of global temperature
change in 2100?

• Based on this IPCC figure most people are likely
  to say something around 3ºC. However, because
  scenarios were used to construct these figures
  and because the scenarios have no probabilities
  associated with it the correct answer would be
  the range 1.4-5.8ºC. Of course some text could
  make this clear, but this illustrates the
  difficulty of communicating uncertainty using
  graphics when there is deep uncertainty.

Does the IPCC not know how to communicate
uncertainties ?!?
19
References

• Centre for Risk Community Safety (2003)
  Communicating uncertainty in short-term weather
  forecasts. Report to the Bureau of Meteorology
• COMET (2003) http//www.comet.ucar.edu
• Ibrekk, H. and M.G. Morgan (1987) Graphical
  communication of uncertain quantities to
  nontechnical people. Risk Analysis, 7 (4),
  519-529
• Kandlikar, M., J.S. Risbey and S. Dessai (2004)
  Representing and communicating deep uncertainty
  in climate change assessment. Geosciences (in
  press).
• Morgan, M.G. and M. Henrion (1990) Uncertainty a
  guide to dealing with uncertainty in quantitative
  risk and policy analysis. Cambridge University
  Press
• Murphy, A., S. Lichtensten, B. Fischhoff and R.
  Winkler (1980) Misinterpretations of
  precipitation probability forecasts. Bulletin of
  the American Meteorological Society, 61, 695-701