Uncertainty Assessment and Communication

1
Uncertainty Assessment and Communication

• Jeroen van der Sluijs
• Copernicus Institute for
• Sustainable Development and Innovation
• Utrecht University

2
Copernicus Institute

• Research topics
• Energy and material demands and
  efficiencies (Dr. Martin Patel)
• Possibilities for a more sustainable energy
  supply (Dr. Andre Faaij)
• Land use, biodiversity and ecosystem functioning
  (Prof. Dr. Peter de Ruiter)
• Innovation systems, processes and
  policies (Prof. Dr. Ruud Smits)
• Governance for sustainable development (Prof.
  Dr. Pieter Glasbergen)
• Integrative models and methods and the management
  of risks and uncertainties (Dr. Jeroen van der
  Sluijs)

3
Complex environmental risks

• Typical characteristics (Funtowicz Ravetz)
• Decisions will need to be made before conclusive
  scientific evidence is available
• Decision stakes are high potential error costs
  of wrong decisions can be huge
• Values are in dispute
• Knowledge base is mixture of knowledge and
  ignorance
• large (partly irreducible) uncertainties,
  knowledge gaps, and imperfect understanding
• Assessment dominated by models, scenarios, and
  assumptions
• Many (hidden) value loadings in problem frames,
  indicators, assumptions
• Coping with uncertainty is essential

4
RIVM / De Kwaadsteniet (1999)

• RIVM over-exact prognoses based on virtual
  reality of computer models
• Newspaper headlines
• Environmental institute lies and deceits
• Fuss in parliament after criticism on
  environmental numbers
• The bankruptcy of the environmental numbers
• Society has a right on fair information, RIVM
  does not provide it

5
Crossing the disciplinary boundaries

• Once environmental numbers are thrown over the
  disciplinary fence, important caveats tend to be
  ignored, uncertainties compressed and numbers
  used at face value
• e.g. Climate Sensitivity, see Van der Sluijs,
  Wynne, Shackley, 1998

?
!
Resulting misconception Worst case 4.5C
1.5-4.5 C
6
The certainty trough (McKenzie, 1990)
7

• In model based assessment and foresight of
  complex environmental problems,unquantifiable
  uncertainties dominate the quantifiable ones
• Unquantifiable uncertainties include those
  associated with
• problem framings
• system boundaries
• model structures
• assumptions
• indeterminacies
• value ladenness

8
Insights on uncertainty

• Uncertainty is partly socially constructed and
  its assessment always involves subjective
  judgement
• Omitting uncertainty management can lead to
  scandals and crisis
• More research does not necessarily reduce
  uncertainty
• may reveal unforeseen complexities
• irreducible uncertainty (intrinsic or
  practically)
• High quality ? low uncertainty
• Quality relates to fitness for function
  (robustness, PP)
• Shift in focus needed from reducing uncertainty
  towards systematic attempts to explicitly cope
  with uncertainty

9
Dimensions of uncertainty

• Technical (inexactness)
• Methodological (unreliability)
• Epistemological (ignorance)
• Societal (limited social robustness)

10
RIVM Uncertainty Guidance

• Goals
• Structure an approach to environmental assessment
  that facilitates an awareness, identification and
  incorporation of uncertainty
• Specifically address and relate the role of
  uncertainties in the context of policy advice
• May not reduce uncertainties, but provides means
  to assess their potential consequences and avoid
  pitfalls associated with ignoring or ignorance of
  uncertainties
• Guidance for use and help against misuse of
  uncertainty tools
• Provide useful uncertainty assessments
• Promote the adoption of uncertainty methods
• Facilitate design of effective strategies for
  communicating uncertainty.

11
Steps in uncertainty management
Communication
Problem framing context analysis
Process assessment
Reporting
Environmental Assessment methods
Uncertainty Management
Review Evaluation
Uncertainty identification prioritization
Uncertainty Analysis
12
Systematic reflection on uncertainty issues in

• Problem framing
• Involvement of stakeholders
• Selection of indicators
• Appraisal of knowledge base
• Mapping and assessment of relevant uncertainties
• Reporting of uncertainty information

13
RIVM-MNP Uncertainty Guidance
Reminder listInvokes ReflectionPortal to QS
Advice on Quantitative Qualitative tools for UA
Download all volumes www.nusap.net
14
Problem framing and context

• Explore rival problem frames
• Relevant aspects / system boundary
• Typify problem structure
• Problem lifecycle / maturity
• Role of study in policy process
• Uncertainty in socio-political context

15
Type-III error Assessing the wrong problem by
incorrectly accepting the false meta-hypothesis
that there is no difference between the
boundaries of a problem, as defined by the
analyst, and the actual boundaries of the problem
(Dunn, 1997). Context validation (Dunn, 1999).
The validity of inferences that we have
estimated the proximal range of rival
hypotheses. Context validation can be performed
by a participatory bottom-up process to elicit
from scientists and stakeholders rival hypotheses
on causal relations underlying a problem and
rival problem definitions.
16
In different phases of problem lifecycle,
different uncertainties are salient Different
problem-types need different uncertainty
management strategies
17
Involvement of stakeholders

• Identify relevant stakeholders.
• Identification of areas of agreement and
  disagreement among stakeholders on value
  dimensions of the problem.
• Recommendations on when to involve different
  stakeholders in the assessment process.

18
Indicators

• How well do indicators used address key aspects
  of the problem?
• Alternative indicators?
• Limitations of indicators used?
• Controversies in science and society about these
  indicators?

19
Adequacy ofavailable knowledge base?

• What are strong and weak points in the
  knowledgebase?
• Use of proxies, empirical basis, theoretical
  understanding, methodological rigor, validation
• NUSAP Pedigree analysis
• What parts of the knowledge are contested
  (scientific and societal controversies)?
• Is the assessment feasible in view of available
  resources? (limitations implied)

20
Example Pedigree matrix parameter strength
21
Example Pedigree results
Trafic-light analogy lt1.4 red 1.4-2.6 amber
gt2.6 green
This example is the case of VOC emissions from
paint in the Netherlands, calculated from
national sales statistics (NS) in 5 sectors
(Ship, Building Steel, Do It Yourself, Car
refinishing and Industry) and assumptions on
additional thinner use (Th) and a lump sum for
imported paint and an assumption for its VOC
percentage. See full research report on
www.nusap.net for details.
22
NUSAP Diagnostic Diagram
high
Danger zone
Combine results from sensitivity analysis and
pedigree analysis by plotting each model
parameter in this diagram. This reveals the
parameters whose uncertainty is most
problematic.
Sensitivity
Safe zone
low
weak
strong
Pedigree
23
NUSAP Diagnostic Diagram
VOC imp.paint
Example result from the case of emission
monitoring of VOC from paint in NL. The most
problematic parameter is the assumed VOC
percentage in imported paint.
Thin Ind
NS Decor
Overlap VVVF/CBS imp
NS Ind
Imp. Paint
Imp. Below threshold
NS DIY
NS Car
Thin. DIY-rest
Thin. Car
Gap VVVF-RNS
NS Ship
Th. decor
24
Mapping and prioritization of relevant
uncertainties

• Highlight uncertainties in typology relevant to
  this problem
• Set priorities for uncertainty assessment
• Select uncertainty assessment tools from the tool
  catalogue

25
Typology of uncertainties

• Location
• Level of uncertainty
• statistical uncertainty, scenario uncertainty,
  recognised ignorance
• Nature of uncertainty
• knowledge-related uncertainty,
  variability-related uncertainty
• Qualification of knowledge base (backing)
• weak, fair, strong
• Value-ladenness of choices
• small, medium, large

26
Locations of uncertainties

• Context
• ecological, technological, economic, social and
  political representation
• Expert judgement
• narratives, storylines, advices
• Model
• model structure, technical model, model
  parameters, model inputs
• Data
• measurements, monitoring data, survey data
• Outputs
• indicators, statements

27
(No Transcript)
28
Tool catalogue

• For each tool
• Description
• Goals and use
• What sorts and locations of uncertainty does this
  tool address?
• What resources are required to use it?
• Strengths and limitations
• guidance on application complementarity
• Typical pitfalls of each tool
• References to handbooks, example case studies,
  web-sites, experts etc.

29
Tool catalogue

• Sensitivity Analysis
• Error propagation
• Monte Carlo
• NUSAP
• Expert Elicitation
• Scenario analysis
• PRIMA
• Checklist model quality assistance
• Assumption analysis
• ...

30
Reporting

• Make uncertainties explicit
• Assess robustness of results
• Discuss implications of uncertainty findings for
  different settings of burden of proof
• Relevance of results to the problem
• Progressive disclosure of information -gt
  traceability and backing

31
IPCC WGI Proposal for Interpretation and Use of
Probabilistic Terms
32
Weiss 2003 uncertainty scale

• 10. Beyond any doubt
• 9. Beyond a reasonable doubt
• 8. Clear and Convincing Evidence
• 7. Clear Showing
• 6. Substantial and credible evidence
• 5. Preponderance of the Evidence
• 4. Clear indication
• 3. Probable cause reasonable grounds for belief
• 2. Reasonable, articulable grounds for suspicion
• 1. No reasonable grounds for suspicion
• 0. Impossible

33
Triggers that increase policy relevance of
uncertainty

• Close to a norm or target
• Near a threshold of severe impact
• On steep part of cost-curve or impact curve
• Low pedigree
• High (potential) valueladenness
• temperatute of scientific or societal
  controversies

34

• More information
• www.nusap.net

35

• References
• Funtowicz, S.O. and J.R. Ravetz, Uncertainty and
  Quality in Science for Policy. Kluwer, 229 pp.,
  Dordrecht, 1990.
• James Risbey, J.P. van der Sluijs and J. Ravetz,
  Quality assistance in environmental modelling,
  Environmental Modelling and Assessment,
  forthcoming
• Van der Sluijs, Jeroen P., A way out of the
  credibility crisis of models used in integrated
  environmental assessment, Futures, Vol. 34, 2002,
  pp. 133-146.
• Jeroen van der Sluijs, Matthieu Craye, Silvio
  Funtowicz, Penny Kloprogge, Jerry Ravetz, and
  James Risbey Combining Quantitative and
  Qualitative Measures of Uncertainty in Model
  based Environmental Assessment the NUSAP System,
  Risk Analysis, forthcoming
• Walker, W., Harremoës, P., Rotmans, J., Van der
  Sluijs, J., Van Asselt, M., Jansen, P., Krayer
  von Krauss, M.P., 2003. Defining Uncertainty A
  Conceptual Basis for Uncertainty Management in
  Model-Based Decision Support. Journal of
  Integrated Assessment, 4 (1) 5-17.
• Weiss, C., Scientific Uncertainty and Science
  Based Precaution. International Environmental
  Agreements Politics, Law and Economics 3, 2003,
  p.137-166
• Weiss, C., Expressing Scientific Uncertainty.
  Law, probability and risk, 2003, 2, p. 25-46
• The four volumes of the RIVM uncertainty
  guidance
• J.P. van der Sluijs, J.S. Risbey, P. Kloprogge,
  J.R. Ravetz, S.O. Funtowicz, S.Corral Quintana, Â
  Guimarães Pereira, B. De Marchi, A.C. Petersen,
  P.H.M. Janssen, R. Hoppe, and S.W.F. Huijs.
  RIVM/MNP Guidance for Uncertainty Assessment and
  Communication Detailed Guidance Utrecht
  University RIVM, 2003 (available from
  www.nusap.net).
• A.C. Petersen, P.H.M. Janssen, J.P. van der
  Sluijs, J.S. Risbey, J.R. Ravetz RIVM/MNP
  Guidance for Uncertainty Assessment and
  Communication Mini-Checklist Quickscan
  Questionairre.. RIVM/MNP ISBN 90-6960-105-1,
  2003 (available from www.nusap.net).
• P.H.M. Janssen, A.C. Petersen, J.P. van der
  Sluijs, J.S. Risbey, J.R. Ravetz . RIVM/MNP
  Guidance for Uncertainty Assessment and
  Communication Quickscan Hints Actions List.
  RIVM/MNP, ISBN 90-6960-105-2, 2003 (available
  from www.nusap.net).
• J.P. van der Sluijs, P.H.M. Janssen, A.C.
  Petersen, P. Kloprogge, J.S. Risbey, W. Tuinstra,
  M.B.A. van Asselt, J.R. Ravetz, RIVM/MNP Guidance
  for Uncertainty Assessment and Communication
  Tool Catalogue for Uncertainty Assessment Utrecht
  University RIVM, , 2004 (available from
  www.nusap.net).

Most refs available on request as pdf file, just
mail me at j.p.vandersluijs_at_chem.uu.nl