RIVMMNP Guidance for Uncertainty Assessment and communication

1
RIVM/MNP Guidance for Uncertainty Assessment and
communication

• Jeroen van der Sluijs, James Risbey, Penny
  Kloprogge
• (Copernicus Institute, Utrecht)
• Jerry Ravetz (RMC, London)
• Silvio Funtowicz, Serafin Corral, Ângela Pereira
  (JRC, Ispra)
• Bruna de Marchi
• Rob Hoppe, Simone Huijs (Fac. Public Policy,
  Twente Univ.)
• Marjolein van Asselt (ICIS, Maastricht)
• Peter Jansen, Arthur Petersen, Anton van der
  Giessen (RIVM)

2
RIVM / De Kwaadsteniet (1999)

• RIVM over-exact prognoses based on virtual
  reality of computer models
• Newspaper headlines
• Environmental institute lies and deceits
• Fuss in parliament after criticism on
  environmental numbers
• The bankruptcy of the environmental numbers
• Society has a right on fair information, RIVM
  does not provide it

3
RIVM learning process

• lt1999 Innovative methodological RDD on
  uncertainty assessment and management (e.g
  TARGETS)
• 1999 De Kwaadsteniet affair
• ?Fact sheets
• 1999-2000 National review
• 2000 International audit
• gt2000 Multi-disciplinary project
• 20012002 Development of Guidance

4
Insights on uncertainty

• Uncertainty is partly socially constructed and
  its assessment always involves subjective
  judgement
• Omitting uncertainty management can lead to
  scandals, crisis and misunderstandings
• More research does not necessarily reduce
  uncertainty
• High quality ? low uncertainty
• Shift in focus needed from reducing uncertainty
  towards a systematic management of uncertainty
• Uncertainty is a multi-dimensional concept and
  can manifest itself at different locations

5
Goals

• Structured and transparent approach that
  facilitates awareness, identification, and
  incorporation of uncertainty
• May not reduce uncertainties, but provides a
  means to assess their potential consequences and
  avoid pitfalls associated with ignoring or
  ignorance of uncertainties
• Guidance for use and help against misuse of
  uncertainty tools
• Provide useful uncertainty assessments (robust
  knowledge)
• Facilitate effective communication on
  uncertainties in terms of robustness of knowledge
• Fit RIVM's specific role in the decision analytic
  cycle

6
Project phases

• 25 October 2001 Expert workshop
• 22 November 2001 User workshop
• Expert review (2002)
• User review (2002)
• Implementation (2003)

7
Guidance components
8
Mini check

• Problem framing
• Involvement of stakeholders
• Selection of indicators
• Appraisal of knowledge base
• Mapping key uncertainties
• Reporting

9
Main steps

• Minicheck/Quick scan
• Problem framing
• Communication dialogue
• Process/context assessment (history,
  stakeholders, values)
• (Assess limitations of) Environmental assessment
  methods
• Uncertainty identification and prioritization
• Uncertainty analysis
• Review, evaluation, interpretation
• Reporting

10
Uncertainty typology

• Level of uncertainty
• Statistical uncertainty (range / pdf)
• Scenario uncertainty (what-if)
• Recognized ignorance
• Nature of uncertainty
• Knowledge related
• Variability related
• Qualification of knowledge base (NUSAP/Pedigree)
• valueladenness of choices and assumptions

11
Locations of uncertainty

• Context
• Expert judgment
• Model
• structure
• technical (computer code algorithms)
• parameters
• inputs
• Data
• ? Outputs

12
(No Transcript)
13
Outputs Quickscan (1)

• Description of the problem
• Gauge of how well assessment tools address the
  problem
• List of which uncertainties are salient on the
  basis of problem structure
• Indication whether to involve stakeholders
• Indication where in policy life cycle the problem
  is
• List of stakeholders
• Identification of areas of agreement/disagreement
  on value dimensions

14
Outputs Quickscan (2)

• Prioritized list of salient uncertainties
• Communication plan when and how to involve what
  stakeholders
• First indication of appropriate tools to address
  uncertainties identified
• Assessment of attainable robustness of results
  indication what it might take to increase
  robustness
• Assessment of the relevance of results to the
  problem
• Pitfalls and hints to facilitate effective
  communication of results

15
Toolbox uncertainty assessment

• Sensitivity Analysis (screening, local global)
• Error propagation equation (TIER 1)
• Monte Carlo (TIER 2)
• NUSAP
• Expert Elicitation
• Scenario analysis
• Extended Quality Assurance (pedigree scheme)
• Perspective based approaches (cultural theory)
• Checklist model quality assistance
• (see www.nusap.net)
• ....

16
Toolbox

• For each tool
• Main purpose and use
• Sorts and locations of uncertainty addressed
• Required resources
• Strengths and limitations
• Guidance on application
• Hints on complementarity with other tools
• Typical pitfalls
• Key references (handbooks, user-guides, web
  resources, example studies, experts)

17
Review, synthesis and evaluation

• Synthesise quantitative qualitative results
• Revisit problem and assessment steps
• Frame findings in terms of robustness of the
  environmental assessment concerned
• Relevance of results to the problem
• Discuss implications of findings for different
  settings of burden of proof

18
Reporting

• Context of communication
• Who are target audiences
• Language
• Method and style
• Content

19

• More information
• www.nusap.net