Quantitative methods to manage uncertainty in science by

1
Quantitative methods to manage uncertainty in
science by Andrea Saltelli, Stefano Tarantola
and Michela Saisana, Joint Research Centre of the
European Communities in Ispra (I),
Andrea.Saltelli_at_jrc.it
Mini-symposium The management of uncertainty in
risk science and policy, World Congress on
Risk Brussels, 22-25 June 2003.
2
Models mimic systems
Rosens formalisation of the modelling process
3
Models mimic systems (Rosen)
World (the natural system) and Model (the
formal system) are internally entailed - driven
by a causal structure. Nothing entails with one
another World and Model the association is
hence the result of a craftsmanship. But this
does not apply to natural systems only give 10
engineers the blueprint of the same plant and
they will return you 10 model based risk
assessments for the same plant.
4
Models mimic systems (Rosen)
It can help the craftsman that the uncertainty in
the information provided by the model (the
substance of use for the decoding exercise) is
carefully apportioned to the uncertainty
associated with the encoding process.
5
Models maps assumptions onto inferences ... but
often too narrowly
ltlt most simulation models will be complex,
with many parameters, state-variables and non
linear relations. Under the best circumstances,
such models have many degrees of freedom and,
with judicious fiddling, can be made to produce
virtually any desired behaviour, often with both
plausible structure and parameter values.gtgt,
HORNBERGER and Spear (1981) ltltCynics say that
models can be made to conclude anything provided
that suitable assumptions are fed into them.gtgt,
The Economist, 1998. KONIKOV and Bredehoeft,
1992 ? Oreskes et al. 1994.
6
Use of models in the scientific discourse
But yet models are used ... and a
legitimate question is the following If we
had mapped the space of uncertain assumptions
honestly and judiciously, would the space of
inference still be of use1? 1Read do we still
have peak around some useful inference (e.g. YES
or NO, safe or unsafe, hypothesis accepted or
rejected, policy effective or ineffective etc. )
or do we have as many YES as NO etc.?
7
Models maps assumptions onto inferences
ltltI have proposed a form of organised sensitivity
analysis that I call global sensitivity
analysis in which a neighborhood of alternative
assumptions is selected and the corresponding
interval of inferences is identified. Conclusions
are judged to be sturdy only if the neighborhood
of assumptions is wide enough to be credible and
the corresponding interval of inferences is
narrow enough to be useful.gtgt Leamer,
Sensitivity Analysis would help, 1990
8
Models maps assumptions onto inferences
Leamers view of global Sensitivity Analysis
(SA)
...
Other assumptions
9
Models maps assumptions onto inferences
(Parametric bootstrap version of UA/SA )
(?Estimation)
(?Parametric bootstrap we sample from the
posterior parameter probability)
Uncertainty and sensitivity analysis
10
Bootstrapping-of-the-modelling-process version
of UA/SA, after Chatfield, 1995
(?Model Identification)
(?Estimation)
(?Bootstrap of the modelling process)
11
Bayesian Uncertainty and Sensitivity
Analysis (Draper 1995, Planas and Depoutot 2000)
(?Sampling)
Inference
Posterior of Parameters
12
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
The space of the model induced choices (the
inference) swells and shrinks by our swelling and
shrinking the space of the input assumptions. How
many of the assumptions are relevant at all for
the choice? And those that are relevant, how do
they act on the outcome singularly or in more or
less complex combinations?
13
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
I desire to have a given degree of robustness
in the choice, what factor/assumptions should be
tested more rigorously? (gt look at how much
fixing any given f/a can potentially reduce the
variance of the output) Can I confidently fix
a subset of the input factors/assumptions? The
Beck and Ravetz relevance issue. How do I find
these factors?
14
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
Reduced variance
Expected reduced variance it is small if the
factor is important.
15
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
Big if factor important
Small if factor important
First order effect
16
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
Also used is the total effect term This is the
expected fractional value of the variance that
would be left if all factor but Xi were fixed.
The use of different sensitivity measures
should be seen as the answer to a rigorous
question concerning the relative importance of
input factors.
17
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
One can thus relate the total effect term to a
question relative to the possibility to fix
factor(s), (Factor Fixing Setting), while the
first order effect frames into the Factors
Prioritisation Setting.
18
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
Other setting (questions) can easily be imagined.
Settings to frame the uncertaitny and sensitivity
analyses are crucial. The alternative would be
to have different SA methods suggesting
different factors relative imortance. Settings
should be audited! Let us agree on what
importance means before we engage in the
analysis.
19
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
Is the model-induced choice weak (non robust)
because there is an insufficient number of
observations, or because the experts cannot agree
on an accepted theory?
20
Useful inference versus falsification of the
analysis
Example imagine the inference is Y the
logarithm of the ratio between the two
pressure-on-decision indices (Tarantola et als.
2000).
Region where Region where Incineration
Landfill is preferred is
preferred
Frequency of occurrence
YLog(PI 1/PI 2)
21
Useful inference versus falsification of the
analysis
22
Use of models in the scientific discourse and
role of uncertainty - sensitivity analysis
What happens if I address the space of the
policy options?
23
Gauging the leverage of the policy options
latitude
24
Conclusions

• The output from global uncertainty and
  sensitivity analyses can feed back into the
  extended peer review process via e.g.
• - refocusing of the critical issues/factor,
• - (re-assignment of weights for multiple
  criteria, or)
• - inference falsification
• identification of policy relevance/ irrelevance
• Note EC Guidelines for Extended Impact
  Assessment inlcude explicit and detailed
  indication for global SA!

25
References
ROSEN R., Life Itself - A Comprehensive Inquiry
into Nature, Origin, and Fabrication of Life.
Columbia University Press 1991. HORNBERGER G.M.,
and R. C. Spear (1981) An approach to the
preliminary analysis of environmental systems.
Journal of Environmental management, 12,
7-18. KONIKOV and Bredehoeft, 1992, "Groundwater
models cannot be validated" Advances in Water
Resources 15(1), 75-83. ORESKES, N. ,
Shrader-Frechette K., Belitz, K., 1994,
Verification, Validation, and Confirmation of
Numerical Models in the Earth Sciences, SCIENCE,
263, 641-646 Edward E. Leamer, Sensitivity
Analysis would help, in Modelling Economic
Series, Edited by CWJ Granger, 1990, Clarendon
Press, Oxford. CHATFIELD C., Model uncertainty ,
data mining and statistical inference, J. R.
Statist. Soc. A, 158 (3) , 419-466, 1993
26
Further reading on SA
Papers - Saltelli et als., Statistical Science,
2000 Saltelli and Tarantola, JASA, 2002 Book
- Saltelli et al. Eds., Sensitivity Analysis,
2000, John Wiley Sons publishers, Probability
and Statistics series Book - A primer
(Sensitivity Analysis in Practice) will appear by
end 2003, with Wiley. A forum -
http//sensitivity-analysis.jrc.cec.eu.int/ Prese
ntations of the mini-symposium on www.nusap.net