Francesca Campolongo

1
Screening methods in sensitivity analysis
Francesca Campolongo
Summer School on Sensitivity Analysis of Model
Output U n i v e r s i t y o f V e n i c e, J u n
e 12 - 15, 2002
2
Screening methods
Goal Select subset of most influential factors
How design an experiment, evaluate the model,
analyze results
Computational cost low
Type of information qualitative
Uses model verification, model simplification
3
Some screening methods...
Sequential Bifurcation, Bettonvil (1990)
Iterated Fractional Factorial Design, Andres
Hajas (1993)
The Morris method, (1991)
4
Sequential bifurcation
SB is a group-screening method that assumes known
signs
SB is sequential combinations to be run are
selected as results are obtained
Bifurcation each group containing an important
factor is split into 2
5
Sequential bifurcation
A factor is low when gives low response, it is
high when gives high response
6
Sequential bifurcation
Assume 128 factors. Important factors are 68th,
113th, 120th
Stage 1 Evaluate Y0 and Y128
Important factors ?
Y0 lt Y128
Y64 ? Y0 Y64 ? Y128
Stage 2 Evaluate Y64
Eliminate first 64 factors, search the second half
7
Sequential bifurcation
Stage 2 Bifurcate the second group...
Final stage Important factors are identified
Main effects are estimated
Cost N lt k
8
Iterated Fractional factorial Design
IFFD is a based on grouping input factors
Each factor may assume 3 values (levels) L, M, H
IFFD is a composite design multiple FFD
Cost N lt k
Performs well when few influential factors !
9
Iterated Fractional factorial Design
Assume 1 important factor F.
Group factors. Find group A1 containing F
Regroup factors. Find group A2 containing F. F
lies in A1? A2.
Iterate the above steps. After M iterations F
lies in A1? A2 ? .?AM
10
Iterated Fractional factorial Design
11
The Morris method
Strategy to vary One-factor-At-a-Time holding
constant the others
Information qualitative, ranking
Main effect the average of r local sensitivity
measures (derivatives) computed at different
points of the input factors space
Cost O(k), k number of input factors
12
The method of Morris
Elementary Effect for the ith input factor
13
The method of Morris
The EEi is a local measure of sensitivity for
factor ith that depends on the point X at which
it is computed
The idea is to compute a number r of EEi EE1i
EE2i . EEri at r different points X1 , ,
Xr, and then take the average
14
The method of Morris
Assume there are 2 input factors x1 and x2
15
The method of Morris
Assume for each factor i, i 1, .., k we have
computed a number r of elementary effects EE1i
EE2i . EEri
Average of EEis ? main effect ? St. Dev.
of EEis ? sum of inter. ?
Warning the method estimates the sum of all
interactions involving a factor but ignores the
single two-factor interactions
16
The method of Morris
Simplest sampling strategy ? Cost is 2 r k
Morris sampling strategy ? Cost is r ( k
1)
17
The method of Morris
Assume there are 3 input factors x1 , x2 , and
x3
Y4
A trajectory of the Morris design
Y3
Y1
Y2
18
The method of Morris
19
Morris in practice
Select input factors and their ranges of
variation (distribution function)
Select the function of interest
Select the Morris sample size r (? comp. cost.)
Select l number of levels at which factors are
sampled (l ?? ? l / 2 (l -1) )
20
Morris in practice
The starting point X0 (x10 , x20 , ... ,xk0)
xi0 sampled in 0, 1/(l -1), 2/(l-1), , 1
with xi0 ? 1 i1,..,k
Example. X is U(0,1) l 4 ? ? 2/3 l1 0 l2
1/3 l3 2/3 l4 1
When factor distribution are not uniform,
distribution quantiles are sampled
21
An analytical example
Assume 20 inputs w1, w2,..., w20 in -1,1
Remainder coefficients are generated from N(0,1)
22
An analytical example
23
A fish population model
Stage based modeling
Assume a single specie evolving through time.
Assume 13 life stages Larvae (1-4), Juvenile
(5-9), Adult (10-13)
n t q-vector of the population stages at time t
n t1 A n t
24
A fish population model
For each life stage i, i 1,2, ..., q
Pi probability of surviving and staying in
stage i
Gi probability of growing into the next stage
mi maternity per fish per unit time (d)
25
A fish population model
A 3-species stage-based model Isardines,
Janchovies, Kmackerels
Each population is represented by its discrete
matrix transition A
Density-dependent competition between different
life-stages, i.e. larvae, juveniles, etc, is
modelled by a matrix of interactions
The 3-species model is represented by a block
matrix.
26
A fish population model
27
A fish population model
28
A fish population model
Function of interest the annual population
growth rate
Population growth rate is ?, the dominant
eigenvalue of population matrix.
Factors base values are such that ? 1
(equilibrium)
We study ?365
29
Morris on the fish model
Objective function ? 365
Input factors 103, uniformly distributed
A screening is necessary !
Morris parameters r 10, l 4 , ? 2/3
Comp. cost N 1040
30
Morris on the fish model
31
Morris on the fish model
The rankings provided respectively by ? and ? is
almost identical.
Reason the majority of the models encountered in
natural sciences are highly non-linear.
A factor that is important in the model (high ?)
is usually also involved in curvatures or
interaction effects (high ?).
32
Morris on the fish model
What did we learn?
Inter-specific competition parameters are not
very important ? call for model simplification
Fecundity factors are also not very significant
for any of the 3 species at any life stage
Parameters of early life-stages have the greatest
effect on population growth
Parameters related to sardines are NOT amongst
the most important ones ? model revision?
33
When to use what
34
References
Andres, T. H., and Hajas, W. C. (1993). Using
Iterated Fractional Factorial Design to Screen
Parameters in Sensitivity Analysis of a
Probabilistic Risk Assessment Model. Proceedings
of the Conf. on Math. Methods and Supercomputing
in Nuclear Applications, Karlsruhe, Germany,
April 1993, Ed. H. Küsters, E. Stein, and W.
Werner, 2 328-337. Andres, T.H. (1997). Sampling
Methods and Sensitivity Analysis for Large
Parameter Sets. Journal of Statist. Comput.
57(1-4), 77-110. Bettonvil, B. (1990). Detection
of important factors by sequential bifurcation.
Tilburg University Press, Tilburg. Bettonvil. B.,
and Kleijnen, J.P.C. (1997). Searching for
important factors in simulation models with many
factors sequential bifurcation. European Journal
of Operational Research, 96(1) 180-194. Morris,
M. D. (1991). Factorial sampling plans for
preliminary computational experiments.
Technometrics, 33(2) 161-174.