Model Predictive Uncertainty

1
Model Predictive Uncertainty
2
Sensitivity analysis ..
3
(No Transcript)
4
GM Seam Inflows
5
Permian Inflows
6
Tertiary Sands Inflow
7
Hydraulic property heterogeneity
correlation length
8
Hydraulic property correlation decreases with
distance
correlation length
C(K1 , K2 )
distance
9
Hydraulic property correlation decreases with
distance
correlation length
variogram
C(K1 , K2 )
distance
10
Hydraulic property measurement points
11
Hydraulic property correlation decreases with
distance
correlation length
variogram
C(K1 , K2 )
distance
12
Hydraulic property realisation
13
Hydraulic property realisation
14
constrained by point measurements
15
constrained by point measurements
16
Particularly useful in pathline analysis ...
17
...
18
Sensitivity analysis calibrated model ..
19
Estimated parameter values
p2
Objective function minimum
p1
20
Estimated parameter values
Objective function minimum
p2
Maximum probability for p1 and p2
p1
21
Estimated parameter values
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
22
Estimated parameter values
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
23
Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
24
Field or laboratory measurements and model
output-
Model output
value
calibration dataset
prediction
q2
q1
q3
etc
distance or time
25
Field or laboratory measurements and model
output-
Model output
value
Lower predictive limit
calibration dataset
q2
q1
q3
etc
distance or time
26
Field or laboratory measurements and model
output-
Model output
value
Upper predictive limit
calibration dataset
q2
q1
q3
etc
distance or time
27
Field or laboratory measurements and model
output-
Model output
value
Confidence interval for prediction
calibration dataset
q2
q1
q3
etc
distance or time
28
Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
29
Estimated parameter values nonlinear case
knowledge constraints
p2
Allowed parameter values
p1
30
A certain model prediction
p2
Increasing value
p1
31
Defining a confidence interval
p2
The critical points
p1
32
Residuals
Model output
value
calibration dataset
prediction
q2
q1
q3
etc
distance or time
33
The variance of the residuals is- ?2
? / (m - n)
m number of observations n number of
parameters
34
Field or laboratory measurements and model
output-
Model output
value
Confidence interval for prediction
calibration dataset
q2
q1
q3
etc
distance or time
35
Field or laboratory measurements and model
output-
Model output
Predictive uncertainty interval
value
calibration dataset
q2
q1
q3
etc
distance or time
36
Software for predictive uncertainty analysis

• UCODE
• assumes model linearity
• only works with a few parameters
• PEST
• full nonlinear predictive analysis
• unlimited number of parameters

37
For linear models
38
Estimated parameter values-
p2
Extreme values of p1 and p2
p1
39
A simple lumped parameter model
40
A simple lumped parameter model
par1 par2
par5 par6
par3 par4
41
The covariance matrix of the estimated parameter
set is given by C(p) ?2 (Mt QM)-1
For a nonlinear model replace M by J, the
Jacobian matrix. C(p) ?2 (Jt QJ)-1
42
Let M (ie. green M) represent the action of the
model in predictive mode and o the model outputs
in predictive mode. Then C(o) M C(p)Mt
For a nonlinear model- C(o) J C(p)Jt
Notice that predictions can be correlated.
43
probability
value of prediction 1
44
Maximum probability
Bivariate probability density function.
45
For linear and nonlinear models
46
PESTs predictive analyzer
p2
Initial parameter estimates
The critical point
p1
47
PESTs predictive analyzer
p2
The critical point
Initial parameter estimates
p1
48
Major problem with this approach

• assumes that there is an objective function
  minimum
• assumes that this defines a unique set of
  parameters
• thus it assumes that parameters are lumped and
  that there arent many of them

49
objective function contours
50
A Confined Aquifer
head
Fixed Inflow
T1
T3
T2
Fixed head
51
objective function contours
52
Hillside and Piezometers
53
Transmissivity distribution - I
100 m2/day
54
Transmissivity distribution - II
12 m2/day
360 m2/day
55
SNOW section of the PERLND module of HSPF
56
PWATER section of the PERLND module of HSPF
57
PWATER section of the PERLND module of
HSPF (continued)
58
Daily Flow
59
Monthly Volume
60
Exceedence fraction
61
In cases such as these (and most real-world cases
are the same), quantification of predictive
uncertainty must take place by other means.
62
The linear method of confidence interval
estimation is impossible to apply because
parameter covariance matrices are singular and
the mathematics breaks down.
63
PESTs ability to maximise/minimise a prediction
while still maintaining calibration constraints
can still be applied. This is an excellent method
to test the wiggle room of a prediction.
However quantification of uncertainty limits
becomes mathematically more difficult.
64
Reality
Exit time 3256 Exit point 206
65
Calibration to 12 observations (no noise)
Exit time 7122 true3256 Exit point 241
true206
66
Use predictive analysis to minimize travel time
K ranges from 1.9e-5 to 8813 m/day
Exit time 280 true3256 Exit point 226
true206
67
Mathematically, because of parameter
nonuniqueness, the parameter and predictive
uncertainty limits are huge. Limits will then be
set by parameter plausibility and parameter
relationships plausibility, rather than simply by
model-decalibration considerations.
68
Special calibration-constrained monte carlo
methods are often used (see later).
69
Another major problem is this.
70
Field or laboratory measurements and model
output-
Model output
Predictive uncertainty interval
value
calibration dataset
q2
q1
q3
etc
distance or time
71
We know the residual variance when our
predictions are of the same type as the data
against which we calibrate. But what if we are
making a prediction of a different type or at a
different place? That is why we are using a
physically based model in the first case. The
residual noise can be enormous (remember our
travel time example from earlier lecture).
72
Some examples
73
A surface water modelling example
74
(No Transcript)
75
Observed and modelled flows over part of
calibration period.
76
Observed and modelled monthly volumes
77
Observed and modelled exceedence fractions
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
(No Transcript)
83
(No Transcript)
84
(No Transcript)
85
Points joined only as a method of enhancing
visual comparison of field measurements with
model outputs
86
Observations used in calibration process

• Individual TSS values
• mean TSS (observed and modelled after
  interpolation)
• std dev of TSS (observed and modelled after
  interpolation
• bed sand, silt and clay unchanged over
  calibration period

87
Points joined only as a method of enhancing
visual comparison of field measurements with
model outputs
88
Measured and model-generated TSS
time-interpolated to measurement times
89
Measured and model-generated TSS
90
Measured and model-generated TSS
91
Aim of predictive analysis
Maximise/minimise a key model prediction while
maintaining the model in a calibrated state. In
our case we will maximise/minimise the total
sediment mass carried by the creek over the whole
calibration period of 1970 to 1995.
92
Calibration limits
Objective function under calibration conditions
4.2E4 Model deemed to be uncalibrated when
objective function 4.8E4. So very tight
calibration constraints.
93
Maximum/minimum sediment mass
Maximum sediment mass 1.5E6 tonnes Minimum
sediment mass 7.9E5 tonnes
94
Measured and model-generated TSS
Minimum phi
95
Measured and model-generated TSS
Minimum phi
Minimized prediction
96
Measured and model-generated TSS
Minimum phi
Minimized prediction
97
a groundwater modelling example
98
A hillside
99
A hillside and finite difference grid
100
Observation points
101
Transmissivity field- average100m2/day exponenti
al variogram X-correlation length twice
Y-correlation length Recharge
100mm/yr Concentration of Effluent 100
units Effective porosity 5 Long. dispersivity
10m Trans dispersivity 1m
102
Log transmissivity distribution average
100m2/day green is higher
103
Piezometric surface
Contour interval 5m
104
Solute concentration after 3 years leakage
Maximum concentration 20 units
105
Remediation strategy interception by pumping
106
Solute concentration after 3 years leakage
Maximum concentration 20 units
Well pumps at 600m3/day
107
Solute concentration after 30 years remediation
Maximum concentration 2.6 units /m3
Contaminant outflow to river is 83000 units over
30 years
Well pumps at 600m3/day
108
However in real life we do not know what is in
the ground. So we must calibrate a model.
109
What is the worst-case scenario ie. how
ineffective could pumping be while ensuring that
our model is still calibrated?
110
Parameterisation using pilot points
Pilot point
Observation bore
111
Observed and model-generated water levels at bores
Contour interval 2m
112
Concentration residuals
113
Solute concentration after 30 years remediation
Maximum concentration 1.6 units /m3
Contaminant outflow to river is 8.0?106 units
over 30 years
Well pumps at 600m3/day
114
Log transmissivity distribution green is higher
Contaminant outflow to river is 8.0?106 units
over 30 years
Well pumps at 600m3/day
115
What is the best-case scenario ie. how effective
could pumping be?
Make sure that the model is still calibrated.
116
Observed and model-generated water levels at bores
Contour interval 2m
117
Concentration residuals
118
Solute concentration after 10 years remediation
Maximum concentration 5.9 units (minimum
contour at .01 units)
Contaminant inflow to creek is 14400 units over
30 years
Well pumps at 600m3/day
119
Log transmissivity distribution green is higher
Contaminant inflow to creek is 14400 units over
30 years
Well pumps at 600m3/day
120
Fail-safe Remediation

• Plot worst-case river pollution against pumping
  rate
• Choose minimum pumping rate that meets regulatory
  requirements

121
Worst-case outflow
Results from individual predictive analysis runs
122
Solute concentration after 30 years of pumping
Add a second bore
123
Worst-case outflow
One extraction bore
Two extraction bores
124
This method of predictive uncertainty analysis
has the advantage that it is computationally
reasonably efficient. However in some cases
predictive uncertainty limits could have been
narrower if known or suspected parameter
relationships are enforced. However this is very
difficult where these relationships are
smoothed or modified by the regularisation
process which is necessary to achieve
calibration. This is an area of active research.
125
Calibration-constrained Monte-Carlo ..
126
GLUE Method (Lancaster University)
127
Estimated parameter values nonlinear case
Allowed parameter values
p2
Maximum probability for p1 and p2
p1
128
Estimated parameter values nonlinear case
p2
Allowed parameter values based on linear
approximation
p1
129
Advantages-

• does not rely on linearity assumption
• sometimes predictive probabilities can be
  estimated
• robust

Disadvantages-

• Only a few parameters can be examined so model
  may be over-simplified
• Hit to miss ratio probably extremely low
• Extremely computationally expensive

130
Bayes Theorem
p(?,?yi) ? p(yi ?,?) p(?,?)
131
Bayes Theorem
p(?,?yi) ? p(yi ?,?) p(?,?)
132
Markov-Chain Monte Carlo
p2
Allowed parameter values based on linear
approximation
p1
133
Advantages-

• two orders of magnitude more efficient than GLUE
• Provides posterior parameter probability
  distribution (from which predictive distribtution
  can be derived)
• robust

Disadvantages-

• Only a few parameters can be examined so model
  may be over-simplified
• Computationally very expensive

134
Warping
135
A model grid
136
The model domain
Fixed head
Recharge 1.0e-4 m/day
Transmissivity 100 m2/day
137
Calculated heads
138
Observation bore locations
139
Pilot point locations
140
Methodology

• Generate a random field (rather, the log of a
  random field).
• Multiply random field by field interpolated from
  pilot points (use regularisation to make that
  field as smooth as possible).
• Use multiplied field in model.
• Estimate field multipliers at pilot points
  through calibration process (ie.estimate field
  multipliers so as to minimise head residuals
  calculated using multiplied field.)

Use of PESTs regularisation mode essential
141
Generated random field
Calibrated multiplier field
Field used by model
142
Generated random field
Calibrated multiplier field
Field used by model
143
Calibration Process

• Measurement Objective function
• comprised of differences between measured and
  calculated heads at bores
• maximum permitted measurement objective function
  supplied by user

• Regularisation Objective function
• rises with heterogeneity of multipliers at pilot
  points
• weights assigned to homogeneity constraints
  determined geostatistically
• regularisation process maximises homogeneity of
  multiplier field

144
The following fields all calibrate the model with
an rms error of 0.12m (ie. 12 cm).
145
Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 667 m2/day
146
Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 617 m2/day
147
Calibrated transmissivity field
Transmissivity ranges from 15 m2/day to 994 m2/day
148
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 815 m2/day
149
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 471 m2/day
150
Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 643 m2/day
151
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 524 m2/day
152
Calibrated transmissivity field
Transmissivity ranges from 13 m2/day to 572 m2/day
153
Calibrated transmissivity field
Transmissivity ranges from 10 m2/day to 488 m2/day
154
Calibrated transmissivity field
Transmissivity ranges from 11 m2/day to 1809
m2/day
155
Calibrated transmissivity field
Transmissivity ranges from 7 m2/day to 722 m2/day
156
Study area
157
Model domain
158
Observed heads
159
Model grid
160
Pilot points
161
rms 2in
The heterogeneity that MUST exist.
162
rms 3.5in
The heterogeneity that MUST exist.
163
rms 2.9in
The heterogeneity that MAY exist.
164
rms 2.9in
The heterogeneity that MAY exist.
165
rms 2.9in
The heterogeneity that MAY exist.
166
Generate field and warp them to enforce
calibration constraints
565
593
675
506
167
perform stochastic analysis
Exit points
168
evaluate probabilities.
Exit times
169
Objective function contours
p2
Objective function minimum
Initial parameter estimates
p1
170
Objective function contours
p2
Objective function minimum
Initial parameter estimates
p1
171
Objective function contours
p2
Objective function minimum
Final parameter estimates
p1
172
(No Transcript)
173
Observed and modelled flows over part of
calibration period.
174
Observed and modelled monthly volumes
175
Observed and modelled exceedence fractions
176
Parameter LZSN 2.0 UZSN
2.0 INFILT 0.0526 BASETP 0.200 AGWETP 0.00108 LZET
P 0.50 INTFW 10.0 IRC
0.677 AGWRC 0.983
177
Observed and modelled flows over part of
calibration period.
178
Observed and modelled monthly volumes
179
Observed and modelled exceedence fractions
180
Parameter Set 1 Set 2 Set 3 Set 4
Set 5 Set 6 LZSN 2.0 2.0
2.0 2.0 2.0 2.0 UZSN
2.0 1.79 2.0 2.0 1.76
2.0 INFILT 0.0526 0.0615 0.0783 0.0340
0.0678 0.0687 BASETP 0.200 0.182 0.199
0.115 0.179 0.200 AGWETP 0.00108 0.0186
0.0023 0.0124 0.0247 0.0407 LZETP 0.50
0.50 0.20 0.72 0.50
0.50 INTFW 10.0 3.076 1.00 4.48
4.78 2.73 IRC 0.677 0.571
0.729 0.738 0.759 0.320 AGWRC 0.983
0.981 0.972 0.986 0.981 0.966
181
Advantages-

• simple to implement
• provides posterior parameter probability
  distribution (from which predictive distribtution
  can be derived)
• can generate pre-warped parameter values
  according to known prior probability and
  correlation relationships
• robust

Disadvantages-

• can computationally expensive (but getting more
  efficient)
• Convenient but not quite theoretically correct

182
Some Conclusions
183
Some conclusions

• The potential for model predictive uncertainty is
  often very high especially when a model needs
  to predict something which is different from the
  data used in its calibration.
• In most cases there is no such thing as the
  model.
• Not only is the model just one of many models,
  but the regularisation that allows uniqueness to
  exist will probably introduce bias into
  predictions.
• Predictive uncertainty analysis should be an
  essential part of model deployment

184

• In view of this, the separation between the
  calibration and predictive process is artificial.
  A model is never calibrated.
• Calibration is simply the imposition of a set of
  constraints on parameter values when making a
  prediction, only use parameters which respect
  what we know about the system, and which allow
  the model to replicate past measurements
• In most cases (especially those involving
  complexity) the level of parameter nonuniqueness
  is still very high
• There is thus potential for high predictive
  uncertainty as well.