Calibration Guidelines

1
Calibration Guidelines
Model development
Model testing
9. Evaluate model fit 10. Evaluate optimal
parameter values 11. Identify new data to
improve parameter estimates 12. Identify new data
to improve predictions 13. Use deterministic
methods 14. Use statistical methods
1. Start simple, add complexity carefully 2. Use
a broad range of information 3. Be well-posed
be comprehensive 4. Include diverse observation
data for best fit 5. Use prior information
carefully 6. Assign weights that reflect
observation error 7. Encourage convergence by
making the model more accurate 8. Consider
alternative models
Potential new data
Prediction uncertainty
2
Model DevelopmentGuideline 7 Encourage
convergence of the regression by making the model
more accurate
3
Learn from Failed Regressions!

• Nonlinear regression can be difficult
• Even when composite scaled sensitivities and
  correlation coefficients indicate the data
  provide sufficient information to estimate the
  defined parameters, nonlinear regression may not
  converge.
• Substantial insight about the observations,
  potential model inaccuracies, and model fit can
  be obtained from values calculated in failed
  regressions
• dimensionless scaled sensitivities
• composite scaled sensitivities
• correlation coefficients
• influence statistics
• weighted and unweighted residuals
• parameter updates during the regression.

4
Make the model more accurate

• Working to make the model represent the system
  more accurately obviously is beneficial to model
  development, and generally also improves the
  behavior of the regression.
• Use the information from failed regressions, or
  regressions that converge to unrealistic
  parameter values, to guide changes.
• The most advantageous modifications for improving
  model accuracy are usually
• Modify the parameter definition
• Modify other aspects of model construction, such
  as its physical features or the processes it
  simulates
• Estimate fewer parameters
• Add observations to the regression
• Scrutinize existing observations for error in
  interpretation

5
The Main Problems that Plague Convergence

• Insensitivity
• Nonlinearity
• Inconsistencies

6
and Possible Solutions to Consider

• Insensitivity
• Specify parameters with small css
• Combine existing parameters
• Redesign the parameterization
• Creatively use system information (Guideline 2)
• Nonlinearity
• Evaluate results from intermediate iterations
• Evaluate large weighted residuals, observations
  omitted because simulated equivalents could not
  be obtained, whether parameter values are
  realistic
• If forward model nonlinearities are a suspected
  cause of nonconvergence, consider using a linear
  approximation
• Inconsistencies
• Check parameter representation, dominant
  observations
• Evaluate observations prior, and their
  simulated equivalents

7
Calibration Guidelines
Model development
Model testing
9. Evaluate model fit 10. Evaluate optimal
parameter values 11. Identify new data to
improve parameter estimates 12. Identify new data
to improve predictions 13. Use deterministic
methods 14. Use statistical methods
1. Start simple, add complexity carefully 2. Use
a broad range of information 3. Be well-posed
be comprehensive 4. Include diverse observation
data for best fit 5. Use prior information
carefully 6. Assign weights that reflect
observation error 7. Encourage convergence by
making the model more accurate 8. Consider
alternative models (MMA)
Potential new data
Prediction uncertainty
8
Guideline 8 Consider alternative modelsbook p.
308-314

• Developing alternative models
• Deterministic methods
• Alternative conceptual models about depositional
  environment
• Alternative theories about rainfall distribution
  and(or) infiltration dynamics
• Stochastic methods
• Alternative realizations of gravel/sand/clay
  distribution developed using indicator kriging
• Combined methods
• Discard some alternative realizations based on
  deterministic depositional theories
• Generate stochastic variations within a
  deterministically derived hydrogeologic framework

9
Guideline 8 Consider alternative models

• Better models have three attributes
• Better fit (But not too good!)
• Weighted residuals that are more randomly
  distributed
• More realistic optimal parameter values

This graph shows model discrimination criteria
for 5 models of the Maggia Valley, southern
Switzerland. SSWR Sum of squared, weighted
residuals. AICc, BIC Model discrimination
criteria.
Foglia, in press, GW. Book, p. 311.