Calibration and validation

1
Calibration and validation

• Calibration and validation of the Environment
  Explorer land use model

2
Calibration

• The process of tuning parameters of a model with
  the purpose of obtaining the best goodness-of-fit
• iterative and local
• historical goodness-of-fit
• A shift from expert judgement and manual
  calibration to empirical and automated approach
• Gain efficiency and applicability
• Help building theory

3
Calibration and validation project(2003)

• Major (re-)calibration effort
• aimed at the development tools to support
  (semi-) automatic calibration
• Emphasis of policy exercises change, hence the
  model, the set of variables and the land uses
  modelled change
• Data are updated regularly
• Models improve over time.
• Calibration period 1989-1996
• Validation period(s) 1996-2000 1989-2030

4
Stepwise Calibration procedure

• Modular model ? enables use of modular
  calibration routines
• One main disadvantage
• Essential feedbacks get lost ? calibrate coupled
  models ? some duplication of tasks.
• Many advantages
• Model specific calibration techniques and tools
• Emphasis on model specific parameters
• Model specific GOF and analysis
• Reduction of processing time.
• Iterative process
• First decoupled use stored time series, then
  coupled use model output
• First Local (cellular), then Regional, then
  coupled

5
Objective function Regional model(Van Loon, 2004)
Measuredvalues
Simulatedvalues

• Minimize error
• Emphasis on sector(s)
• Emphasis on two parameter sets
• Attractiveness parameter set
• Parameters influences the attractiveness and
  hence activity levels (jobs and residents)
• Density parameter set
• Parameters influence the density and hence number
  of cells

Initialvalues
6
Calibration algorithmRegional Model

• Many parameters and local optima but,
  relatively short processing time
• Combined optimisation algorithms
• Hill climbing / Golden section search ?
  Convergence towards a local optimum
• Random search ( mutation step in GAs)? Search
  for a global optimum
• Simulated annealing
• Combine their strengths and get rid of their
  weaknesses.

7
Goal function Local modelHagen, IJGIS,2003

• Fuzzy map comparison Maximize similarity at
  higher level of abstraction

1989
2030
8
Calibration algorithm Local model(Improved
Straatman et al., CEUS, 2004)

• Iterative optimization of CA-distance rules
• Improves an initial rule-set
• Semi-automatic includes expert evaluation of the
  resulting rules to remove rules not to be
  explained by theory
• Processing time versus Time for analysis.
• Carry out selective optimization
• Where are the major errors in the simulated maps?
• Which can be solved?
• Which adjustments will be successful?
• Adjusting the rules turn the model inside out
• What should have been the correct land use?
• hence, the transition potential?
• hence, the neighbourhood effect?
• and hence the interaction (distance decay) rules?

9
Find best chance of improvement

• Three land uses are required
• Simulated land use
• Real land use
• Influencing land use
• Simulated and real land use based on Fuzzy Kappa
• Influencing land use on the basis of
  neighbourhood analysis
• Compare false positives to true positives
• Compare false negative to true negatives

10
Two types of competition and errors

• Competition between locations
• The land use ends up at the location that appeals
  the most.
• i.e. has the highest total potential for the
  particular land use
• Competition between land use types
• A cell ends up with the land use type to which it
  appeals the strongest
• Both processes are active simultaneously
• But our calibration routine addresses only one at
  a time
• For now, only the second

11
Mean neighbourhood

• Calculate required difference in neighbourhood
  effect for all cells
• Select n cells with smallest difference
• Calculate their mean neighbourhood (NBH)

12
Reduced parameter set
13
Partial derivatives of nbh effect

• Inner product of mean and partial derivative of
  weight at distance to parameter

dP/da
X
14
Find best adjustment

• Adjust parameters (a,b,c,d,e) and run model, both
  for simulated and real land use
• Adjustment is ?Potential/D
• Run the model and evaluate Fuzzy Kappa
• Keep the best adjustment

15
Iterate but not fruitlessly

• Unsuccesful attempts into tabu list
• Limited size
• First in first out
• The combinations of real, simulated and
  influencing land use in tabulist are not
  considered

16
Further work on CA calibration

• More (all) parameters at the same time
• Further exploitation of two types of competition
  and errors
• Better ability to deal with low quality data
• Via comparison methods
• By addressing long term emerging patterns
• Stronger interaction between modeller and
  software
• Which rules to concentrate on
• Which to ignore
• The general shape of rules (library)

17
Pilot long term emerging patterns
18
Results
Calibration period
Validation period
19
Interpretation of ResultsNaive predictors

• Minimizing the goal functions, yes, but how good
  are the results in absolute terms?
• Interpretation of the level of error
• Comparison with a minimalist model (null-model, a
  naive predictor)
• Situation today is the best prediction for
  tomorrow
• Local Random Constraint Match
• Map changes minimally due to the number of
  required and known changes
• Changes are distributed randomly
• Regional Constant Share model
• Proportional distribution of activities over all
  regions remains constant

Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
Inh. 10 Jobs 15
20
Results

• Compare EE results and naive predictors with
  observed data
• Micro model Random Constraint Match (RCM) Fuzzy
  Kappa match
• Macro model Constant Share model (CS) growth
  not captured

• Good calibration 1989-1996
• Mediocre validation 1996-2000

21
ResultsIndustry
22
ResultLocal model
23
Influence of the length of the validation period
For the short time horizon, naive predictors are
better models, but, what about the long term ?
1989
24
Influence of quality of the data

• Base maps 1989, 1993, 1996 and 2000
• Dominant land use at 500 m resolution
• Dubious land use changes

Few (1) dubious cells on the whole map,but
many (25-35 ) of all observed changes
25
Further support of the modelling process

• Sensitivity analysis
• Test of model robustness
• Determination of parameter ranges

• Calculation of fit factors
• Incorporation of mid-period (1993) data in GOF
• MS Excel evaluation sheets for the local and
  regional models
• Incorporated as prototypes in Environment Explorer

26
Conclusions Calibration/Validation

• Calibration lead to a modification and
  simplification of the model!!
• Calibration methods work reasonably fine
• They produce much better results and faster than
  the expert
• but, do not guarantee an optimal solution (search
  space is too big)
• and, do not take into consideration data quality
  sufficiently
• and, lack currently the intelligence to
  distinguish between the process and pure
  hazard
• and, are likely to over-calibrate the model on
  just one possible path of the system ( the
  historic path)