How Well Do You Know Your Model

1
How Well Do You Know Your Model?

• A Methodology for Map Comparison-Based Model
  Validation

2
Why map comparison?

• Good modelling practice
• Verification
• Global behaviour analysis
• Sensitivity analysis
• Calibration
• Validation

3
Criteria of model performance

• Landscape structure
• Edge, Patch, Fractal, Diversity
• Presence
• Overlap / Near overlap
• Multiple scale

Patch size
Euclidean
Edge
Diversity
Fuzzy Kappa
4
The meaning of a comparison

• Typically two outputs
• Layer (map) of similarity / difference
• Summary statistic
• A single summary statistic is meaningless,
  multiple comparisons are necessary to
• Get understanding of distributions
• Understand multi-facetted nature of GoF
• Attach absolute meaning-gt Judgement

5
A sensitivity analysis

• Exploring parameter space

6
Model Constrained Cellular Automata

• Urban Non-Urban
• Total land claim exogenous
• Allocate urban to most attractive location
• Neighbourhood
• Random factor
• Five parameters, 5 possible values
• Brute force calibration 3125 runs

White, Engelen, Uljee 1992
7
Model A sample of results for Portugal
Porto
Lisbon
Algarve
8
Sensitivity

• Plotting parameters to GoF is not workable
• 5 parameters 1 GoF metric 6 dimensions
• Plotting GoF to GoF

Clusters appear! Understanding clusters
understanding parameter space ?
9
The most pronounced clusters
10
Separated clusters
Cluster
Parameter 1
Parameter 2
11
Sensitivity / Bifurcation

• Same approach can be used to identify
  bifurcations as a consequence of stochasticity

12
Calibration

• Using map comparison to select the best parameters

13
Calibration requires a single metric

• Can be a composite of metrics
• Metric drives convergence and solution!

This parameter set?
Or this one?
14
Best fit result overview

• Low intra-metric sensitvity
• Strong inter-metric sensitivity

15
Best fit Results per metric
DATA 1990
DATA 2000
16
Confronting expert judgment

• High agreement
• Aggregation
• Euclidean distance
• Patch size (moving window)
• Fractal dimension (moving window)
• Low agreement
• Patch size (global)
• Fractal dimension (global)
• Kappa
• Fuzzy Kappa

Contradicting earlier empirical work ! Kuhnert et
al 2005, Hagen 2002
17
Conclusions

• Preferred method Patch size moving window
• Best result
• With least cost of information (aggregation)
• Methods must discredit local overestimation
• Otherwise edge growth at the cost of emerging
  clusters
• Methods must be spatial explicit
• Otherwise fringe solution

18
Validation

• Judging and understanding

19
Make sense of diverse comparison results

• Step 1. Normalize to make results mutually
  comparable
• Step 2. Establish a reference level to be able to
  pass absolute judgement

20
Land use change in La Réunion

• Recent urban expansion causes great concern
• What will the future bring
• Exploration of alternative scenarios
• Calibration of a Constraint Cellular Automata

1989
2002
N
60 km
21
Multi-scale, Multi-criteria

• Comparison Model 2002 Reality 2002
• Moving window based structure
• Edge
• Patch size
• Euclidean distance

22
First step Normalization

• Normalize to level of historical change

23
Multi-scale, Multi-criteria
24
Accounting for constraints

• Is the model performance caused by the intrinsics
  of the model or by the constraints that are
  exogeneously imposed on it?
• Neutral models include constraints and exclude a
  process of interest
• Full model performs better than neutral model?
• Increased confidence in the process

Fractal
Source McGarical 2001
25
Neutral models of landscape change

• Growing clusters
• Subject to area constraints
• -Minimal area of change
• -Location alongside existing clusters

• Random constraint match
• Subject to area constraints
• -Minimal area of change
• -Random locations

26
Random Constraint Match
Open -32 City -15 River 18 Park 29
Before
After
RCM
27
Expressing similarity relative to neutral models
28
Significance

• Assess significance by means of Monte Carlo
  simulation and application on multiple time
  periods and regions

29
Thanks for your attention

• Map Comparison Kit
• Software, papers documentation
  www.riks.nl/mck/
• Todays slides and exercises www.riks.nl/news
• Contact
• Alex Hagen-Zanker
• ahagen_at_riks.nl