Global Land Cover: Approaches to Validation

1
Global Land CoverApproaches to Validation

• Alan Strahler
• GLC2000 Meeting
• JRC Ispra 3/02

2
Three Approaches to Validation

• Statistical Approaches
• Sampling a map using design-based inference to
  make accuracy statements about the map with known
  precision
• Characterizing a classification process using
  model-based inference to estimate the accuracy of
  individual pixel labels
• Confidence-Building Measures
• Carrying out confidence-building measures
  includes making studies or comparisons without a
  firm statistical basis that provide confidence in
  the map

3
Design-Based Inference

• Definition
• Sampling to infer characteristics of a finite
  population, such as the pixels in a digital land
  cover map
• Probability Sampling
• Sampled units are drawn with known probabilities
• Example Random or stratified random sampling
• Consistent Estimators
• An estimator of a population parameter must equal
  the population parameter if the sample size
  includes the entire population

4
Design-Based Inference, Cont.

• Consistent estimators include
• Proportion of pixels correctly classified
• Users Accuracy
• Given that a pixel is mapped as A, what is the
  probability that it actually is A on the ground?
• Producers Accuracy
• Given that a pixel is actually of class A, what
  is the probability that it is mapped correctly?
• Confusion Matrix
• The confusion matrix is the primary tool used to
  find consistent estimators
• Diagonals count matches and marginal totals count
  agreements

5
Problems in Design-Based Inference

• Ground Truth
• Determination of the correct class for a
  sampled pixel is not without error
• Photointerpretation errors occur when fine-scale
  imagery is used instead of ground visits
• Misregistration errorsthe wrong location is
  visited or viewed at higher resolution
• Equivocal Classification Schemes
• Classes may not be mutually exclusive or be
  difficult to resolve
• Example Permanent wetland may also be forest
  (IGBP). Are both labels correct?
• Classes may not be well defined
• Example What is a golf course? Is it
  agriculture? Grassland? Urban?

6
Problems in Design-Based Inference, Cont.

• Correctness of Match and Mismatch
• Some errors are worse than otherse.g., open
  shrubland vs. closed shrubland may be minor,
  while forest classified as water may be major
• Leads to fuzzy agreement measures as better
  indicators of map utility
• Mixed Pixels
• Ground truth pixels may contain multiple classes
• Which label is correct?
• Leads to fuzzy confusion matrixes
• Map Comparisons
• Given their error structures, how do we conclude
  that two maps are different?
• If they are different, what differences are
  significant? Which map is more accurate?

7
Model-Based Inference

• Focuses on the classification process, not the
  map
• E.g., Which classifier works better?
• Maps as realizations of a classification process
  that makes random errors
• Reliability Measures
• Parameters that are inferred from the
  classification process
• E.g., maximum likelihood classification gives the
  probability that a pixel belongs to a particular
  class
• Can be mapped and summarized to provide
  information about the quality of a map

8
Confidence-Building Measures

• Looks good! Reconnaissance Measures
• Map conforms well to regional landscape
  attributesmountains, valleys, agricultural
  regions, etc.
• Spatial structure is sensible, not
  salt-and-pepper noise or excessively smooth
• Land-water boundaries are clear, indicating good
  registration of input data
• Free of major glitches, such as cities in the
  Sahara
• Ancillary Comparisons
• Does the classifiers output conform to the
  patterns of land cover documented in other
  datasets or maps?
• Systematic Assessment
• Qualitative assessment of map accuracy in a
  systematic (wall-to-wall) fashion

9
Summary

• Design-based inference provides statements of
  accuracy with known precision at highest cost
• Model-based inference characterizes the accuracy
  of the map-making process at lesser cost
• Confidence-building measures assess map quality
  at low cost
• Validation can, and should, rely on all three
  approaches.