A New Approach For Testing the Accuracy

1
A New Approach For Testing the Accuracy of
Predicted Vertebrate Occurrences
Sandra M. Schaefer William B. Krohn Raymond J.
OConnor Maine Cooperative Fish and Wildlife
Research Unit, and Department of Wildlife
Ecology University of Maine, Orono
2
intro
Testing Wildlife Occurrence Models
Traditionally focused on single species being
predicted on relatively small study areas
3
Habitat Model Testing
Multiple Species
Large geographic areas
4
Traditionally, GAP models are tested by comparing
model predictions to site-specific occurrence
information
Omission Error (OE) failure to predict
species known to occur on the site.
Commission Error (CE) predicting the presence
of species that do not occur on the site.
5
ME-GAP Test Sites
1. North Maine Forestlands 2. Nesowadnehunk
Field, Baxter State Park 3. White Mountains
National Forest 4. Sunkhaze Meadows National
Wildlife Refuge 5. Holt Research Forest 6. Petit
Manan National Wildlife Refuge 7. Rachel
Carson National Wildlife Refuge 8. Moosehorn
National Wildlife Refuge 9. Mount Desert
Island/ Acadia National Park
6
Site-Specific calculations
of species predicted but not present on the
test site Total number of species present on the
site
CE
of species present but not predicted on the
test site Total number of species present on the
site
OE
7
Overall results of accuracy assessment on ME-GAP
predicted species distributions. Medians and
ranges were calculated within taxonomic group
across all sites.
8
Challenges in Testing

• Purpose of the model needs to be considered in
  testing,
• because it may influence how the errors are
  interpreted.

• Errors can be caused by multiple factors (i.e
  test site
  
• size, field inventory effort, species
  observability).

• Interpretation can be complex. Especially for
  commission
• error where the cause may be either apparent or
  actual.

9
Species-Specific Testing Approach
Assessing model accuracy by calculating OE and
CE for each species across multiple sites within
the study area.
10
How complete are the field inventories? (Nichols
et al. 1998)
complete assumes that all species occurring on
the site were found during the
field inventories.
incomplete assumes that not all species on the
site were
found during the field inventories.
Error Range Difference in the highest and lowest
possible OE and CE. Calculated based on
assumptions of field inventory completeness.
11
Objectives
1) Calculate species-specific Error Range for
avian species known to regularly breed in
Maine. 2) Determine if there is a relationship
between the Error Range and the extent of a
species distribution. 3) Determine if there is a
relationship between the Error Range and
how likely a species is to occur during a field
survey (for statewide species). 4) Compare
the test results for the species-specific and
site-specific approaches.
12
MethodsCalculating the species-specific Error
Range
13
Potential Occurrence
Species could potentially occur on a site if the
site is within the range limit. Field survey
data was also included if it indicates the
species occurs on the site.
14
Species Occurrence Table
1 Within ME-GAP Range P Presence A
Absence
15
Species-Specific calculations
of sites where the species was predicted but
not present Total number of potential occurrence
sites
CE
of sites where the species is present but not
predicted Total number of potential occurrence
sites
OE
16
MethodsData Analysis
17
1) Spearmans rho used to test for a relationship
between a species distribution and the
commission Error Range .
2) Spearmans rho used to test for a relationship
between an a a priori ranking system called
Likelihood of Occurrence Ranks and the
commission Error Range .
3) Site-specific commission error results
compared to the species-specific Error
Range .
18
Likelihood of Occurrence Ranks (LOORs) (Boone and
Krohn 1999)
LOORs are an a priori system of ranking species
based on how likely they are to be seen during a
standard wildlife inventory.
Developed to help interpret the causes of
commission error. (Schaefer and Krohn 2002).
Atlas occurrence information was used to generate
a spatial incidence for each species. The
incidence came from dividing the number of
survey blocks in the atlas having confirmed or
potential breeding by the number of survey
blocks within the species range.
19
Results
20
Assumes Complete Field Survey
Assumes Incomplete Field Survey
A.
B.
Number of Species
Commission Error
Commission Error
C.
D.
Omission Error
Omission Error
Frequency distribution of OE and CE from
predicted avian occurrences with the assumption
of complete (A and C) and incomplete (B and D)
field survey data.
21
Commission Error Range
Number of potential occurrence sites
Commission Error Range for all avian species
across all test sites. rho -0.583 P 0.001
22
Commission Error Range
Likelihood of Occurrence Ranks
Relationship between commission Error Range and
the Likelihood of Occurrence Ranks for those
species that essentially occur statewide. rho
-0.657 P 23
ANOVA F Ratio
p-value Method 20.6
0.000 Habitat 4.42
0.001 MethodHabitat 1.54
0.180
24
Conclusions

• Calculating species-specific Error Range
  provided an opportunity
• to assess the overall predictive quality of
  the habitat models, as well
• as determine the variability of error for each
  species.
• Commission Error Range was significantly
  correlated with species
• distribution as well as with how likely a
  species was to be observed on
• a field inventory.
• The species-specific method generated
  significantly lower commission
• error then the site-specific method.

25
Conclusions (cont)

• If a high Error Range is reported for a species
  that has a high
• likelihood of occurrence then the most likely
  cause for the over
• prediction is in the model.
• However, if a species has a low likelihood of
  occurrence
• and a high Error Range , then the over
  prediction error is likely due to
• having incomplete field surveys for the
  species.

26
Summary

• Both methods are influenced by the data available
  for use in the
• testing process. They also need data for
  the temporal and
• spatial scale being assessed.
• The site-specific method provides a generalized
  idea of how well
• the models are capturing species presence and
  absence
• and across the entire state.
• The species-specific approach gives a more
  detailed description
• of which species are reporting the highest
  levels of error.
• This helps to answer the question of why the
  predictive error
• is being reported.

27
Take Home
Because species-specific and the site-specific
methods provide different information about the
quality of predicted occurrences, we recommend
using both when assessing model accuracy .
28
Acknowledgements
Daniel J. Harrison Steve R. Sader Randall B.
Boone William Halteman
Supporting Organizations and Agencies Gap
Analysis Program, USGS BRD Maine Cooperative Fish
and Wildlife Research Unit Department of Wildlife
Ecology University of Maine