Measures of Estimates Quality

1
Measures of Estimates Quality

• Charles D. Coleman
• Population Estimates Branch
• U.S. Census Bureau
• 4700 Silver Hill Rd., Stop 8800
• Washington, DC 20233-8800
• ccoleman_at_census.gov
• Prepared for Population Association of America
  meetings, 2002

2
Aspects of Quality
1

• Bias
• Is a typical error zero on average?
• Scale (a.k.a. accuracy)
• How far is the typical error from zero?
• Outliers
• Which observations appear to be generated by
  anomalous processes?
• Extreme Errors
• What is the worst that a method can do?

3
Non-i.i.d. Errors
2

• Not independent, independently distributed
  errors.
• Impossible to measure typical errors and their
  transformations.
• Can view set of errors as being n samples of size
  one from n unknown distributions.
• Set of errors is not a single population.
• Can do hypothesis testing.

4
Non-i.i.d. Errors (cont.)
3

• Can test differences.
• Is a set of estimates biased?
• Is estimates set A more accurate than set B
  relative to a given scale measure?

5
Bias
4

• Is an error zero on average?
• Testable hypothesis.
• Cannot measure typical error.
• Can test for presence of bias.

6
Bias (cont.)
5

• Two types of bias
• Mean
• Is the expected value of one or more errors
  nonzero?
• Matched-pairs t-test
• Median
• Is (Are) the median(s) of the distribution(s)
  generating one or more errors nonzero?
• Sign Test

7
Bias Example
6

• Is this dataset positively biased?

8
Scale
7

• Often called accuracy.
• How close is a typical error to zero?
• Subjective
• Based on decision-makers preferences.
• Even if scale can be measured objectively,
  decision-makers preferences may not match
  objective scale measure.
• Paper lays out axioms for scale measures.

9
Scale (cont.)
8

• Scale can be expressed in terms of loss
  functions.
• Loss function measures badness of error.
• Axiomatic basis for loss functions given.
• Viewing population estimates as apportionments
  leads to Websters Rule.
• Special case of loss function.

10
Scale (cont.)
9

• As number of area grows, levels equivalent to
  shares for certain class of loss functions.
  Asymptotic equivalence
• Ratios for different estimates sets converge.
• E.g., Index of Dissimilarity (ID) asymptotically
  equivalent to Mean Absolute Error (MAE).
• Can demonstrate undesirable properties.
• Since ID is asymptotically equivalent to MAE, is
  it desirable for summarizing differences?

11
10
Scale Example

• How close to zero is this dataset on average?

12
Outliers
11

• Can indicate problems with their data-generation
  processes.
• Can be true, but unusual, statements about
  reality.
• Loss functions used for detection.
• Variation of loss functions used to measure
  scale.
• In a sense, outlier detection loss functions can
  be identical to scale measure loss functions.

13
12
Outlier Example

• Where are the outliers?

14
Extreme Errors
13

• The worst that an estimation method can do.
• Statistic Average worst ß of losses used by
  scale measure.
• Coleman and Martindale (2000) recommend ß 5.
• Purely descriptive statistic.
• Not testable.

15
14
Extreme Errors Example

• How much weight is under the right tail?

Loss
0