Estimating the Expected Warning Time of Outbreak-Detection Algorithms

1
Estimating the Expected Warning Time of
Outbreak-Detection Algorithms

• Yanna Shen, Weng-Keen Wong, Gregory F. Cooper
• RODS Laboratory, Center of Biomedical
  Informatics, University of Pittsburgh

2
Overview

• Objective
• Background
• Methods
• Experimental Results
• Conclusions
• Future Work

3
Objective

• A new measure for evaluating alerting algorithms,
  which is called Expected Warning Time (EWT).
• It is a generalization of the standard AMOC curve.

4
Why Useful?

• Can compare expected clinician detection time to
  expected computer-based algorithm detection time
• Can provide a promising new approach for
  optimizing and comparing outbreak detection
  algorithms

5
Background
hit
hit
hit
hit



Time
Release occurs
6
Model

• A simple model of clinician outbreak detection
• Assumes that
• People with disease D are diagnosed independently
  of each other.
• The probability of a person with disease D being
  diagnosed is constant (p).

7
Equation

• Definitions
• p probability that a person with D is diagnosed
  as having D upon presentation with that disease
• time(i) maps patient case i to the time at
  which that patient presented with D to clinicians
• M total of patient cases with D
• t time at which the alerting score first
  exceeds a given threshold

8
Equation
x
x

Probability that clinicians will detect the
outbreak on the ith case
WT if clinicians never detect the outbreak
WT if clinicians first detect the outbreak on the
ith case
Probability that clinicians will never detect the
outbreak
9
Experiment Setup

• Apply PANDA to simulated cases of inhalational
  anthrax
• For various value of p, derive EWT for PANDA

10
PANDA

• An outbreak detection system
• Uses causal Bayesian networks to model
  spatio-temporal patterns of a non-contagious
  disease in a population (Cooper, 2004)
• Contains a model to detect inhalational anthrax

11
BARD

• BARD simulator produces the simulated cases of
  anthrax.
• It models the effects of an outdoor airborne
  anthrax release using the Gaussian plume model of
  atmospheric dispersion and a model of
  inhalational anthrax (Hogan, 2004).

12
Performance of Clinicians

• p Clinician detection proficiency
• P(CD) Probability that clinicians will detect
  the outbreak at all
• ECDT Expected clinician detection time given
  that clinicians detect the outbreak
  

p P(CD) ECDT (hours)
0.001 0.992 167.9
0.005 0.999 104.1
0.01 0.9999 88.6
0.05 0.99999 64.5
1 1 0
13
Experimental Results

• p increases, EWT decreases
• p 1, EWT 0

14
Experimental Results
False alert rate 1 per month
Clinician-detection-proficiency (p) Expected Warning Time (EWT)
0.001 76 hours
0.005 13 hours
0.01 5 hours
0.05 34 minutes
1 0

• If and false alert rate 1 per
  month, then EWT minutes.
  

15
Conclusions

• The Expected Warning Time (EWT) is a useful
  concept for evaluating outbreak-detection
  algorithms.
• We illustrated the general idea of EWT using a
  simple model of clinician detection and simulated
  cases of inhalational anthrax.
• Our example analysis suggests that PANDA is most
  helpful when clinicians detection proficiency lt
  5.

16
Future Work

• Extend the model
• Instead of a constant (p), use the function p(t),
  where t is time
• Develop and apply more disease-specific models of
  clinician detection (please see the poster by
  Christina Adamou)

17
Acknowledgements

• This research was supported by grants from the
  National Science Foundation (IIS-0325581), the
  Department of Homeland Security
  (F30602-01-2-0550), and the Pennsylvania
  Department of Health (ME-01-737).
• We thank members of the Bayesian Biosurveillance
  Project for helpful comments.