Performance of a diagnostic test

1
Performance of a diagnostic test
Dagmar Rimek EPIET-EUPHEM Introductory Course
2012 Lazareto, Menorca, Spain
Based on the Lecture of 2011 by Steen Ethelberg
2
Outline

• Performance characteristics of a test
• Sensitivity
• Specificity
• Choice of a threshold
• Performance of a test in a population
• Positive predictive value of a test (PPV)
• Negative predictive value of a test (NPV)
• Impact of disease prevalence, sensitivity and
  specificity on predictive values

3
Performance characteristics of a test in a
laboratory setting
4
Population with affected and non-affected
individuals
5
A perfect diagnostic test identifies the affected
individuals only
6
In reality, tests are not perfect
7
Sensitivity of a test
The sensitivity of a test is the ability of the
test to identify correctly the affected
individuals Proportion of persons testing
positive among affected individuals
Sensitivity (Se) TP / (TP FN)
7
8
Estimating the sensitivity of a test

• Identify affected individuals with a gold
  standard
• Obtain a wide panel of samples that are
  representative of the population of affected
  individuals
• Recent and old cases
• Severe and mild cases
• Various ages and sexes
• Test the affected individuals
• Estimate the proportion of affected individuals
  that are positive with the test

9
Example Estimating the sensitivity of a new
ELISA IgM test for acute Q-fever

• Identify persons with acute Q-fever with a gold
  standard (IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are
  representative of the population of individuals
  with acute Q-fever
• Recent and old cases
• Severe and asymptomatic cases
• Various ages and sexes
• Test the persons with acute Q-fever
• Estimate the proportion of persons with acute
  Q-fever that are positive with the ELISA IgM test

10
Example Sensitivity a new ELISA IgM test for
acute Q-fever
Patients with acute Q-fever Patients with acute Q-fever
ELISA IgM test result True positive (TP) 148
ELISA IgM test result - False negative (FN) 2
150
Sensitivity TP / (TP FN)148 / 150 98.7
10
11
What factors influence the sensitivityof a test?

• Characteristics of the affected persons?
• YES Antigenic characteristics of the pathogen in
  the area(e.g., if the test was not prepared with
  antigens reflecting the population of pathogens
  in the area, it will not pick up infected persons
  in the area)
• Characteristics of the non-affected persons?
• NO The sensitivity is estimated on a population
  of affected persons
• Prevalence of the disease?
• NO The sensitivity is estimated on a population
  of affected persons
• Sensitivity is an INTRINSIC characteristic of the
  test

12
Specificity of a test
The specificity of a test is the ability of the
test to identify correctly non-affected
individuals Proportion of persons testing
negative among non-affected individuals
Specificity (Sp) TN / (TN FP)
12
13
Estimating the specificity of a test

• Identify non-affected individuals
• Negative with a gold standard
• Unlikely to be infected
• Obtain a wide panel of samples that are
  representative of the population of non-affected
  individuals
• Test the non-affected individuals
• Estimate the proportion of non-affected
  individuals that are negative with the test

14
Example Estimating the specificity of a new
ELISA IgM test for acute Q-fever

• Identify persons without Q-fever
• Persons without sign and symptoms of the
  infection
• Persons at low risk of infection, negative with
  gold standard (IgM Immunofluorescence Assay)
• Obtain a wide panel of samples that are
  representative of the population of individuals
  without Q-fever
• Test the persons without Q-fever
• Estimate the proportion of persons without
  Q-fever that are negative with the new ELISA IgM
  test

15
Specificity of a new ELISA IgM testfor acute
Q-fever
Persons without acute Q-fever Persons without acute Q-fever
ELISA IgM test result False positive (FP) 10
ELISA IgM test result - True negative (TN) 190
200
Specificity TN / (TN FP)190 / 200 95
15
16
What factors influence the specificity of a test?

• Characteristics of the affected persons?
• NO The specificity is estimated on a population
  of non-affected persons
• Characteristics of the non-affected persons?
• YES The diversity of antibodies to various other
  antigens in the population may affect cross
  reactivity or polyclonal hypergammaglobulinemia
  may increase the proportion of false positives
• Prevalence of the disease?
• NO The specificity is estimated on a population
  of non-affected persons
• Specificity is an INTRINSIC characteristic of the
  test

17
Performance of a test
Disease

18
To whom sensitivity and specificity matters most?

• INTRINSIC characteristics of the test
• ? To laboratory specialists!

19
Distribution of quantitative test results among
affected and non-affected people
Ideal situation
Non-affected
Affected
Number of people tested
TN
TP
0 5 10
15 20
Quantitative result of the test
20
Distribution of quantitative results among
affected and non-affected people
Realistic situation
Non-affected
Affected
TN
TP
Number of people tested
FN
FP
0 5 10
15 20
Quantitative result of the test
21
Effect of Decreasing the Threshold
Non-affected
Threshold for positive result
Affected
FP
Number of people tested
TP
TN
FN
0 5 10
15 20
Quantitative result of the test
22
Effect of Decreasing the Threshold
Disease

23
Effect of Increasing the Threshold
Non-affected
Threshold for positive result
Affected
TN
Number of people tested
TP
FN
FP
0 5 10
15 20
Quantitative result of the test
24
Effect of Increasing the Threshold
Disease

25
Performance of a test and threshold

• Sensitivity and specificity vary in opposite
  directions when changing the threshold (e.g. the
  cut-off in an ELISA)
• The choice of a threshold is a compromise to best
  reach the objectives of the test
• consequences of having false negatives?
• consequences of having false positives?

26
Using several tests

• One way out of the dilemma is to use several
  tests that complement each other
• First use test with a high sensitivity(e.g.
  screening for HIV by ELISA, or for syphilis by
  TPHA)
• Second use test with a high specificity(e.g.
  confirmation of HIV or syphilis by western blot)

27
ROC curves

• Receiver Operating Characteristics curve
• Representation of relationship between
  sensitivity and specificity for a test
• Simple tool to
• Help define best cut-off value of a test
• Compare performance of two tests

28
Prevention of blood transfusion malariaChoice
of an indirect IFA threshold
Sensitivity ()
100
1/10
1/20
1/40
80
1/80
1/160
60
IFA Dilutions
1/320
40
1/640
20
0
0
20
40
60
80
100
100 - Specificity () Proportion of false
positives
29
Comparison of performance of IFA and ELISA IgM
tests for detection of acute Q-fever
Sensitivity ()
100
80
IFA ELISA
60
Area under the ROC curve (AUC)
40
20
0
0
25
50
75
100
100 - Specificity ()
30
Performance of a test in a population
31
How well does the test perform in a real
population?

• The test is now used in a real population
• This population is made of
• Affected individuals
• Non-affected individuals
• The proportion of affected individuals is the
  prevalence

Status of persons Status of persons
Affected Non-affected
Test Positive True False AB
Test Negative False - True - CD
AC BD ACBD
32
Predictive value of a positive test

• The predictive value of a positive test is the
  probability that an individual testing positive
  is truly affected
• Proportion of affected persons among those
  testing positive

33
Positive predictive value (PPV) of a test
Status of persons Status of persons
Affected Non-affected
Test Positive A B AB
Test Negative C D CD
A C BD ACBD
PPV A / (AB) This is only valid for the
sample of specimens tested
33
34
What factors influence the positive predictive
value of a test?
Status of persons Status of persons
Affected Non-affected
Test Positive A B AB
Test Negative C D CD
A C BD ACBD

• Sensitivity?
• YES To some extend.
• Specificity?
• YES The more the test is specific, the more it
  will be negative for non-affected persons (less
  false-positive results).
• Prevalence of the disease?
• YES Low prevalence Low pre-test probability for
  positives.The test will pick up more false
  positives.
• YES High prevalence High pre-test probability
  for positives.The test will pick up more true
  positives.

35
Positive predictive value of a test according to
prevalence and specificity
Specificity
PPV ()
36
Predictive value of a negative test

• The predictive value of a negative test is the
  probability that an individual testing negative
  is truly non-affected
• Proportion of non-affected persons among those
  testing negative

37
Negative predictive value (NPV) of a test
Status of persons Status of persons
Affected Non-affected
Test Positive A B AB
Test Negative C D CD
AC BD ACBD
NPV D / (CD) This is only valid for the
sample of specimens tested
37
38
What factors influence the negative predictive
value of a test?
Status of persons Status of persons
Affected Non-affected
Test Positive A B AB
Test Negative C D CD
AC BD ACBD

• Sensitivity?
• YESThe more the test issensitive, the more it
  captures affected persons (less false negatives).
• Specificity?
• YES But to a lesser extend.
• Prevalence of the disease?
• YES Low prevalence High pre-test probability
  for negatives. The test will pick up more true
  negatives.
• YES High prevalence Low pre-test probability
  for negatives. The test will pick up more false
  negatives.

39
Negative predictive value of a test according to
prevalence and sensitivity
40
Relation between predictive values and
sensitivity (Se), specificity (Sp), prevalence
(Pr)
41
Calculate PPV and NPV
42
Relation between predictive values and
sensitivity / specificity
Increasing specificity ? increasing PPV
Increasing sensitivity ? increasing NPV
43
Relation between predictive values and prevalence
Increasing prevalence ? increasing PPV
Decreasing prevalence ? increasing NPV
44
Example Screening for acute Q-fever in two
settings

• ELISA IgM test
• Sensitivity 98
• Specificity 95
• Population in low endemic area
• Prevalence 0.5
• Patients with atypical pneumonia
• Prevalence 20
• 10,000 tests performed in each group

45
Example Screening for acute Q-fever in a
population in a low endemic area
IgM ELISA test sensitivity 98 IgM ELISA test
specificity 95
Prevalence 0.5
Q-fever Q-fever Q-fever Q-fever Q-fever
Yes No Total
IgM ELISA 49 497 546
IgM ELISA - 1 9,453 9,454
50 9,950 10,000
PPV 8.97 NPV 99.98
46
Example Screening for acute Q-fever in patients
with atypical pneumonia
IgM ELISA test sensitivity 98 IgM ELISA test
specificity 95
Prevalence 20
Q-fever Q-fever Q-fever Q-fever Q-fever
Yes No Total
IgM ELISA 1,960 400 2,360
IgM ELISA - 40 7,600 7,640
2,000 8,000 10,000
PPV 83.05 NPV 99.48
47
To whom predictive values matters most?

• Look at denominators!
• Persons testing positive
• Persons testing negative
• ? To clinicians
• probability that a individual with a positive
  test is really sick?
• probability that a individual with a negative
  test is really healthy?
• ? To epidemiologists!
• proportion of positive tests corresponding to
  true patients?
• proportion of negative tests corresponding to
  healthy subjects?

48
Summary

• Sensitivity and specificity matter to laboratory
  specialists
• Studied on panels of positives and negatives
• Intrinsic characteristics of a test
• Capacity to identify the affected
• Capacity to identify the non-affected
• Predictive values matter to clinicians and
  epidemiologists
• Studied on homogeneous populations
• Dependent on the disease prevalence
• Performance of a test in real life
• How to interpret a positive test
• How to interpret a negative test

49
Where will you do your rain dance?
There?
Here?