Assessing Information from Multilevel and Continuous Tests

1
Assessing Information from Multilevel and
Continuous Tests

• Likelihood Ratios for results other than or
  -

Michael A. Kohn, MD, MPP 10/2/2008
2
Four Main Points

• 1) Dichotomizing a multi-level test by choosing a
  fixed cutpoint reduces the value of the test.
• 2) The ROC curve summarizes the ability of the
  test to differentiate between D and D-
  individuals.
• 3) LR(result) P(resultD)/P(resultD-) slope
  of ROC curve.
• (NOTE Do not calculate an LR() or LR(-) for a
  multilevel test.)
• 4) Pre-Test Odds x LR(result) Post-Test Odds

3
NOTE

• Do not calculate an LR() or LR(-) for a test
  with more than two possible results.

4
Additional Topics

• Optimal Cutoffs
• Walking Man
• C Statistic

5
Example from Chapter 3

• 65-year-old woman with mammogram suspicious for
  malignancy
• Pre-test probability 0.015
• LR(suspicious for malignancy) 100
• Post-test probability ?

6
Update Pre-Test Probability Using LR(test result)

1. Convert pre-test probability (P) to pre-test
   odds. Pre-Test Odds P/(1-P)
2. Calculate LR. P(resultD)/P(resultD-).
3. Post-Test Odds Pre-Test Odds LR
4. Convert post-test odds to post-test probability.
   Prob Odds/(1Odds)

7
Update Pre-Test Probability Using LR(test result)

• 1) Pre-test probability P 0.015
• Pre-test odds P/(1-P) 0.015
• 2) LR(Suspicious for Malignancy) 100
• 3) Post-Test Odds 0.015 100 1.5
• 4) Post-test probability Odds/(1Odds)
  1.5/2.5 0.60

8
Can Use Slide Rule
9
Can Use Excel
10
Can Use Web-Based Calculator

• We will come back to this

(This ends the example for Chapter 3.)
11
Evaluating the Test--Test Characteristics

• For dichotomous tests, we discussed sensitivity
  P(D) and specificity P(-D-)
• For multi-level and continuous tests, we will
  discuss the Receiver Operating Characteristic
  (ROC) curve

12
Using the Test Result to Make Decisions about a
Patient

• For dichotomous tests, we use the LR() if the
  test is positive and the LR(-) if the test is
  negative
• For multilevel and continuous tests, we use the
  LR(r), where r is the result of the test

13
Septic Arthritis
Bacterial infection in a joint.
14
Clinical ScenarioDoes this Adult Patient Have
Septic Arthritis?
15
Clinical ScenarioDoes this Adult Patient Have
Septic Arthritis?

• A 48-year-old woman with a history of rheumatoid
  arthritis who has been treated with long-term,
  low-dose prednisone presents to the emergency
  department with a 2-day history of a red, swollen
  right knee that is painful to touch. She reports
  no prior knee swelling and no recent trauma or
  knee surgery, illegal drug use, rash, uveitis, or
  risky sexual behavior. On examination, she is
  afebrile and has a right knee effusion. Her
  peripheral white blood cell (WBC) count is
  11 000/µL and her erythrocyte sedimentation rate
  (ESR) is 55 mm/h. An arthrocentesis is performed,
  and the initial Gram stain is negative.

You have the synovial white blood cell (WBC)
count.
Margaretten, M. E., J. Kohlwes, et al. (2007).
Jama 297(13) 1478-88.
16
Clinical ScenarioDoes this Adult Patient Have
Septic Arthritis?
Assume pre-test probability of septic arthritis
is 0.38.
How do you use the synovial WBC result to
determine the likelihood of septic arthritis?
Margaretten, M. E., J. Kohlwes, et al. (2007).
Jama 297(13) 1478-88.
17
Why Not Make It a Dichotomous Test?

• Synovial Septic Arthritis
• WBC Count Yes No
• gt25,000 77 27
• 25,000 23 73
• TOTAL 100 100

Note that these could have come from a study
where the patients with septic arthritis (D
patients) were sampled separately from those
without (D- patients).
Margaretten, M. E., J. Kohlwes, et al. (2007).
Jama 297(13) 1478-88.
18
Why Not Make It a Dichotomous Test?

• Sensitivity 77
• Specificity 73
• LR() 0.77/(1 - 0.73) 2.9
• LR(-) (1 - 0.77)/0.73 0.32
• gt 25,000/uL
• - 25,000/uL

19
Clinical ScenarioSynovial WBC 48,000/mL

• Pre-test prob 0.38
• LR() 2.9
• Post-Test prob ?

20
Clinical ScenarioSynovial WBC 48,000/mL

• Pre-test prob 0.38
• Pre-test odds 0.38/0.62 0.61
• LR() 2.9
• Post-Test Odds Pre-Test Odds x LR()
• 0.61 x 2.9 1.75
• Post-Test prob 1.75/(1.751) 0.64

21
Clinical ScenarioSynovial WBC 48,000/mLSlide
Rule

• Pre-test prob 0.38
• LR() 2.9
• Post-Test prob

(Demonstrate Slide Rule)
22
Can Use Excel
Pre-test prob 0.38 LR() 2.9 Post-Test prob
23
Can Use Web-Based Calculator

• http//www.quesgen.com/Calculators/PostProdOfDisea
  se/PostProdOfDisease.html

P(D) Sensitivity 77 P(D-) 1 -
Specificity 1 - 73 27
24
Clinical ScenarioSynovial WBC 128,000/mL

• Pre-test prob 0.38
• LR ?
• Post-Test prob ?

25
Clinical Scenario Synovial WBC 128,000/mL

• Pre-test prob 0.38
• Pre-test odds 0.38/0.62 0.61
• LR 2.9 (same as for WBC48,000!)
• Post-Test Odds Pre-Test Odds x LR()
• 0.61 x 2.9 1.75
• Post-Test prob 1.75/(1.751) .64

26
Why Not Make It a Dichotomous Test?

• Because you lose information. The risk
  associated with a synovial WBC48,000 is equated
  with the risk associated with WBC128,000.
• Choosing a fixed cutpoint to dichotomize a
  multi-level or continuous test throws away
  information and reduces the value of the test.

27
Main Point 1 Avoid Making Multilevel Tests
Dichotomous

• Dichotomizing a multi-level or continuous test by
  choosing a fixed cutpoint reduces the value of
  the test

28
WBC (/uL) Interval of Septic Arthritis of No Septic Arthritis
gt100,000 29 1
gt50,000-100,000 33 7
gt25,000-50,000 15 19
0 - 25,000 23 73
TOTAL 100 100
29
Synovial Fluid WBC Count
30
Histogram

• Does not reflect prevalence of D (Dark D
  columns add to 100, Open D- columns add to 100)
• Sensitivity and specificity depend on the
  cutpoint chosen to separate positives from
  negatives
• The ROC curve is drawn by serially lowering the
  cutpoint from highest (most abnormal) to lowest
  (least abnormal).

Just said that choosing a fixed cutpoint
reduces the value of the test. The key issues are
1) the ROC curve is for evaluating the test, not
the patient, and 2) drawing the ROC curve
requires varying the cutpoint, not choosing a
fixed cutpoint.
31
(No Transcript)
32
(No Transcript)
33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
WBC Count (x1000/uL) Sensitivity 1 - Specificity
gt 8 0 0
gt 100 29 1
gt 50 62 8
gt 25 77 27
0 100 100
Margaretten, M. E., J. Kohlwes, et al. (2007).
Jama 297(13) 1478-88.
37
Cutoff 0
Cutoff gt 25k
Cutoff gt 50k
Cutoff gt 100k
Cutoff gt 8
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
Test Discriminates Fairly Well Between D and D-
D
D-
Test Result
43
Test Discriminates Well Between D and D-
44
Test Discriminates Poorly Between D and D-
D
D-
Test Result
45
Test Discriminates Poorly Between D and D-
46
Area Under ROC Curve
Cutoff 0
Cutoff gt 25k
Cutoff gt 50k
Area Under Curve 0.8114
Cutoff gt 100k
Cutoff gt 8
47
Area Under ROC Curve

• Summary measure of tests discriminatory ability
• Probability that a randomly chosen D individual
  will have a more positive test result than a
  randomly chosen D- individual

48
Area Under ROC Curve

• Corresponds to the Mann-Whitney U Test Statistic
  ( Wilcoxon Rank Sum), which is the
  non-parametric equivalent of Students t test.
• Also corresponds to the c statistic reported in
  logistic regression models

49
Main Point 2ROC Curve Describes the Test, Not
the Patient

• Describes the tests ability to discriminate
  between D and D- individuals
• Not particularly useful in interpreting a test
  result for a given patient

50
ROC Curve Describes the Test, Not the Patient

• Clinical Scenario
• Synovial WBC count 48,000
• Synovial WBC count 128,000

51
Synovial WBC count 48,000
52
Cutoff 0
Cutoff gt 25k
Cutoff gt 50k
Cutoff gt 100k
Cutoff gt 8
53
Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs
WBC (/uL) Sensitivity Specificity Specificity LR LR-
gt100,000 29 99 99 29.0 0.7
gt50,000 62 62 92 7.8 0.4
gt25,000 77 77 73 2.9 0.3
Synovial WBC Count 48,000/uL Which LR should we
use?
54
Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs Sensitivity, Specificity, LR(), and LR(-) of the Synovial Fluid WBC Count for Septic Arthritis at 3 Different Cutoffs
WBC (/uL) Sensitivity Specificity Specificity LR LR-
gt100,000 29 99 99 29.0 0.7
gt50,000 62 62 92 7.8 0.4
gt25,000 77 77 73 2.9 0.3
Synovial WBC Count 48,000/uL Which LR should we
use? NONE of THESE!
55
Likelihood Ratios

• LR() Sensitivity/(1 Specificity)
• P(D)/P(D-)

LR(-) (1 Sensitivity)/Specificity
P(-D)/P(-D-)
56
Likelihood Ratios
P(Result) in patient WITH disease -------------
--------------------------------------- P(Result)
in patients WITHOUT disease

• LR(result) P(resultD)/P(resultD-)

57
WOWO

• With Over WithOut

58
Likelihood Ratios
The ratio of the height of the D distribution to
the height of the D- distribution
LR 15/19 0.8
19
15
59
gt 25k
gt 50k
15
Slope 15/19 0.8
19
60
Likelihood Ratio
WBC (/uL) Interval of D of D- Interval LR
gt100,000 29 1 29.0
gt50,000-100,000 33 7 4.7
gt25,000-50,000 15 19 0.8
0 - 25,000 23 73 0.3
61
Common Mistake

• When given an ROC Table, it is tempting to
  calculate an LR() or LR(-) as if the test were
  dichotomized at a particular cutoff.
• Example LR(,25,000) 77/27 2.9
• This is NOT the LR of a particular result (e.g.
  WBC gt25,000 and 50,000) it is the LR() if you
  divide from - at 25,000.

62
Common Mistake

WBC (/uL) Sensitivity Specificity LR LR-
gt100,000 29 99 29.0 0.7
gt50,000 62 92 7.8 0.4
gt25,000 77 73 2.9 0.3
63
Common Mistake
gt 25,000
77
27
64
Common Mistake

• From JAMA paper
• Her synovial WBC count of 48,000/µL increases
  the probability from 38 to 64. (Used LR 2.9)
• Correct calculation
• Her synovial WBC count of 48,000/µL decreases the
  probability from 38 to 33. (Used LR 0.8)

65
Main Point 3 Likelihood Ratio

• P(Result) in patients WITH disease
• -------------------------------------------------
  -----
• P(Result) in patients WITHOUT disease
• Slope of ROC Curve
• Do not calculate an LR() or LR(-) for a
  multilevel test.

66
NOTE

• Do not calculate an LR() or LR(-) for a test
  with more than two possible results.

67
Clinical ScenarioSynovial WBC 48,000/uL

• Pre-test prob 0.38
• Pre-test odds 0.38/0.62 0.61
• LR(WBC btw 25,000 and 50,000) 0.8
• Post-Test Odds Pre-Test Odds x LR(48)
• 0.61 x 0.8 0.49
• Post-Test prob 0.49/(0.491) 0.33

Can use slide rule, Excel, or web page
68
Clinical ScenarioSynovial WBC 128,000/uL

• Pre-test prob 0.38
• Pre-test odds 0.38/0.62 0.61
• LR(128,000/uL) 29
• Post-Test Odds Pre-Test Odds x LR(128)
• 0.61 x 29 17.8
• Post-Test prob 17.8/(17.81) 0.95

Can use slide rule, Excel, or web page
69
Clinical Scenario

• WBC 48,000/uL Post-Test Prob 0.33
• WBC 128,000/uL Post-Test Prob 0.95
• (Recall that dichotomizing the WBC with a fixed
  cutpoint of 25,000/uL meant that WBC 48,000/uL
  would be treated the same as WBC 128,000/uL and
  post-test prob 0.64)

70
Main Point 4Bayess Rule
Pre-Test Odds x LR(result) Post-Test Odds
What you knew before What you learned
What you know now
71
Summary

• Dichotomizing a multi-level test by choosing a
  fixed cutpoint reduces the value of the test.
• The ROC curve summarizes the discriminatory
  ability of the test.
• LR(result) P(resultD)/P(resultD-) Slope of
  ROC Curve (NOTE Do not calculate an LR() or
  LR(-) for a multilevel test.)
• Pre-Test Odds x LR(result) Post-Test Odds

72
Conforms to Clinical Intuition
73
Synovial WBC for Septic Arthritis

• WBC lt 2000 very reassuring
• WBC 2000 25,000 somewhat reassuring
• WBC 25,000 50,000 indeterminate
• WBC 50,000 100,000 worrisome
• WBC gt 100,000 very worrisome

74
Peripheral WBC Count for Bacteremia in Febrile
Infant
Bacteremia Bacteremia No Bacteremia No Bacteremia
WBC Number Number LR
lt5 8 21 201 5 3.95
5-15 13 34 2727 72 0.47
15 17 45 844 22 2.00
20 9 24 255 7 3.50 0.82
Total 38 3772
Interval LRs as reported in the paper (Ann
Emerg Med 42216-225)
What if WBC count is 18? Which LR should you
use? LR 2.0 because 18 15, or LR 0.82
because 18 lt 20?
75
Peripheral WBC Count for Bacteremia in Febrile
Infant
Actual Interval LRs
Bacteremia Bacteremia No Bacteremia No Bacteremia
WBC Number Number LR
lt5 8 21 201 5 3.95
5-15 13 34 2727 72 0.47
15-20 8 21 589 16 1.35
gt 20 9 24 255 7 3.50
38 3772
What if WBC count is 18? Which LR should you
use? LR 1.35.
76
LR does not decrease steadily as WBC count
decreases. Interval LRs still useful, but AUROC
not a good measure of tests discrimination.
77
Peripheral WBC Count for Bacteremia in Febrile
Infant lt 3 Months Old

• lt 5 Very concerning
• 5 15 Slightly reassuring
• 16 20 Slightly concerning
• gt 20 Concerning

78
Additional Topics

• Optimal Cutoffs
• Walking Man
• C Statistic

79
Optimal Cutoffs
BNP to distinguish between COPD exacerbation and
CHF in the ED patient with dyspnea

• Wang, C. S., J. M. FitzGerald, et al. (2005).
  "Does this dyspneic patient in the emergency
  department have congestive heart failure?" JAMA
  294(15) 1944-56.
• Refers to
• Maisel, A. S., P. Krishnaswamy, et al. (2002).
  "Rapid measurement of B-type natriuretic peptide
  in the emergency diagnosis of heart failure." N
  Engl J Med 347(3) 161-7.

80
Optimal Cutoffs

• What is the single best cutoff to define a BNP as
  positive for CHF?

81
BNP, 500 pg/ml?
BNP, 1000 pg/ml ?
82
Optimal Cutpoints

• Dichotomizing a continuous test by choosing a
  fixed cutoff reduces the value of the test.

And do NOT choose the point where the ROC curve
is closest to the upper left hand corner.
83
Optimal Cutoffs

• But, for a continuous variable, you do have to
  define intervals.
• How do you choose your cutpoints to define the
  intervals?

84
BNP, 500 pg/ml?
BNP, 1000 pg/ml ?
85
BNP for CHF

• BNP lt 100 Not CHF
• BNP 100 500 doesnt change likelihood much
• BNP 500 1000 increases likelihood of CHF
• BNP gt 1000 really increases likelihood of CHF.

86

• THE WALKING MAN OR

87
WHAT CAN YOU LEARN FROM ROC CURVES LIKE THESE?
Bonsu, B. K. and M. B. Harper (2003). "Utility of
the peripheral blood white blood cell count for
identifying sick young infants who need lumbar
puncture." Ann Emerg Med 41(2) 206-14.
88
Walking Man Approach to ROC Curves

• Divide vertical axis into d steps, where d is the
  number of D individuals
• Divide horizontal axis into n steps, where n is
  the number of D- individuals
• Sort individuals from most to least abnormal test
  result
• Moving from the first individual (with the most
  abnormal test result) to the last (with the least
  abnormal test result)

89
Walking Man (continued)

• call out D if the individual is D and N if
  the individual is D-
• Let the walking man know when you reach a new
  value of the test
• The walking man takes a step up every time he
  hears D and a step to the right every time he
  hears N
• When you reach a new value of the test, he drops
  a stone.

90
Synovial WBC Count in 5 Patients with Septic
Arthritis
Patient WBC Count (x 1000/uL)
D1 128
D2 92
D3 64
D4 37
D5 24
91
Synovial WBC Count in 10 Patients Without Septic
Arthritis
Patient WBC Count (x 1000)
N1 71
N2 48
N3 37
N4 23
N5 12
N6 12
N7 8
N8 7
N9 6
N10 0
92
Septic Arthritis No Septic Arthritis
128
92
71
64
48
37 37
24
23
12
12
8
7
6
0
93
DDNDN(DN)DN(NN)NNNN
94
(No Transcript)
95
(No Transcript)
96
WHAT CAN YOU LEARN FROM ROC CURVES LIKE THESE?
97
(No Transcript)
98
Calculating the c Statistic
In the walking man approach to tracing out the
ROC curve, the actual values of the test are not
important for the shape of the ROC curve or the
area under it--only the ranking of the values.
The c statistic for the area under an ROC curve
is calculated using the same information as the
Wilcoxon Rank Sum statistic (or Mann-Whitney U,
which is equivalent) and gives identical P
values.
Non-parametric equivalent of the t test statistic
comparing two means.
99
Septic Arthritis No Septic Arthritis
128
92
71
64
48
37 37
24
23
12
12
8
7
6
0
100
Boxes under Curve 43.5 Total Boxes 50 Area
Under Curve 43.5/50 0.87
101
Replace Test Results with Ranks
BACTEREMIA NO BACTEREMIA
1
2
3
4
5
6.5 6.5
8
9
10.5
10.5
12
13
14
15
S 21.5
102
Calculating the C Statistic
S 21.5 Smin d(d1)/2 5(6)/2 15 Smax dn Smin 5(10) 15 65 C (Smax S) / (Smax Smin) (65 21.5) / (65 15) 43.5/50 0.87 Smax Smin dn