Recognizing Common Biostatistical Errors: A CaseBased Approach

1
Recognizing Common Biostatistical ErrorsA
Case-Based Approach

• Thomas B. Newman, MD, MPH
• Kristine A. Madsen, MD, MPH

2
Conflicts of Interest/Off-Label Use

• TN co-investigator on an educational grant
  Marketing of Medicines, funded from settlement
  of a case for illegal promotion of Neurontin

3
Why is the graph misleading?
4
Agenda

• Review and discuss concepts
• Break
• Divide into groups, work on unknowns
• Review unknowns in large group

5
Concepts to be Illustrated

• Type III errors
• Standard Deviation/Standard Error
• Between- vs Within-group comparisons
• Relative Risk/Absolute Risk NNT
• Relative Risk/Odds Ratio
• Violation of the Independence Assumption
• Negative Studies and Confidence Intervals

6
(No Transcript)
7
Q Whats wrong here?
8
Safety, efficacy, and tolerance of intestinal
lavage
The safety, efficacy, and patient tolerance of an
enterally administered isotonic intestinal lavage
solution was evaluated in 20 pediatric patients
(ages 1 1/2 to 19 years) undergoing diagnostic
colonoscopy. Emesis occurred in 4 patients,
nausea in 11, and abdominal distension in 5.
Clear stools were produced in a mean (/- SE)
time of 2.6 /- 0.3 hours. Of 20 patients, 11
required or requested nasogastric administration
of the lavage solution because of its unpleasant
taste. We conclude that whole intestinal
perfusion with a balanced electrolyte solution
containing polyethylene glycol is safe,
acceptable, and efficacious in children. J
Pediatr 1991 119148-52
9
Intestinal Lavage

• What is meant by mean time to clear stools 2.6
  0.3 hours (mean SE)?
• Do you agree that intestinal lavage is safe,
  acceptable and efficacious?

10
Behavior Abnormalities and Poor School
Performance Due to Oral Theophylline
Table 4. Teacher Behavioral Assessment Score
Pre- and Post-treatment
Results are average scores SD. Score is sum
for 53 items, each of which was scored as 1, as
much as other students 2, somewhat more than
other students 3, much more than other students
Peds 1986781133
Q1 What do the 5 p-values mean and which is most
relevant? Q2 Do you see a Type 3 error?
11
Effects of Iron Therapy on Developmental Scores
of Iron-Deficient Infants
Q Does iron therapy improve outcomes?
J Pediatr. 19789221-5
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
Mortality rates after 10.5 years in the Multiple
Risk Factor Intervention Trial

• The MRFIT was a primary prevention trial to test
  the effect of multifactor intervention on
  coronary heart disease mortality in high-risk men
  randomly assigned to special intervention (n
  6428) or to usual health care (n 6438) This
  report describes the mortality findings after
  10.5 years Mortality rates were lower for men
  who received special intervention than for men
  who received their usual care by 10.6 for
  coronary heart disease These data suggest that
  multiple risk factor intervention confers a
  mortality benefit in middle-aged men over a
  period of about 10 years.

JAMA 1990 2631795-801
Q What is missing here?
16
Multiple Risk Factor Intervention Trial
Table 2 - Mortality End-Point Findings Through
December 31, 1985, for MRFIT SI and UC
Men No. () of Deaths One- Mortality
Intervention Usual Care Relative Difference,
Sided End Point (n6428)
(n6438) (95 Confidence Interval) P

• Coronary heart disease death 202 (3.1) 226 (3.5)
  -10.6 .12
• (ICD-9 410-414, 429.2) (-23.7 to 4.9)
• Cardiovascular disease death 266 (4.1) 290 (4.5)
  -8.3 .16
• (ICD-9 390-459) (-20.2 to 5.5)
• All-cause mortality 496 (7.7) 537 (8.3) -7.7 .10
• (-16.6 to 2.3)

MRFIT indicates Multiple Risk Factor
Intervention Trial SI, special Intervention and
UC, usual care. ICD-9 indicates International
Classification of Diseases, Ninth Revision,
Clinical Modification. (1-RR)x100, where the RR
(relative risk) is estimated from the
proportional hazards regression model.
17
Effect Size Relative vs Absolute (Dichotomous
Outcome Variables)

• RR Relative Risk (Risk Ratio) Risk in
  intervention group/Risk in control group
• RRR Relative Risk Reduction 1-RR
• ARR Absolute Risk Reduction Risk in control
  group - Risk in intervention group
• NNT Number Needed to Treat 1/ARR

18
RRR and ARR Example

• If a treatment reduced failure rates from 4 in
  the placebo group to 1 in the treated group,
  what would be the relative and absolute risk
  reductions?
• RRR 75
• ARR 3

19
Q What does the 34 reduction mean?
20
Nimotop Ad Graph
21
100 90 80 70 60 50 40 30 20 10 0
Original figure
To scale
11
22
Which Risk Reduction is Most Relevant?

• Questions of etiology Relative Risk
• Does the treatment work?
• Is this a risk factor?
• Clinical decisions weighing benefits vs. costs
  or risks Use absolute risk
• Is the treatment worth giving

23
Diabetes Control and Complications Trial (DCCT)
Adolescents
53 reduction in retinopathy (plt0.05), but
tripling of hypoglycemia (plt0.05) Study
conclusion We conclude that the benefits ..of
intensive treatment outweigh the increased risk
of hypoglycemia that accompanies such treatment.
J Pediatr, 1994. 125177-88
Q Would you agree with this conclusion?
24
DCCT - Rates per 100 person-years of follow-up
19 episodes of hypoglycemia and 5.5 episodes of
seizures or coma per gt3 point change in
retinopathy score
25
Zelnorm for Constipation

• Randomized, double-blind trial of Zelnorm
  (tageserod) for chronic constipation in 1264
  patients from 128 centers in 18 countries
• Results Zelnorm superior (P lt0.0001)
• Does this mean it worked well?

Am J Gastroenterol 2005 100362-72
26
Zelnorm for Constipation

• Outcome increase of 1 complete spontaneous
  bowel movement (CSBM) per week over 4 week period
  
• Response rates 40 with Zelnorm 6 mg bid, 27
  with placebo

Am J Gastroenterol 2005 100362-72
27
Cost of a CSBM

• Risk difference 40 - 27 13
• NNT 1/.13 7.4 to get at least 1 more
  CSBM/week
• 1 week of Zelnorm 6 mg tablets costs 42
• 7.4 x 42 310 per person getting at least 1
  more CSBM/wk
• Assume average improvement is 2 CSBM/wk
• 310/2 CSBM 155/CSBM
• P lt0.0001 does not mean treatment is worth the
  cost!

28

• Incidence of vomiting 2.9 (32 during 1115
  feedings)
• vs 5.4 (67 during 1248 feedings), p0.001
• Q Whats wrong with this analysis?

29
Incidence of Bacteriuria in Patients using Clean
Intermittent Catheterization (CIC) vs Ileal Loop
Diversion (ILD)
P lt 0.001
Pediatrics 1982 70665-9.
Q Whats wrong with this analysis?
30
(No Transcript)
31
Factors Associated with Resident LP Success

• Of 377 performed by trainees, 279 (74) were
  successful.
• LPs were 3 times more likely to be successful
  among infants gt12 weeks of age than among younger
  infants (odds ratio OR 3.1 95 confidence
  interval CI 1.28.5).
• Success rate 70 vs 88

Pediatrics 2006117876
Q Were LPs 3 times more likely to be successful
on older infants?
32
Factors Associated with Resident LP Success
Pediatrics 2006117876
33
RR vs OR

• RR is RISK of (disease/side effects/etc) in
  study population divided by the risk in the
  comparison population
• OR is ODDS of (disease/side effects/etc) in study
  population divided by odds in control population
• When the outcome is rare, the OR approximates the
  RR when the outcome is common (gt10), the OR
  overestimates RR

34
CONVENTION

• Use risk or odds of something BAD, and put
  comparison group on bottom
• RR or OR gt 1 means exposure is bad
• RR or OR lt 1 means exposure is good
• RR or OR 1 means no relationship

35
Confidence Intervals

• Extended Follow-up of Long-Term Survivors of
  Childhood Acute Lymphoblastic Leukemia
• The death rate for the irradiated group slightly
  exceeded the expected rate in the general U.S.
  population (standardized mortality ratio, 1.90
  95 percent confidence interval, 1.12 to 3.00),
  whereas that for the nonirradiated group did not
  differ from the population norm (standardized
  mortality ratio, 1.75 95 percent confidence
  interval, 0.34 to 5.00).

N Engl J Med 2003 349640-649
Q Do the data suggest the irradiated group had
worse survival?
36
Confidence Intervals for negative studies

• Oral amoxicillin to treat possible occult
  bacteremia in febrile children
• Randomized, double-blind trial
• 3-36 month old children with Tgt 39 C (N 955)
• Treatment Amox 125 mg tid (lt 10 kg) or 250 mg
  tid (gt 10 kg)
• Outcome major infectious morbidity

New Engl J Med 19873171175-80
37
Amoxicillin for possible occult bacteremia, cont.

• Overall 27 children (3) bacteremic
• Of these 27, major infectious morbidity occurred
  in 3 (2 persistent bacteremia, 1 periorbital
  cellulitis)
• 2/19 (10.5) with amoxicillin vs 1/8 (12.5) with
  placebo. (P 0.9)
• Conclusion Data do not support routine use of
  standard doses of amoxicillin

Q Do you agree with the conclusion?
38
5 levels of sophistication

• Level 1 P gt 0.05 treatment does not work
• Level 2 Look at power for study. (Authors
  reported power 0.24 for OR4. Therefore, study
  underpowered and negative study uninformative.)

39
5 levels of sophistication, contd

• Level 3 Look at 95 CI for RRRR 0.84 95 CI
  (.09 to 8.0)
• Level 4 Make sure you do an Intention to Treat
  analysis! (Not OK to restrict attention to
  bacteremic patients!)
• So its 2/507 vs 1/448 RR 1.8 (amoxicillin
  worse) 95 CI (0.05 to 6.2)

40
Level 5 the clinically relevant quantity is the
Absolute Risk Reduction (ARR)!

• 2/507 (0.4) with amoxicillin vs 1/448 (0.2)
  with placebo
• ARR -0.17 amoxicillin worse
• 95 CI (-0.9 harm to 0.5 benefit)
• Therefore, LOWER limit of 95 CI for benefit
  (i.e., best case) is NNT 1/0.5 200
• So this study suggests need to treat gt 200
  febrile children to prevent Major Infectious
  Morbidity in one

41
When Is It Safe to Discontinue Penicillin
Prophylaxis in Children With Sickle Cell Anemia?

• Randomized, double-blind trial
• Subjects 400 Children with Sickle Cell Anemia
  who had received PCN 2 years and received
  23-valent pneumococcal vaccine
• Intervention PCN 250 mg BID or placebo
• Outcome pneumococcal bacteremia or meningitis
  over average 3.2 years of follow-up
• Power calculation 80 to detect 67 decrease in
  outcome, from 12 to 4

Falletta et al. J Pediatr 1995 127(5) 685-90.
42
When Is It Safe to Discontinue Penicillin
Prophylaxis in Children With Sickle Cell Anemia?
Results

• Pneumococcal bacteremia or meningitis over 3.2
  years 2/200 (1) with continued prophylaxis vs.
  4/200 (2) with placebo
• RR .5 (95 CI .09 to 2.7)
• ARR 1/3.2 years (95 CI -1.3 to 3.3)
• What can you conclude? Did the study have enough
  power?
• What is the smallest NNT consistent with these
  results?

Falletta et al. J Pediatr 1995 127(5) 685-90.
43
Unknowns
44
Unknown 1
Number () of children with temperature 38ºC
after receiving DTaP vs DTP vaccine
Peds ID 199312131-5
P-value for 5/252 vs. 5/65 0.032

• Q What is wrong with how the P-value was
  obtained?

45
Unknown 2 The Beneficial Effects of Early
Dexamethasone Administration in Infants and
Children with Bacterial Meningitis

• Q What is wrong here?
• At follow-up examination 7 of the 51
  dexamethasone-treated children (14 percent) and
  18 of 48 controls (38 percent) had one or more
  neurologic or audiologic sequelae (P 0.007)
  the relative risk of sequelae for a child
  receiving placebo as compared with a child
  receiving dexamethasone was 3.8 (95 percent
  confidence interval, 1.3 to 11.5).

N Engl J Med 1991 3241525-31
46
Dexamethasone Calculations

• Normally, control group on bottom they arrived
  at their answers by putting it on top
• RR 37.5 2.7
• 14
• OR 18 44 3.8
• 7 30

47
Unknown 3 Randomized trial of Antibiotic
Prophylaxis after Acute Pyelonephritis

Pediatrics 2006117629

• For SCARRING, ARR - 5.6 (95 CI -14.6 to
  3.3)

Q Based upon the 95 CI for the ARR, if
treatment actually worked, what is the lowest
number of children with reflux youd need to
treat to prevent scarring in one child?
48
Unknown 4 Impact of a diabetes prevention
program on fasting capillary glucose
There was a significant difference in change in
fasting capillary glucose levels in
intervention and control school students
(adjusted difference, 2.24 mg/dL 95 CI, 6.53
to 2.05 mg/dL P .03).
Q What is wrong here?
49
Unknown 5 Infrared Tympanic Membrane
Thermometer in Pediatric Patients
Table 2 Observed Difference of Temperature by
Glass-Mercury vs. Tympanic Membrane Thermometer

• Mean difference 0.08 degrees
• 95 CI (0.05 to 0.11 degrees)

Pediatrics 1990 85854-8
Q Does this mean that tympanic temperatures are
accurate?
50
Slides and handouts available atwww.epibiostat.u
csf.edu/epidem/personnel/newman_document_repositor
y

• (Or, just google Thomas B. Newman)