Recognizing Common Biostatistical Errors: A CaseBased Approach

1
Recognizing Common Biostatistical ErrorsA
Case-Based Approach

• Thomas B. Newman, MD, MPH
• Kristine A. Madsen, MD, MPH

2
Conflicts of Interest/Off-Label Use

• Tom is promoting his book but hes donating all
  his royalties to Physicians for Social
  Responsibility
• No other conflict or off-label use

3
Toms Book a must read
4
Why is the graph misleading?
5
Agenda

• Review and discuss concepts
• Break
• Divide into groups, work on unknowns
• Review unknowns in large group
• Answers/discussion (and these slides) posted in
  Tom's Document Repository. Google "Thomas B
  Newman"

6
Concepts to be Illustrated

• Type III errors
• Standard deviation/standard error
• Between- vs within-group comparisons
• Relative Risk/Absolute Risk NNT
• Relative Risk/Odds Ratio
• Violation of the independence assumption
• Understanding regression coefficients
• Statistical vs. clinical significance

7
(No Transcript)
8
Q Whats wrong here?
9
Safety, efficacy, and tolerance of intestinal
lavage
The safety, efficacy, and patient tolerance of an
enterally administered isotonic intestinal lavage
solution was evaluated in 20 pediatric patients
(ages 1 1/2 to 19 years) undergoing diagnostic
colonoscopy. Emesis occurred in 4 patients,
nausea in 11, and abdominal distension in 5.
Clear stools were produced in a mean (/- SE)
time of 2.6 /- 0.3 hours. Of 20 patients, 11
required or requested nasogastric administration
of the lavage solution because of its unpleasant
taste. We conclude that whole intestinal
perfusion with a balanced electrolyte solution
containing polyethylene glycol is safe,
acceptable, and efficacious in children. J
Pediatr 1991 119148-52
10
Intestinal Lavage

• What is meant by mean time to clear stools 2.6
  0.3 hours (mean SE)?
• Do you agree that intestinal lavage is safe,
  acceptable and efficacious?

11
An Infrared Tympanic Membrane Thermometer in
Pediatric Patients

• Mean difference 0.08 degrees
• 95 CI (0.05 to 0.11 degrees)
• Does this mean that tympanic temperatures are
  accurate?
• If the tympanic temp is 38.0, what can we say
  about the probable rectal temp?

Table 2 Temperature Measurement by Glass-Mercury
and Tympanic Membrane Thermometer
Pediatrics 1990 85854-8
12
Behavior Abnormalities and Poor School
Performance Due to Oral Theophylline
Table 4. Teacher Behavioral Assessment Score
Pre- and Post-treatment
Results are average scores SD. Score is sum
for 53 items, each of which was scored as 1, as
much as other students 2, somewhat more than
other students 3, much more than other
students. t-tests
Peds 1986781133
Q1 What do the 5 p-values mean and which is most
relevant? Q2 Do you see a Type 3 error?
13
Combination Therapy With Desmopressin and an
Anticholinergic Medication for Nonresponders to
Desmopressin

• Significant decrease in wet nights with
  anticholinergic med added, not with placebo
• Does this show ACh works?

Pediatrics, Nov 2008 1221027-1032
14
Combination Therapy With Desmopressin and an
Anticholinergic Medication for Nonresponders to
Desmopressin

• "With a generalized estimating equations
  approach, there was a significant 66 decrease in
  the risk of a wet episode"
• Responders ( 50 reduction in wet nights)
• 8/18 (44) vs 5/16 (31) P 0.43
• Conclusion "This study represents the first
  prospective, placebo-controlled trial examining
  the effect of desmopressin in combination with
  long-acting, anticholinergic, bladder-relaxing
  therapy for monosymptomatic primary nocturnal
  enuresis. "

Pediatrics, Nov 2008 1221027-1032
15
Mortality rates after 10.5 years in the Multiple
Risk Factor Intervention Trial

• The MRFIT was a primary prevention trial to test
  the effect of multifactor intervention on
  coronary heart disease mortality in high-risk men
  randomly assigned to special intervention (n
  6428) or to usual health care (n 6438) This
  report describes the mortality findings after
  10.5 years Mortality rates were lower for men
  who received special intervention than for men
  who received their usual care by 10.6 for
  coronary heart disease These data suggest that
  multiple risk factor intervention confers a
  mortality benefit in middle-aged men over a
  period of about 10 years.

JAMA 1990 2631795-801
Q What is missing here?
16
Multiple Risk Factor Intervention Trial
Table 2 - Mortality End-Point Findings Through
December 31, 1985, for MRFIT SI and UC Men
MRFIT indicates Multiple Risk Factor
Intervention Trial SI, special Intervention and
UC, usual care. ICD-9 indicates International
Classification of Diseases, Ninth Revision,
Clinical Modification. (1-RR)x100, where the RR
(relative risk) is estimated from the
proportional hazards regression model.
17
Effect Size Relative vs Absolute (Dichotomous
Outcome Variables)

• RR Relative Risk (Risk Ratio) Risk in
  intervention group/Risk in control group
• RRR Relative Risk Reduction 1-RR
• ARR Absolute Risk Reduction Risk in control
  group - Risk in intervention group
• NNT Number Needed to Treat 1/ARR

18
(No Transcript)
19
RRR and ARR Example

• If a treatment reduced failure rates from 4 in
  the placebo group to 1 in the treated group,
  what would be the relative and absolute risk
  reductions?

20
Which Risk Reduction is Most Relevant?

• Questions of etiology Relative Risk
• Does the treatment work?
• Is this a risk factor?
• Clinical decisions weighing benefits vs. costs
  or risks Use absolute risk
• Is the treatment worth giving

21
Q What does the 34 reduction mean?
22
Nimotop Ad Graph
23
100 90 80 70 60 50 40 30 20 10 0
Original figure
To scale
11
24
(No Transcript)
25
(No Transcript)
26
Penciclovir for Recurrent Oral HSV

• Randomized, double-blind trial sponsored by
  manufacturer
• 1573 patients from 31 US Clinics
• 1 Penciclovir or placebo applied q 2 hours while
  awake
• Penciclovir group more rapid resolution (Plt0.001)

Spruance et al. JAMA 19972771374-9
27
Penciclovir Effect Size

• Median time to healing 5.5 vs. 4.8 days (13
  shorter with penciclovir, P lt 0.001)
• When sample size is large, P-value can be small
  even if effect size is meager

28
Diabetes Control and Complications Trial (DCCT)
Adolescents
53 reduction in retinopathy (plt0.05), but
tripling of hypoglycemia (plt0.05) Study
conclusion We conclude that the benefits ..of
intensive treatment outweigh the increased risk
of hypoglycemia that accompanies such treatment.
J Pediatr, 1994. 125177-88
Q Would you agree with this conclusion?
29
DCCT - Rates per 100 person-years of follow-up
19 episodes of hypoglycemia and 5.5 episodes of
seizures or coma per gt3 point change in
retinopathy score
J Pediatr, 1994. 125177-88
30
Zelnorm for Constipation

• Randomized, double-blind trial of Zelnorm
  (tageserod) for chronic constipation in 1264
  patients from 128 centers in 18 countries
• Results Zelnorm superior (P lt0.0001)
• Does this mean it worked well?

Am J Gastroenterol 2005 100362-72
31
Zelnorm for Constipation

• Outcome increase of 1 complete spontaneous
  bowel movement (CSBM) per week over 4 week period
  
• Response rates 40 with Zelnorm 6 mg bid, 27
  with placebo

Am J Gastroenterol 2005 100362-72
32
Cost of a CSBM

• Risk difference 40 - 27 13
• NNT 1/.13 7.4 to get at least 1 more
  CSBM/week
• 1 week of Zelnorm 6 mg tablets costs 42
• 7.4 x 42 310 per person getting at least 1
  more CSBM/wk
• Assume average improvement is 2 CSBM/wk
• 310/2 CSBM 155/CSBM
• P lt0.0001 does not mean treatment is worth the
  cost!

33

• Incidence of vomiting 2.9 (32 during 1115
  feedings)
• vs 5.4 (67 during 1248 feedings), p0.001
• Q Whats wrong with this analysis?

34
Incidence of Bacteriuria in Patients using Clean
Intermittent Catheterization (CIC) vs Ileal Loop
Diversion (ILD)
P lt 0.001
Pediatrics 1982 70665-9.
Q Whats wrong with this analysis?
35
Factors Associated with Resident LP Success

• Of 377 performed by trainees, 279 (74) were
  successful.
• LPs were 3 times more likely to be successful
  among infants gt12 weeks of age than among younger
  infants (odds ratio OR 3.1 95 confidence
  interval CI 1.28.5).
• Success rate 70 vs 88

Pediatrics 2006117876
Q Were LPs 3 times more likely to be successful
on older infants?
36
Factors Associated with Resident LP Success
Pediatrics 2006117876
37
RR vs OR

• RR is RISK of (disease/side effects/etc) in
  study population divided by the risk in the
  comparison population
• OR is ODDS of (disease/side effects/etc) in study
  population divided by odds in control population
• When the outcome is rare, the OR approximates the
  RR when the outcome is common (gt10), the OR
  overestimates RR

38
CONVENTION

• Use risk or odds of something BAD, and put
  comparison group on bottom
• RR or OR gt 1 means exposure is bad
• RR or OR lt 1 means exposure is good
• RR or OR 1 means no relationship

39
(No Transcript)
40
Understanding regression coefficients
41
What is wrong with this table?
Pediatrics, Apr 2009 123 1108 - 1115
42
Coefficients

• Explain in words what the OR of 2.23 for birth
  weight and survival means

PEDIATRICS Vol. 123 No. 2 February 2009, pp.
494-502
43
Beta Coefficients
PEDIATRICS Vol. 122 No. 5 November 2008
44
Beta Coefficients

• Linear regression
• Like correlation coefficient
• No units
• Number of SD changes in y per SD change in x
• Logistic regression
• Logarithm of odds ratio
• Both
• Hard to interpret!

45
Interpreting Coefficients

• What do these coeffi-cients mean?

Pediatrics, Nov 2008 122 1051
46
Miscellaneous Issues
47
Predictors of Ventilator Associated Pneumonia
(VAP)
N19 cases of VAP. Any problem?
Pediatrics, Apr 2009 123 1108 - 1115
48
Overfitting
49
Confidence Intervals

• Extended Follow-up of Long-Term Survivors of
  Childhood Acute Lymphoblastic Leukemia
• The death rate for the irradiated group slightly
  exceeded the expected rate in the general U.S.
  population (standardized mortality ratio, 1.90
  95 percent confidence interval, 1.12 to 3.00),
  whereas that for the nonirradiated group did not
  differ from the population norm (standardized
  mortality ratio, 1.75 95 percent confidence
  interval, 0.34 to 5.00).

N Engl J Med 2003 349640-649
Q Do the data suggest the irradiated group had
worse survival?
50
Unknowns
51
Unknown 1
Number () of children with temperature 38ºC
after receiving DTaP vs DTP vaccine
P-value for 5/252 vs. 5/65 0.032
Peds ID 199312131-5

• Q What is wrong with how the P-value was
  obtained?

52
Unknown 2 The Beneficial Effects of Early
Dexamethasone Administration in Infants and
Children with Bacterial Meningitis

• Q How did they calculate the relative risk?
• At follow-up examination 7 of the 51
  dexamethasone-treated children (14 percent) and
  18 of 48 controls (38 percent) had one or more
  neurologic or audiologic sequelae (P 0.007)
  the relative risk of sequelae for a child
  receiving placebo as compared with a child
  receiving dexamethasone was 3.8 (95 percent
  confidence interval, 1.3 to 11.5).

N Engl J Med 1991 3241525-31
53
Unknown 3 Randomized trial of Antibiotic
Prophylaxis after Acute Pyelonephritis

• ARR in renal scars - 5.6 (95 CI -14.6 to
  3.3)

Q Based upon the 95 CI for the ARR, if
treatment actually worked, what is the lowest
number of children with reflux youd need to
treat to prevent scarring in one child?
Pediatrics 2006117629
54
Unknown 4 Impact of a diabetes prevention
program on fasting capillary glucose
There was a significant difference in change in
fasting capillary glucose levels in
intervention and control school students
(adjusted difference, 2.24 mg/dL 95 CI, 6.53
to 2.05 mg/dL p 0.03).
Arch Pediatr Adolesc Med 2004 158911-8
Q What is wrong here?
55
Unknown 5 Odds ratios for lifetime suicide
attempts

• What is the meaning of the OR for sex?

Arch Pediatr Adolesc Med. 2008 Nov162(11)1015-21

56
Unknown 6 Effects of Iron Therapy on
Developmental Scores of Iron-Deficient Infants
Q Does this analysis show that iron therapy
improves outcomes?
J Pediatr. 19789221-5
57
Slides and handouts available atwww.epibiostat.u
csf.edu/epidem/personnel/newman_document_repositor
y

• (Or, just google Thomas B. Newman)

58
Dexamethasone Calculations

• Normally, control group on bottom they
  arrived at their answers by putting it on top
• RR 37.5 2.7
• 14
• OR 18 44 3.8
• 7 30