Kristian Thorlund M'Sc'

1
The risk of random error in meta-analyses

• Kristian Thorlund M.Sc.
• Copenhagen Trial Unit
• Centre for Clinical Intervention Research
• Rigshospitalet

2
Definition and property

• Random error (due to play of chance) is the
  unpredictable variation between observed values
  and some true value
• Random errors tend to decrease as the information
  size increases

3
Variation and sample size
RR
q
1
Required sample size
Number of patients randomised
4
Meta-analysis context

• How does random error affect meta-analyses?
• Statistical significance
• Estimated treatment effects
• Heterogeneity estimates
• Everything we attempt to estimate may be subject
  to some degree of random error

5
Apparently conclusive meta-analyses

• Brok et al. 2009 looked at 25 systematic reviews
  (including 54 binary outcome meta-analyses) that
  recommended an intervention for clinical use

Applying meta-analysis information size
calculations and trial sequential analysis, 19
meta-analyses provided insufficient evidence
6
Spurious inferences

• Thorlund et al. 2009 investigated the risk of
  early (positive) spurious findings by looking at
  interim results from 33 meta-analyses that had
  surpassed their required information size
• Trikalinos et al. 2005 looked at 100 mental
  health meta-analyses to investigate how estimated
  effect sizes evolve as more trials are accrued

7
False positives

• Thorlund et al.
• In the interim, 3 of the 12 meta-analyses showing
  no effect were temporarily statistically
  significant before reaching their required
  information size

Trikalinos et al. In the interim, 8 of the 44
meta-analyses showing no effect were temporarily
statistically significant (no information size
considerations included)
8
Effect sizes
Thorlund et al. found false positive results
typically occured at an early stage

9
Effect sizes
Thorlund et al. When considering a treatment
effect statistically significant upon reaching a
P-value less than 5, 6 of the
21 meta-analysis yielded clinically important
overestimates of treatment effects at first
statistical significance

10
Effect sizes
Thorlund et al. When considering a treatment
effect statistically significant according to
trial sequential analysis, none of the 21
meta-analysis overestimated treatment effects at
first statistical significance

11
Effect sizes
Clinically important overestimates typically
occured at an early stage

12
Effect sizes

• Trikalinos et al.
• With 100 patients randomised, subsequent changes
  in odds ratios of 3- to 5-fold were common
• With 500 patients randomised, 1.5-fold changes
  were only observed in 5 of the meta-analyses
• With more than 2000 patients randomised,
  subsequent changes were unlikely

13
Heterogeneity

• Ioaniddis et al. evaluated the uncertainty of
  heterogeneity by looking at 95 confidence
  intervals for I2 estimates in 1011 Cochrane
  meta-analyses

14
Sum up

• Random error (due to play of chance) is the
  unpredictable variation between observed values
  and some true value
• Everything we attempt to estimate may be subject
  to some degree of random error

15
Sum up

• Random error affects
• Statistical significance
• Estimated treatment effects
• Heterogeneity estimates
• Only a sufficient number of trials and patients
  will ensure an acceptable risk of random error

16
Thank you!