Effects of lack of independence in meta-epidemiology

1
Effects of lack of independence in
meta-epidemiology

• Peter Herbison
• Preventive and Social Medicine
• University of Otago

2
The problem

• Median number of trials in a meta-analysis in the
  Cochrane Library is 2-3.
• In spite of this many of these reviews make quite
  strong recommendations.
• Are they justified in making these
  recommendations?

3
What we wanted to do

• Used an existing data set that has 65
  meta-analyses from 18 systematic reviews that was
  collected for another purpose
• Using cumulative meta-analysis we looked at what
  the answer was after the first three and the
  first five studies and compared this with the
  answer from all the studies (final answer)

4
Referees

• Paper came back from the journal saying that it
  was a good idea but they were not certain if
  using multiple outcomes from the same systematic
  review was reasonable
• Most similar meta-epidemiology studies only
  select one outcome from each systematic review
• This would leave us with only 18 results

5
Lack of independence

• I find it hard to imagine that this lack of
  independence will influence how quickly results
  settle down
• Especially since there is often a different mix
  of studies for the different outcomes
• One referee suggested a sensitivity analysis
  using one outcome from each review

6
Bootstrapping

• Why just randomly choose one outcome from each
  review when you can do this repeatedly?
• Using strata and size in the bootstrap command
• This should give some idea whether the lack of
  independence is important or not

7
Results

• Does the confidence interval include the final
  value?

Estimate Binomial 95 CI Bootstrap 95 CI
After 3 studies 68.8 57.1 80.4 56.8 80.7
After 5 studies 75.0 64.1 85.9 62.1 87.9
8
Results

• Does the confidence interval overlap with that of
  the final value?

Estimate Binomial 95 CI Bootstrap 95 CI
After 3 studies 75.0 64.1 85.9 66.3 83.7
After 5 studies 81.2 71.5 91.1 70.6 91.7
9
More traditional meta-epidemiology

• Use the same data set to see if lack of
  allocation concealment is associated with bias.
• Assuming independence
• ROR 0.91 (95CI 0.84 0.98)
• Bootstrap
• ROR 0.91 (95CI 0.83 0.99)

10
Conclusion

• In this data set at least, lack of independence
  does not seem to make much difference.