Metaanalysis

1
Meta-analysis

• The EBM workshop
• A.A.Haghdoost, MD PhD of Epidemiology
• Ahaghdoost_at_kmu.ac.ir

2
Definition

• Meta-analysis a type of systemic review that
  uses statistical techniques to quantitatively
  combine and summarize results of previous
  research
• A review of literature is a meta-analytic review
  only if it includes quantitative estimation of
  the magnitude of the effect and its uncertainty
  (confidence limits).

3
Function of Meta-Analysis(1)

• 1-Identify heterogeneity in effects among
  multiple studies and, where appropriate, provide
  summary measure
• 2-Increase statistical power and precision to
  detect an effect
• 3-Develop ,refine, and test hypothesis
• continued

4
Function of Meta-Analysis(2)

• continuation
• 4-Reduce the subjectivity of study comparisons by
  using systematic and explicit comparison
  procedure
• 5-Identify data gap in the knowledge base and
  suggest direction for future research
• 6-Calculate sample size for future studies

5
Historical background

• Ideas behind meta-analysis predate Glass work by
  several decades
• R. A. Fisher (1944)
• When a number of quite independent tests of
  significance have been made, it sometimes happens
  that although few or none can be claimed
  individually as significant, yet the aggregate
  gives an impression that the probabilities are on
  the whole lower than would often have been
  obtained by chance (p. 99).
• Source of the idea of cumulating probability
  values
• W. G. Cochran (1953)
• Discusses a method of averaging means across
  independent studies
• Laid-out much of the statistical foundation that
  modern meta-analysis is built upon (e.g., inverse
  variance weighting and homogeneity testing)

6
Basic concepts

• The main outcome is the overall magnitude of the
  effect.
• It's not a simple average of the magnitude in all
  the studies.
• Meta-analysis gives more weight to studies with
  more precise estimates.
• The weighting factor is 1/(standard error)2.

7
Main magnitude of effects

• Descriptive
• Mean
• Prevalence
• Analytical
• Additive
• Mean difference
• Standardized mean difference
• Risk, rate or hazard difference
• Correlation coefficient
• Multiplicative
• Odds ratio, Risk, Rate or Hazard Ratio

8
Statistical concepts(1)

• You can combine effects from different studies
  only when they are expressed in the same units.
• Meta-analysis uses the magnitude of the effect
  and its precision from each study to produce a
  weighted mean.

9
Statistical concepts(2)
The impact of fish oil consumption on
Cardio-vascular diseases
10
Forest plot

• the graphical display of results from individual
  studies on a common scale is a Forest plot.
• In the forest plot each study is represented by a
  black square and a horizontal line (CI95).The
  area of the black square reflects the weight of
  the study in the meta-analysis.
• A logarithmic scale should be used for plotting
  the Relative Risk.

11
Forest plot
12
Statistical concepts(3)

• There are two basic approach to Quantitative meta
  analysis
• Weighted-sum
• Fixed effect model
• Random effect model
• Meta-regression model

13
Fixed effect model

• General Fixed effect model- the inverse variance
  weighted method
• Specific methods for combining odds ratio
• Mantel- Haenszel method
• Petos method
• Maximum-Likelihood techniques
• Exact methods of interval estimation

14
Fixed effect model

• In this model, all of the observed difference
  between the studies is due to chance
• Observed study effectFixed effect error
• Xi ? ei ei is N (0,d2 )
• Xi Observed study effect
• ? Fixed effect common to all studies

15
General Fixed effect model

• T? wiTi/ ? wi
• The weights that minimize the variance of T are
  inversely proportional to the conditional
  variance in each study
• Wi1/vi
• Var(T)1/ ? wi

16
Mantel- Haenszel method

• Each study is considered a strata.
• T?ai di / ni / ?bi ci /ni

17
Random effect model

• The random effect model, assumes a different
  underlying effect for each study.
• This model leads to relatively more weight being
  given to smaller studies and to wider confidence
  intervals than the fixed effects models.
• The use of this model has been advocated if there
  is heterogeneity between study results.

18
Source of heterogeneity

• Results of studies of similar interventions
  usually differ to some degree.
• Differences may be due to
• - inadequate sample size
• - different study design
• - different treatment protocols
• - different patient follow-up
• - different statistical analysis
• - different reporting
• - different patient response

19

• An important controversy has arisen over whether
  the primary objective a meta-analysis should be
  the estimation of an overall summary or average
  effect across studies (a synthetic goal)
• or the identification and estimation of
  differences among study-specific effects
  (analytic goal)

20
Test of Homogeneity

• This is a test that observed scatter of study
  outcomes is consistent with all of them
  estimating the same underlying effect.
• Q X2homo?i1nwi (mi -M)2
• dfn-1
• wi weight
• Mmeta analytic estimate of effect
• mi effect measure of each study

21
Dealing with statistical heterogeneity

• The studies must be examined closely to see if
  the reason for their wide variation in effect. If
  its found the analysis can be stratified by that
  factor.
• Subgroup analysis
• Exclusion of study
• Choose another scale
• Random effect model
• Meta-regression

22
Random effect model

• Assume there are two component of variability
• 1)Due to inherent differences of the effect being
  sought in the studies (e.g. different design,
  different populations, different treatments,
  different adjustments ,etc.) (Between study)
• 2)Due to sampling error (Within study)

23
Random effect model

• There are two separable effects that can be
  measured
• The effect that each study is estimating
• The common effect that all studies are estimating
• Observed study effectstudy specific (random
  )effect error

24
Random effect model

• This model assumes that the study specific effect
  sizes come from a random distribution of effect
  sizes with a fixed mean and variance.
• There are five approach for this model
• Weighted least squares
• Un-weighted least squares
• Maximum likelihood
• Restricted Maximum likelihood
• Exact approach to random effects of binary data.

25
Random effect

• Xi ?i ei ei is N (0,d2 )
• Xi Observed study effect
• ?i Random effect specific to each study ?i
  Udi
• UGrand mean (common effect)
• di is N (0, ?2 ) Random term

26
Weighted least squares for Random Effect

• W?wi/k
• S2w1/k-1(?wi2-k W2)
• U(k-1)(W-S2w/kW)
• ?20 if Qltk-1
• ?2(Q-(k-1))/U if Qgtk-1
• wi 1/var. ?2 var.within study
  variances

27
Weighted least squares for Random Effect (WLS)

• T.RND? wi Ti/ ? wi
• Var(T.RND)1/ ? wi
• Where Ti is an estimate of effect size and ?i is
  the true effect size in the ith study
• Ti ?i ei ei is the error with which Ti
  estimates ?i
• var(Ti) ??2 vi

28
random versus fixed effect models

• Neither fixed nor random effect analysis can be
  considered ideal.
• Random effect models has been criticized on
  grounds that unrealistic distributional
  assumption have to be made.
• Random effect models are consistent with the
  specific aims of generalization.

29
Petos advocates

• He suggested a critical value .01 instead of
  usual .05 to decide whether a treatment effect is
  statistically significant for a fixed effect
  model.
• This more conservative approach has the effect of
  reducing the differences between fixed and random
  effect models.

30
Meta-regression

• If more than two groups of studies have been
  formed and the characteristic used for grouping
  is ordered, greater power to identify sources of
  heterogeneity may be obtained by regressing study
  results on the characteristic .
• With meta-regression, it is not necessary or even
  desirable to groups the studies.
• The individual study results can be entered
  directly in the analysis.

31
Meta-Regresion

• 1- meta-Regression model( extension of fixed
  effect model)
• 2- Mixed model( extension of random effect model)

32
Fixed-effects regression

• TiB0B1xi1...Bpxip
• Its the covariate predictor variables that are
  responsible for the variation not a random
  effect the variation is predictable, not random.

33
Mixed model

• TiB0B1xi1...Bpxipui
• This model assumes that part of the variability
  in true effects is unexplainable by the model.

34
Between studies variation

• You can and should allow for real differences
  between studiesheterogeneityin the magnitude of
  the effect.
• The t2 statistic quantifies of variation due to
  real differences.

35
Fixed effects model and heterogeneity

• In fixed-effects meta-analysis, you do so by
  testing for heterogeneity using the Q statistic.
• If plt0.10, you exclude "outlier" studies and
  re-test, until pgt0.10.
• When pgt0.10, you declare the effect homogeneous.
• But the approach is unrealistic, limited, and
  suffers from all the problems of statistical
  significance.

36
Random effects model and heterogeneity

• In random-effect meta-analysis, you assume there
  are real differences between all studies in the
  magnitude of the effect.
• The "random effect" is the standard deviation
  representing the variation in the true magnitude
  from study to study.
• You need more studies than for traditional
  meta-analysis.
• The analysis is not available in a spreadsheet.

37
Concept of analysis in random versus fixed effect
models

• Fixed effects models within-study variability
• "Did the treatment produce benefit on average in
  the studies at hand?"
• Random effects models between-study and
  within-study variability
• "Will the treatment produce benefit on
  average?"

38
Limitations

• It's focused on mean effects and differences
  between studies. But what really matters is
  effects on individuals.
• (Aggression bias)
• A meta-analysis reflects only what's published or
  searchable.

39
Aggregation bias

• Relation between group rates or and means may not
  resemble the relation between individual values
  of exposure and outcome.
• This phenomenon is known as aggregation bias or
  ecologic bias.

40
Ecological fallacy
41
Meta-analysis of neoadjuvant chemotherapy for
cervical cancer
42
Type of reporting
43
Selection bias in Meta analysis

• English language bias
• Database bias
• Publication bias
• Bias in reporting of data
• Citation bias
• Multiple publication bias
• Sample size

44
Publication bias

• The results of a meta-analysis may be biased if
  the included studies are a biased sample of
  studies in general.
• The classic form of this problem is publication
  bias, a tendency of journals to accept
  preferentially papers reporting an association
  over papers reporting no association

45
Publication bias

• If such a bias is operating, a meta-analysis
  based on only published reports will yield
  results biased away from the null.
• Because small studies tend to display more
  publication bias, some authors attempt to avoid
  or minimize the problem by excluding studies
  below a certain size.

46

• Some meta-analysts present the effect magnitude
  of all the studies as a funnel plot, to address
  the issue of publication bias.
• A plot of 1/(standard error) vs effect magnitude
  has an inverted funnel shape.
• Asymmetry in the plot can indicate
  non-significant studies that werent published.

47
Funnel plot
48
Funnel plot
49
Measures of Funnel Plot Asymmetry

• 1- Linear Regression Approach (Eggers method)
• SNDa b. precision
• SNDOR/SE
• The intercept a provides a measure of
  asymmetry- the larger its deviation from zero the
  more pronounced the asymmetry.

50
Measures of Funnel Plot Asymmetry

• 2- A rank correlation test
• This method is based on association between
  the size of effect estimates and their variance.
  If publication bias is present, a positive
  correlation between effect size and variance
  emerges because the variance of the estimates
  from smaller studies will also be large.

51
Funnel plot
52
Key Messages

• Funnel plot asymmetry was found in 38 of
  meta-analyses published in leading general
  medicine journals and in 13 of reviews from the
  Cochrane Database of Systematic Reviews.
• Critical examination of systematic reviews for
  publication and related biases should be
  considered a routine procedure.

53
Sources of Funnel Plot asymmetry

• Selection Bias
• True Heterogeneity
• Size of effect differs according to study size
• Intensity of interventions
• Difference on underlying risk
• Data irregularities
• Poor methodological design of small studies
• Inadequate analyses
• Fraud
• Artefactual
• Choice of effect measure
• Chance

54
Sample size as source of bias

• Consider a hypothetical literature summary
  stating, of 17 studies to date, 5 have found a
  positive association,11 have found no
  association, and 1 has found a negative
  association thus, the preponderance of evidence
  favors no association.
• Mere lack of power might cause most or all of the
  study results to be reported as null.

55
Quality score

• Some meta-analysts score the quality of a study.
• Examples (scored yes1, no0)
• Published in a peer-reviewed journal?
• Experienced researchers?
• Research funded by impartial agency?
• Study performed by impartial researchers?
• Subjects selected randomly from a population?
• Subjects assigned randomly to treatments?
• High proportion of subjects entered and/or
  finished the study?
• Subjects blind to treatment?
• Data gatherers blind to treatment?
• Analysis performed blind?

56
Quality score

• Use the score to exclude some studies, and/or
• Include as a covariate in the meta-analysis, but
• Some statisticians advise caution when using
  quality.

57
Quality scoring

• A very common practice is to weight studies on a
  quality score usually based on some subjective
  assignment .
• For example, 10 quality points for a cohort
  design, 8 points for a nested case control
  design, and 4 points for a population based case
  control design.

58
Quality scoring

• Quality scoring submerges important information
  by combining disparate study features into a
  single score.
• It also introduces an unnecessary and somewhat
  arbitrary subjective element in to the analysis.

59
Quality scores as weighing factors

• study weight1/var.
• Quality adjusted weight quality score /var.

60
Quality scores

• The judgment that the studies should or should
  not be combined should be stated and justified
  explicitly.
• There is some of a tendency to make this judgment
  on the basis of the quantitative results, but
  its critical to make a qualitative judgment.

61
What is an IPD Meta-analysis?

• Involves the central collection, checking and
  analysis of updated individual patient data
• Include all properly randomised trials, published
  and unpublished
• Include all patients in an intention-to-treat
  analysis

62
IPD Meta-analysis

• Individual patient data used
• Analysis stratified by trial
• IPD does not mean that all patients are combined
  into a single mega trial

63
IPD Analyses

• Collect raw data from related studies, whether or
  not the studies collaborated at the design stage,
  exposures measures and other covariates that can
  be applied uniformly across the studies combined.
• The major advantage of a IPD over an MA is the
  use of individual-based rather than group-based
  data.

64
sensitivity analysis

• In sensitivity analysis, the sensitivity of
  inference to variations in or violations of
  certain assumptions is investigated.
• For example, the sensitivity of inference to the
  assumption about the bias produced by failure to
  control for smoking can be checked by repeating
  the meta-analysis using other plausible values of
  the bias.

65
sensitivity analysis

• If such reanalysis produces little change in an
  inference, one can be more confident that the
  inference is insensitive to assumptions about
  confounding by smoking.
• In influence analysis, the extent to which
  inferences depend on a particular study or group
  of studies is examined this can be accomplished
  by varying the weight of that study or group.

66
sensitivity analysis

• Thus , in looking at the influence of a study,
  one could repeat the meta-analysis without the
  study, or perhaps with half its usual weight .
• If change in weight of a study produces little
  change in an inference, inclusion of the study
  can not produce a serious problem, even if
  unquantified biases exist in the study

67
Sensitivity and influence analysis

• On the other hand, if an inference hinges on a
  single study or group of studies, one should
  refrain from making that inference

68
conclusion

• Most meta-analysis will require from each study
  both a point estimate of effect and an estimate
  of its standard error .
• A point estimate accompanied only by a P value
  will generally not provide for accurate
  computation of a standard error estimate, and
  should not be considered sufficient for reporting
  purposes.

69
Over conclusion

• Like large epidemiologic studies, meta-analysis
  run the risk of appearing to give results more
  precise and conclusive that warranted.
• The lager number of subjects contributing to a
  meta-analysis will often lead to very narrow
  confidence intervals for the effect estimate.