When is a review systematic

1
When is a review systematic?

• Contains an explicit statement of objectives,
  materials, and methods
• Has been conducted using specific methodology
• Assessment of the quality of the literature
  quantitative (meta-analysis) or subjective
  (systematic review)
• meta-analysis statistical synthesis of the
  numerical results of several trials that all
  examined the same question

2
What is the purpose of the systematic review?

• State of the art literature review, but utilizes
  techniques in article selection and evaluation to
  limit bias contradistinction to the
  traditional narrative review
• Provides a logical framework for appraisal of the
  literature
• comparable studies, similar measures, effect
  measures are combined
• Reviews which are alive allow modification
  incorporating new data as it becomes available
• It has been suggested that prior to any new
  research endeavor, a meta-analysis of the
  literature should be performed based on the
  quality/availability of current literature
• Gain in statistical power/improved precision for
  estimates

3
Greenhalgh, T. How to Read a Paper the basics of
EBM
4
Review of Meta-analyses Users Guide to
ReviewCook et al. CCM 1998 26(4) 692-700

• Are the results of the review valid?
• Did the review address a focused clinical
  question
• were the criteria used to select articles
  appropriate
• is it likely that NB relevant studies were missed
• was the validity of the included articles
  appraised
• were assessments of the included studies
  reproducible
• were the results similar from study to study
• What are the results?
• What are the overall results of the review
• how precise were the results
• Will the results help me in caring for my
  patients?
• Can the results be applied to patients in my care
• were all clinically important outcomes considered
• are the benefits worth the harm and costs

5
Review of Meta-analyses Users Guide to
ReviewGreenhalgh, T. BMJ 1997 315672-5

• Are the results of the review valid?
• State objectives of the review of RCTs and
  outline eligibility criteria
• search for trials that meet eligibility criteria
• tabulate characteristics of each trial and assess
  methodologic quality (methodologic quality,
  precision, external validity)
• justify any exclusions
• assemble the most complete dataset ask PI for
  data
• statistically synthesize data where appropriate
• compare alternative analyses

6
Evaluating systematic reviewsA focused clinical
question?

• The question should be defined very precisely
  similar to the hypothesis or question in any
  clinical study. Needs to be very specific to
  allow investigator to objectively select
  articles for inclusion. For example
• Do anticoagulants prevent strokes in patients
  with atrial fibrillation? YES OR NO IS THE
  QUESTION SPECIFIC?
• Does AF include rheumatic and nonrheumatic
  causes?
• Does it include intermittent/paroxysmal AF?
• Does stroke include hemorrhagic and
  nonhemorrhagic?

7
Evaluating systematic reviews A focused
clinical question?

• To assess the effectiveness and safety of
  warfarin type anticoagulant therapy compared to
  placebo in secondary prevention of stroke in
  patients with non-rheumatic atrial fibrillation.

8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
Search for trials that meet eligibility criteria
What are the potential sources of data?

• Databases
• problem, only indexed journals
• publication bias of positive studies
• Bibliographies
• selection bias of previous authors
• Foreign medical literature
• Granting agencies
• Industry sponsors
• Experts in the area
• Raw data
• Grey literature (ads, non medical literature)

12
Evaluating systematic reviews Search for trials
that meet eligibility criteria What are the
potential sources of data?

• Knipschild and colleagues Vitamin C and the
  common cold
• search of electronic databases identified 22
  articles
• Another 39 articles by hand searching Index
  Medicus
• Another 15 trials by searching biblios
• Another 9 trials by searching biblios of the
  second 15
• One more by searching biblios of the biblios of
  the biblios

13
(No Transcript)
14
Were the criteria used to select articles
appropriate?

• justify exclusions
• is it likely that NB
• relevant studies were
• missed

15
Were the criteria used to select articles
appropriate?
16
Evaluating systematic reviewsWas methodological
quality assessed and the trials weighted
accordingly?

• Few published papers can be outright rejected
  based on concern about methodologic quality.
• What weight do you give a small study of
  perfection versus a large study that has
  methodologic flaws?
• What is the gold standard for the assessment of
  methodologic quality?

17
Was the validity of the included articles
appraised?
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
Cook et al. Endoscopic therapy for acute
nonvariceal upper gastrointestinal hemorrhage a
meta-analysis. Gastroenterology 1992 102 139-48.
22
Tabulate characteristics of each trial and assess
methodologic quality (methodologic quality,
precision, external validity) Have the
numerical results been interpreted with common
sense and due regard to the broader aspects of
the problem?

• Tabulate relevant information on inclusion
  criteria, sample size, characteristics of
  patients at baseline, withdrawal rate, results of
  primary and secondary end points
• What is the appropriate outcome measure?
• Presentation should be in a standard format
• Forest plot
• Pooled Odds ratio

23
(No Transcript)
24
(No Transcript)
25
(No Transcript)
26
Assess methodologic quality precision
27
(No Transcript)
28
Heterogeneity

• Explanation of homogeneity and heterogeneity
• homogeneity results of each individual trials
  are compatible with the results of any others
  (estimating at a glance)
• heterogeneity and statistical testing is there
  greater variation between the results of the
  trials than is compatible with the play of
  chance
• Potentially provides the opportunity to explore
  if there are new associations between effector
  and outcome based on composition of variables
  within the study samples, levels of exposure,
  etc.

29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
Heterogeneity

• Chi-square test rule of thumb chi-square has a
  value equal to its degree of freedom minus 1 ie.
  A meta-analysis with 8 trials, and a chi-square
  7 would provide no evidence of statistical
  heterogeneity
• Examine the confidence regions
• plt.10 identifies significance
• A high chi-square value indicates that there is
  some important heterogeneity, but what is it?
  Explaining clinical heterogeneity is the role of
  the author not the statistician

33
Heterogeneity

• You wont understand the statistics!
• Do you believe there is significant variability
  between studies
• Fixed effects models assume that there is no
  heterogeneity, assume there is the existence of a
  single effect of exposure common to all studies
• Random effects models assume that existing
  studies are a random sample of a population of
  studies, and therefore they include a measure of
  this effect
• Important in cases where there is significant
  heterogeneity a random-effects model will have
  wider confidence intervals (there is should be no
  effect on point estimate)

34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Appraisal of Systematic Reviews
40
Problems with meta-analysis!

• The most important maxim for data analysts to
  heed, and one which many statisticians have
  shunned is this far better an approximate answer
  to the right question, which is often vague, than
  an exact answer to the wrong question, which can
  always be made precise.

41
Problems with meta-analysis!

• Combinability of studies
• Reliability and comparisons with RCTs
• Bias inherent in meta-analysis
• Is meta-analysis research?
• Setting policy with meta-analysis.

42
JAMA 1998 280 278-80.
43
Methodology and Reports of Systematic Reviews and
Meta-analysesJadad AR, et al. JAMA 1998 280
278-80.

• Purpose compare methodolgic quality of Cochrane
  reviews and those published in paper-based
  journals, evaluate frequency data are updated
• Methods all 36 Cochrane reviews, and 39 randomly
  selected paper based reviews from 32 different
  journals
• Results
• none of the Cochrane and only 3 of the paper
  described primary outcome of interest
• more trials were presented in paper based
  journals (13.5 v 5 medians), and more patients
  (1280 v 528)
• better description of inclusion criteria in
  Cochrane (35/36 v 18/36), and assessment of trial
  quality (36/36 v 18/39)
• no difference in heterogeneity testing, or the
  description of quantitative effect estimates

44
Problems with Meta-analysisDiscordant Systematic
ReviewsJadad AR et al. CMAJ 1997 156 1411-6.

• Is discordance important?
• Systematic reviews are becoming central to EBM
  (ahead of large trials)
• Discordance challenges essence of EBM
• Reviews can disagree in 2 ways
• divergent results
• direction of effect
• magnitude of effect
• statistical significance
• interpretations or inferences of the authors can
  be different

45
Problems with Meta-analysisDiscordant Systematic
ReviewsSources of Discordance

• Clinical Question
• population of patients
• interventions
• outcome measures
• settings
• Study selection and inclusion
• selection criteria
• application of the selection criteria
• strategies to search the literature
• Data Extraction
• methods to measure outcomes
• end points
• human error

• Assessment of Study Quality
• methods to assess quality
• interpretation of quality assessments
• methods to incorporate quality assessments
• Assessment of ability to combine studies
• statistical methods
• clinical criteria to judge ability to combine
  studies
• Statistical Methods for Data synthesis

46
Problems with Meta-analysisDiscordant Systematic
ReviewsSources of Discordance

• Are the reviews valid?
• Oxman AD et al. J Clin Epidemiol 1991 44
  1271-8.
• Are the differences among the discordant reviews
  important?
• Direction, or magnitude of effect?
• Will it change your decision?
• Do the reviews ask the same questions?
• Do the reviews include the same trials?

47
(No Transcript)
48
NEJM 1997 337 536-42
49
Discrepancies between Meta-analyses and Large
RCTsLeLorier J, et al. 1997 337 536-42.

• Concern is meta-analysis incorporates sources of
  bias already present within studies (usual bias
  which will over-estimate treatment effect), and
  adds new biases inherent to the systematic review
  process
• Previous retrospective study from Cochrane
  Database demonstrated that when largest trial was
  removed from meta-analysis, Kappa of remaining
  studies to largest study was only .46-.53.
• Objective to compare series of RCTs to
  previously published meta-analyses

50
Discrepancies between Meta-analyses and Large
RCTsLeLorier J, et al. 1997 337 536-42.

• Methods
• search NEJM, Lancet, JAMA, Annals for RCTs of
  gt1000 pts. Jan 1/91-Dec 31/94
• search of meta-analyses on similar topics
  published prior to the RCTs
• selected articles with similar populations and
  primary and secondary outcomes
• excluded underpowered studies
• Results
• 9 tx groups, 12 RCTs 19 meta-analyses
• 40 outcomes assessed

51
Discrepancies between Meta-analyses and Large
RCTsLeLorier J, et al. 1997 337 536-42.

• Tx Groups
• acute MI tx with late STK
• acute MI tx with IVMG
• CHD and Hyperchol tx with drugs
• Tx children with VitA
• CHF tx with ACEI
• Major abdominal sx tx with LMWH
• women at risk of PIH tx with ASA
• Acute MI tx with Ntg
• Breast CA tx with adjuvant drugs
• BP control and electrolytes
• Smoking cessation and interventions

52
ACUTE MI AND LATE STK
1
FAVORS CONTROL
FAVORS TREATMENT
53
CVD AND HYPERCHOLESTEROLEMIA
1
FAVORS TREATMENT
FAVORS CONTROL
54
VITAMIN A SUPPLEMENTATION
1
FAVORS TREATMENT
FAVORS CONTROL
55
(No Transcript)
56
(No Transcript)
57
Discrepancies between Meta-analyses and Large
RCTsLeLorier J, et al. 1997 337 536-42.

• Good Agreement
• MG and MI
• hyperchol and CVD
• Vit A in developing countries
• ACE and CHF
• Adjuvant chemotx in Breast CA
• Smokers tx with multiple interventions

• Considerable divergence
• late thrombolysis
• ASA and PIH
• NTG and MI
• ASA and PIH
• BP control and electrolyte disturbance

58
Discrepancies between Meta-analyses and Large
RCTsLeLorier J, et al. 1997 337 536-42.

• If there had been no RCT ineffective therapy
  would have been adopted in 32 of cases, and
  useful therapy rejected in 33 of cases
• 46 of the divergences were a postive
  meta-analysis followed by a negative RCT
• publication bias favors postive studies
• smaller studies with poor design may increase
  estimate of tx effect
• 54 of divergences were negative meta-analysis
  followed by positive RCT
• may be caused by study heterogeneity not
  adequately explained for example subtle
  differences in sample selection, tx
  administration, and choice of concomitant therapy