Metaanalyses, Systematic Reviews and Holms Levels

1
Meta-analyses, Systematic Reviews and Holms
Levels

• OT 653
• Fall 2004

2
Steps in determining the Validity of a
Meta-Analysis
3

• 1. Was the literature search done well?
  Yes No
• a. Was it comprehensive?
  Yes No
• b. Were the search methods systematic and
  clearly described? Yes No
• c. Were the key words used in the search
  described? Yes
  No
• d. Was the issue of publication bias
  addressed? Yes No
• 
  
• 2. Was the method for selecting articles clear,
  systematic, and appropriate? Yes No
• a. Were there clear, pre-established inclusion
  and exclusion criteria
• for evaluation?
  
  Yes No
• b. Was selection systematic? Yes No
• Was the population defined? Yes No
• Was the exposure/intervention clearly described?
  Yes No
• Were all outcomes described and were they
  comparable? Yes No
• c. Was the selection process reliable?
  Yes No
• Were at least two independent selectors
  used? Yes No
• Was the extent of selection disagreement
  evaluated? Yes No
• 3. Was the quality of the primary studies
  evaluated?
  Yes No
• a. Did all studies, published or not,
  have the same standard applied? Yes
  No
• b. Were at least two independent
  evaluators used, and was the

4

• 4. Were results from the studies combined
  appropriately?
• a. Were the studies similar enough to
  combine results? (Were the
• study designs, populations, exposures,
  outcomes, and direction of effect
• similar in the combined studies?)
  
  Yes No
• b. Was a test for heterogeneity done and
  was its P value nonsignificant? Yes No
• 5. Was a statistical combination (meta-analysis)
  done properly?
• a. Were the methods of the studies
  similar?
  Yes No
• b. Was the possibility of chance
  differences statistically addressed?
• (Was a test for homogeneity done?)
  
  Yes No
• c. Were appropriate statistical analyses
  performed?
  Yes No
• d. Were sensitivity analyses used?
  
  Yes No
• 6. Are the results important?
• a. Was the effect strong?
  
  Yes No
• Was the odds ratio large?
  
  Yes No
• Were the results reported in a
  clinically meaningful manner, such
• as the absolute difference or the
  number needed to treat?
  Yes No
• b. Are the results likely to be
  reproducible and generalizable?
  Yes No
• c. Were all clinically important
  consequences considered?
  Yes No
• d. Are the benefits worth the harm and
  costs?
  Yes No

5
Appraising Systematic Reviews
6
(No Transcript)
7
Holms Levels of Evidence

• Level I - Systematic reviews, meta-analytic
  studies
• Level II - Randomized controlled trials
• Level III - Trials without randomization
• Level IV - Nonexperimental studies from more than
  one center
• Level V - Opinions of respected authorities based
  on clinical evidence, descriptive studies or
  reports of expert committees
• (Holm, 2000, p.581)