Meta-Analysis Critique

1
Meta-Analysis Critique

• The Efficacy of Relapse Prevention on Substance
  Use Disorders
• Ashlee Carter 11/7/05

2
Reference

• Irvin, J.E., Bowers, C.A., Dunn, M.E.,
• Wang, M.C. (1999).
• Efficacy of Relapse Prevention A Meta-Analytic
  Review.
• Journal of Consulting and Clinical Psychology,
  67, 563-570.

3
Research Domain

• Clinical Psychology
• Therapeutic Interventions
• Substance Use Disorders
• Alcohol Use
• Cocaine Use
• Smoking (Tobacco)
• Polysubstance Use

4
Central Questions

• What is the effect of RP on substance use
  behavior
• What is the effect of RP on overall psychosocial
  adjustment?

5
Potential Moderating Variables

• Class of substance use disorder
• Treatment modality
• Setting of treatment
• Adjunctive use of medication
• Outcome measures
• Comparative efficacy of RP
• Length of posttreatment follow-up

6
Methods

• Study identification
• Computerized databases
• Reference lists
• Contact prominent researchers to request
  unpublished data
• Inclusion/Exclusion criteria
• Result 26 studies, 70 hypothesis tests, 9,504
  participants

7
Methods

• Main effect sizes converted to r
• Mini meta-analysis within studies
• r transformed to Fishers z
• Weighted average z score calculated
• z converted back to r
• Aggregate r across studies
• 95 CI calculated
• Weighted average r

8
Results

• Overall treatment effect of RP interventions for
  alcohol use disorders, substance use disorders,
  and smoking is r .14 (95 CI .10 .17, n
  22).
• Overall treatment effect of RP interventions on
  improving psychosocial adjustment is r .48 (95
  CI .42 to .53, n 10).

9
Results File drawer bias?

• Fail-safe n 388 (p 0.05)
• Number of undiscovered studies averaging no
  effect of RP interventions that would be required
  to reduce the obtained relationship to zero
• Conclusion The main effect is tolerant to future
  undiscovered null results

10
Results Moderator Analysis

• Contrast analyses
• Z scores
• Problems
• Small sample sizes
• Differences in effect sizes due to class of
  substance use disorder
• Medication varies

11
Results Moderating Variables

• 1) More effective in treating alcohol and
  polysubstance use than smoking or cocaine use.
• 2) Equally effective in group and individual
  formats.
• 3) Equally effective in outpatient and inpatient
  formats.
• 4) More effective with adjunctive medication than
  without.
• 5) Reported more effective in studies that used
  self-report versus biochemically validated
  self-report measures.

12
Results Moderating Variables

• 6) Effective when evaluated with pretest-posttest
  designs and when compared to physician advice
  only moderately effective when compared with
  psychoeducational groups or discussion controls.
• 7) Treatment effects were largest when outcomes
  were assessed immediately following treatment and
  tended to become smaller as the length of
  follow-up increased.

13
Conclusions Done well

• Large sample of studies
• Appropriate methods
• Weighted averaged r
• Mini meta-analysis in z
• Methods clearly outlined
• Formulas included
• Table
• Original statistic included
• Coding for moderator clearly described

14
Conclusions Done well

• Dot Plot
• General progression of effect size
• No gaps Published and unpublished studies
• File drawer bias addressed
• Statistics consistently presented
• Limitations admitted

15
Conclusions Questionable?

• Study identification method
• Publication bias?
• Study rigor?
• Moderator analyses
• Small sample sizes
• Adjustment of alpha levels?
• Interpretable?