Research Design in Clinical Psychology

1
Research Design in Clinical Psychology

• Lecture 9
• Statistical Methods of Data Analysis
• (Chapter 15 in Kazdin)

2
Statistical Significance

• When two groups are different, with the p value
  (a) chosen indicating the number of times out of
  100 that the difference between groups (rejecting
  null) is not real and instead found by chance

3
Effect size

• Magnitude of an effect
• Difference b/n group means / st dev (pooled)
• Provides a common metric to use across studies
• Also known as Cohens d

4
Power

• Likelihood of finding an effect if it exists
• Minimum of .8
• Calculated using sample size, effect size
  (Cohens d), and alpha
• Affected by w/n group variability
• Must provide adequate power for all primary
  analyses
• What about secondary analyses?

5
Ways to increase power

• Increased sample size
• Use conditions with big effects
• Use pretest or repeated measures to reduce error
  term in effect size
• Adding 1-r2 to denominator often increases ES
• A priori vary alpha and use directional tests
• Improve control to reduce w/n group variability

6
Key Considerations

• Do I have sufficient power given likely ES
• Can I increase power in any way
• Can I increase potency of effect of reduce w/n
  group variability
• Are all groups in the study necessary
• Will there be a need to further divide groups
  based on other variables of interest?

7
Intent-to-Treat

• All subjects are included, so data from last data
  point are used for drop-outs
• Could be baseline if drop-out occurs early
• Contrasted with completer analysis
• Attrition changes randomized study from
  experiment to quasi-experiment
• I-t-T is much more conservative, but likely a
  more pure analysis

8
Multiple Comparisons

• Controlling alpha levels
• To control for of group comparisons or of
  measures used
• Pairwise or percomparison error rate
• Chance finding for one comparison
• Experiment-wise error rate
• Chance finding for any comparison
• Add p for each comparison
• Goal is to keep E-w error rate at .05
• Bonferroni
• P value divided by of comparisons

9
Multivariate Analyses

• Need to take conceptual/empirical inter-relation
  among dvs into account
• Use multivariate analyses
• If inter-related measures are used, preferred to
  univariate analyses which do not take these
  structural issues into account
• Univariate analyses (controlling for EW-error)
  can be used following initial multivariate
  analysis

10
Concerns of significance tests

• Arbitrary criterion
• Binary decision on significance
• Cant accept hypotheses
• Significance a function of n
• Tests are very subjective
• Says nothing about strength or importance of
  effect

11
Misconceptions of significance tests

• p does not reflect
• Degree that the finding is true
• the strength of the effect
• Likelihood of replication
• Cant discuss trends or approaching significance
• Arbitrary standard for calling something a real
  and not-real effect

12
Alternatives and Supplements 1

• Magnitude of effect
• Many others beyond ES
• Confidence intervals
• Includes mean value (z value under normal curve
  St error of measurement)
• St error of measurement ? st dev / sqrt of n
• Provides range of values and likelihood that
  population falls within particular range
• supplements measure of magnitude of effect

13
Alternatives and Supplements 2

• Meta-Analysis
• Extension of ES to evaluate multiple studies
• Combine many ESs to give more stable estimate of
  population parameters
• Allow for quantitative review of literature