sources of bias in experiments and quasi-experiments

1
sources of bias in experiments and
quasi-experiments

• sean f. reardon
• stanford university
• 11 december, 2006

2
three populations

• population of interest (POI) the population for
  whom we would like to estimate the average
  treatment effect
• population of study (POS) the population of whom
  the study sample is representative
• population of causal inference (POC) the
  population for whom we can make a causal inference

3
tradeoffs between bias and generalizability

• we want to estimate the average treatment effect
  (?) in some population of interest (POI).
• ? tells us how much would should we expect the
  outcome Y of a person randomly chosen from P to
  differ depending on whether we assign them to T
  or C.
• but we only estimate ? in POC
• bias arises when POC is not the same as POI
• bias 1 ? POC is not the same as ? POS
• bias 2 ? POS is not the same as ? POI

4
strategies for minimizing bias

• randomized experiments
• we get unbiased estimate of ?POC
• and ?POC?POS, so RCT eliminates bias 1
• but bias 2 may be large or small (external
  validity)
• regression discontinuity
• we get unbiased estimate of ?POC (under weak
  assumptions)
• but at the cost of making ?POC? ?POS
• POC is generally small (POC is the region of the
  population near the discontinuity) relative to
  both POS and POI
• but sometimes the region around discontinuity is
  the population of interest

5
strategies for minimizing bias (cont.)

• matching (including fixed effects)
• attempts to get unbiased estimate of ?POC through
  matching
• but at the cost of making POC smaller relative to
  POS (and POI), because matching allows estimation
  of treatment effect only in region of common
  support
• but observational matching studies are easier to
  do with sample of the population of interest than
  are experiments, so bias 2 may be smaller in
  matching if region of common support (POC)
  approximates POI.
• fixed effects a form of matching (matching on
  invariant observed or unobserved factors)

6
How well do quasi-experimental methods do at
eliminating bias?

• Shadish Clark paper
• Lalonde (1986)-type studies estimate bias
  remaining after matching
• but generally cant disentangle residual bias in
  POC from bias 2
• Shadish Clark paper solves this problem
• Theoretically-informed matching can eliminate
  most/all of bias in POC
• What about bias 2?
• Extensions?
• can use this to assess average effects in
  population of those who would select the
  treatment if available

7
How well do quasi-experimental methods do at
eliminating bias?

• lessons of the Bloom paper
• consider ways of reducing variance of estimated
  treatment effect
• need to worry about functional form
• tie-breaking experiments enable us to evaluate
  bias in regression discontinuity (Black, Galdo,
  Smith 2005)
• RD estimates sensitive to functional form unless
  cases are near threshold
• treatment effect varies across thresholds

8
How well do quasi-experimental methods do at
eliminating bias?

• lessons of the Raudenbush paper
• adaptive centering provides same estimates as
  two-way fixed effects models
• better estimates of uncertainty, computationally
  easier, than fixed effects
• under what conditions do such designs reduce
  bias?
• fixed-effects (or centering) eliminates bias in
  POC under the assumption the within-cell
  assignment to treatment is ignorable (under what
  conditions is this reasonable?)
• fixed-effects may increase bias 2 by reducing
  region of common support (domain of POC)

9
remaining questions

• important to consider population of interest in
  research design, concerns about external validity
• how can we asses the extent to which we should
  worry about bias 2?
• meta-analysis of multiple studies with different
  populations of interest?
• multi-site randomized trials
• draw study samples from known population, assess
  participation selection