Quasi-Experiments

1
Quasi-Experiments
2
The Basic Nonequivalent Groups Design (NEGD)
N O X O N O O

• Key Feature Nonequivalent assignment

3
What Does Nonequivalent Mean?

• Assignment is nonrandom.
• Researcher didnt control assignment.
• Groups may be different.
• Group differences may affect outcomes.

4
Equivalence

• Equivalent groups are not necessarily identical
  on any pre-test measure.
• Merely implies that if the random assignment
  procedure was repeated, the groups would tend
  toward equivalence.

5
Non-Equivalence

• Non-equivalent groups do not necessarily differ
  on any pre-test measure.
• Merely implies that If the same non-random
  assignment procedure was repeated, the groups
  would tend to toward non-equivalence.
• If assignment to groups was based partly on
  income, then groups would tend to have different
  expected mean levels of income but any two
  groups you picked might well be similar in income
  levels.

6
The Point

• Equivalence or non-equivalence is defined by the
  selection procedure.
• Even if the difference in pre-test means across
  groups is small, this does not imply that the
  groups are equivalent.
• Small differences can introduce big threats.

7
Quasi- vs. Natural vs. Experiment

• In a true experiment, the researcher performs the
  random assignment
• Can be in a lab or the field
• In a natural experiment, someone else assigns
  through a random process.
• In a quasi-experiment, assignment is not random,
  introducing selection threats.
• Much stronger if the selection is not done by the
  cases themselves (exogenous sorting).

8
What is a Natural Experiment

• Strict Definition
• Some truly natural process, such as rainfall or
  weather patterns, assigns IV.
• Definition we all use in our own work
• Some exogenous process, rather than our cases,
  ourselves, or a causal process relevant to our
  theory, assigns IV.

9
Genres of Natural Experiments

• The natural border or natural disaster

• The Rule Change

• House seniority system (Crooks and Hibbing)
• GAVEL amendment in Colorado
• Connecticut speeding law
• New Zealand electoral reform
• Propositions
• Relatively easy to spot, hard to defend

• Jared Diamonds islands
• Dan Posners rivers
• Caroline Hoxbys streams
• Settler mortality (Acemoglu, Johnson, and
  Robinson)
• Hurricane Katrina
• Strength is that nature doesnt care about your
  cases or IV

10
Genres of Natural Experiments

• The Court Decision

• The Lottery

• Roe V. Wade for Levitt and Donohue
• Iowa item veto decision
• Strength is that court is not a blatant political
  actor responding to societal shifts or societal
  pressures

• James Fowlers use of Canadian bill introduction
  privilege
• US House Clerk conducts a randomization of the
  order in which members choose office
• Strength is true randomness in first step, but
  human action in 2nd

11
Genres of Natural Experiments

• Staged Implementation

• The Threshold

• Mail ballot assignment in precincts with lt250
  voters
• Need to make the threshold unrelated to DV, or
  else use Trochim-style regression discontinuity

• Two-step reapportionment revolution in the United
  States
• Lots of program evaluations in development
• Helps to rule out history and maturation threats

12
What Makes a Convincing Natural Experiment?

• You can show that the process of selection was
  not related to characteristics of the cases that
  are relevant to your DV
• In a cross-sectional experiment, demonstrate that
  the two groups are quite similar
• In a time-series experiment, demonstrate that
  little else changed when the treatment took
  place.
• In a word, show equivalence

13
Any purported causal test of needs to take into
consideration all of the two-group threats to
validity.
R X O R O
Can be a valid causal test.
N X O N O
Fully exposed to threats.
14
NEGD Design has Multiple Groups AND Multiple
Measures
N O X O N O O
This helps rule out (or at least recognize)
threats.
15
Pre-Tests v. Covariates
N O X O N O O
Pre- Post-Test Design Observations are tests you
administer.
N O1 X O2 N O1 O2
Proxy Pre-Test Design First observations are
covariates on which you collect data.
16
Problems of Internal Validity in NEGDs
17
Internal Validity

• N O X O
• N O O

All designs suffer from threats to validity. In
addition to all the single group
threats, quasi-experiments are particularly
likely to suffer from multi-group threats.

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
18
The Bivariate Distribution
19
The Bivariate Distribution
Program Group has a 5-point pretest advantage.
20
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest advantage,
21
Graph of Means
22
Possible Outcome 1

• Possible local event
• Possible PG initially higher
• Unlikely no change in CG
• Possible scale effects
• Unlikely expect change in CG
• Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
23
Possible Outcome 2

• Likely PG initially higher
• Likely PG initially higher
• Possible
• Possible
• Unlikely expect change in CG
• Possible both lose low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
24
Possible Outcome 3
Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality

• Possible local event
• Unlikely no change in CG
• Unlikely no change in CG
• Possible scale effects
• Likely
• Possible PG loses high scorers

25
Possible Outcome 4

• Possible local event
• Unlikely no change in CG
• Unlikely no change in CG
• Possible scale effects
• Very Likely
• Possible PG loses low scorers

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
26
Possible Outcome 5

• And you should be so lucky

Selection-history Selection-maturation Selection-t
esting Selection-instrumentation Selection-regress
ion Selection-mortality
27
Analysis Requirements
N O X O N O O

• Pre-post (or covariates)
• Two-group
• Treatment-control (dummy 0, 1)

28
Analysis of Covariance (ANCOVA)
yi ?0 ?1Xi ?2Zi ei
where

• yi outcome score for the ith unit
• ?0 coefficient for the intercept
• ?1 pretest coefficient
• ?2 mean difference for treatment
• Xi covariate
• Zi dummy variable for treatment(0 control, 1
  treatment)
• ei residual for the ith unit

29
The Bivariate Distribution
Program group scores 15-points higher on Posttest.
Program group has a 5-point pretest Advantage.
30
The Bivariate Distribution
Slope is B1
Vertical Distance is Mean Treatment Effect, or B2
31
Why Add Covariates to Analysis?

• ANCOVA can include more than one pretest or
  control variable.
• Additional pretests further adjust for initial
  group differences.
• Ideally, in the absence of any treatment effect,
  the covariates would perfectly predict the
  posttest.
• Additional covariates will often improve the
  accuracy of the estimate of the treatment effect.

32
Irrelevant Covariates

• Adding pretests that are completely unrelated to
  the posttest, however, actually decreases
  precision.
• Irrelevant covariates contribute nothing to the
  analysis, but subtract a degree of freedom from
  the error term.
• This reduces the efficiency of the estimate.

33
Omitted Covariates

• Covariates that are related to the posttest but
  not to the treatment can be ignored without
  biasing the estimate of the treatment effect.
• Covariates that are related to the posttest and
  the treatment but that are omitted will bias the
  estimate of the treatment effect.
• We can safely omit control variables even if they
  are highly correlated with the posttest as long
  as they do not correlate with the treatment.

34
Omitted Variables Bias

• Omitted (relevant) covariates that are positively
  correlated with the treatment will lead us to
  overestimate the treatment effect.
• Omitted (relevant) covariates that are negatively
  correlated with the treatment will lead us to
  underestimate the treatment effect.

35
Bottom Line

• We should always try to include omitted relevant
  covariates, except
• When the omitted covariate is itself a
  consequence of the treatment.
• If cannot include a relevant covariate, we can at
  least predict the direction if not magnitude of
  the likely bias.

36
ButWhat about measurement error?

• With multiple covariates, measurement error does
  not always lead to a pseudo-effect.
• As measurement error in any single variable
  increases, it becomes as if the variable is not
  included in the ANCOVA.
• This then mimics an omitted variables problem,
  and the direction of bias depends upon the
  relationship between the noisy covariate and
  the treatment.

37
Other Quasi-Experimental Designs
38
Separate Pre-Post Samples
N1 O N1 X O N2 O N2 O

• Groups with the same subscript come from the same
  context.
• Here, N1 might be people who were in the program
  at Agency 1 last year, with those in N2 at Agency
  2 last year.
• This is like having a proxy pretest on a
  different group.

39
Separate Pre-Post Samples
R1 O R1 X O R2 O R2 O
N
N

• Take random samples at two times of people at two
  nonequivalent agencies.
• Useful when you routinely measure with surveys.
• You can assume that the pre and post samples are
  equivalent, but the two agencies may not be.

40
Double-Pretest Design
N O O X O N O O O

• Strong in internal validity
• Helps address selection-maturation

41
Switching Replications
N O X O O N O O X O

• Strong design for both internal and external
  validity
• Strong against social threats to internal
  validity
• Strong ethically

42
Nonequivalent Dependent Variables Design (NEDV)
N O1 X O1 O2 O2

• The variables have to be similar enough that they
  are affected the same way by all threats.
• The program has to target one variable and not
  the other.
• In simple form, weak internal validity.

43
NEDV Example

• Only works if we can assume that geometry scores
  show what would have happened to algebra if
  untreated.
• The variable is the control.
• Note that there is no control group here.

44
NEDV Pattern Matching

• Have many outcome variables.
• Have theory that tells how affected (from most to
  least) each variable will be by the program.
• Match observed gains with predicted ones.
• With pattern, NEDV can be extremely powerful.

45
NEDV Pattern Matching

• A ladder graph.

r .997
46
NEDV Lake and OMahony 2006
Hypothesis As territory declines in value in
20th century (measured by average state size),
wars fought over territory should decline
in frequency. There should be no pattern in
other Issues.