QuasiExperimental Design

1
Quasi-Experimental Design
2

• The Big Questions . . .
• What two major problems challenge the ability to
  make causal inferences?
• What are the essential ingredients in an
  experiment for making causal inferences?
• Whats missing in a quasi-experiment?
• What four questions guide control of threats in
  quasi-experiments?

3
Once the problem of measurement is solved, the
more important quest for causality can begin.
4

• How Do We Identify A Cause?
• Cause precedes effect (direction)
• Cause covaries with effect (correlation)
• Other explanations are ruled less plausible
  (confounds)

5
Counterfactual Model Effect is what did happen
compared to what would have happened to the same
people had they not had the treatment at the same
time. We try to approximate this impossible
situation and research is convincing to the
extent that it approximates it closely.
6

• The failure of the counterfactual model usually
  creates one or both of two general threats to
  causal inference
• The directionality problem
• The third variable problem

7
The Directionality Problem Which variable is the
causal one? MaritalSatisfaction Depression
or
MaritalDepression
Satisfaction
8
The Third Variable Problem Maybe the relation is
not causal at all. Perhaps it is due to a third
variable. Example Height and weight are strongly
related. Does one cause the other?
9
The Third Variable Problem Height Nutrition
Weight
10
The Third Variable Problem Urban
Crowding ? Crime Rate
11
The Third Variable Problem Urban
Crowding Poverty Crime Rate
12
The Third Variable Problem Ice Cream Sales
? Crime Rate
13
The Third Variable Problem Ice Cream
Sales Temperature Crime Rate
14
Failure to consider third variables can produce
all manner of foolish inference and equally
foolish public policy . . .
What is the third variable here?
15
The counterfactual model is accomplished most
convincingly in an experiment and the experiment
serves as the gold standard against which we
determine the quality of other research designs.
16
Experiments are convincing because they clearly
solve the third variable problem and the
directionality problem. But, experiments are not
always possible, or fail in key ways, and
understanding the resulting threats to causal
inference is critical.
17
Beginning with quasi-experiments forces careful
consideration of threats to causal inference and
the ways that those threats might be reduced. The
tactics used in quasi-experiments can be used in
experiments as well.
18
What Is A Quasi-Experiment? Any design that is
missing one or more of the defining features of
an experiment. When this occurs, causal
inferences are not possible. Nonetheless, there
are degrees of threat to internal validity. A
good quasi-experiment can approach closely the
causal validity of an experiment.
19

• An experiment has three key ingredients
• Manipulation of the independent variable
• Random assignment
• Control of other third variables

20
By manipulating the independent variable, we can
know which variable is causally prior to the
other.
21
Random assignment insures that no pre-existing
differences (third variables) could provide an
alternative explanation for any differences that
we find.
22
Random assignment and random sampling are not the
same. Why is it also important to have random
sampling?
23
In 1936, the Literary Digest predicted that
Republican candidate Alfred Landon would defeat
Franklin Roosevelt by 14 percentage points. This
prediction was based on a nonrandom sample of
over 2 million Americans. The outcome of the
election? Landon lost by 24 percentage points.
Although a very large sample, it did not
represent the population of American voters.
24
Control over other third variables insures that
no differences other than the manipulation arise
during the experiment.
25

• The control of threats to causal inference is
  guided by four basic questions
• What threats are likely to hinder causal
  inference?
• How will they emerge?
• How can the design be modified to reduce the
  threats?
• Can the threats be estimated or removed
  statistically?

26

• General Strategies for Controlling Internal
  Validity
• Patterns of outcome
• Replication
• Better control over selection
• Better control over threats

27
Patterns of Outcome Threats to internal validity
can often be ruled less plausible through prior
knowledge of the pattern of outcome for the
treatment or for a proposed threat.
Treatment
Control
Outcome
Control
Pretest
Posttest
28
Replication Replication is one of the most
convincing ways to rule out threats to internal
validity. It can arise in several ways in
research design. Repeated observation
O O O O O O O O O X O O O O O O O O O
Why is this convincing in the absence of a
control group?
29
Replication Replication is one of the most
convincing ways to rule out threats to internal
validity. It can arise in several ways in
research design. Multiple dependent variables
Just a single treatment is present in this
design. Why do multiple dependent variables
provide a potentially stronger causal inference?
O X O O X O O X O O X O
30
Replication Replication is one of the most
convincing ways to rule out threats to internal
validity. It can arise in several ways in
research design. Multiple treatments or treatment
levels
Why does the inclusion of multiple treatment
levels make this a more convincing design?
O X1 O O X2 O O X3 O O X4 O
31
Replication Replication is one of the most
convincing ways to rule out threats to internal
validity. It can arise in several ways in
research design. Repeated treatments
O O O X O O O X O O O X O O O X O O O X O O O
Why is this convincing in the absence of a
control group?
32
O O O X O O O X O O O X O O O X O O O X O O O
33
Better Control Over Selection Cohort designs
When random assignment is not possible, there
might still be ways to reduce the threat of
pre-existing differences. In a cohort design, the
different groups follow each other through some
institution or system. Different classes of
graduate students would be a good example.
Because of the entry requirements, different
classes can be assumed to be similar on several
key variables.
34
Better Control Over Selection Regression-disconti
nuity designs
Criterion
Cut score
Selection Variable
35
Unadjusted difference
Adjusted difference
Outcome
Error is also reduced
Control
Treatment
Covariate
36
Better Control Over Threats Statistical Control
Treatment
Outcome
Grand Mean
Control
Third Variable
37
Unadjusted difference
Adjusted difference
Outcome
Error is also reduced
Control
Treatment
Covariate
38
Better Control Over Threats Internal analyses
Treatment
Outcome
Control
Pretest
Posttest
39
Internal Validity When any of the defining
features of an experiment are missing, the
inference that X and Y are causally related
cannot be made with confidence. This is a threat
to internal validity.
40

• The ability to recognize and solve threats to
  internal validity is critical to conducting good
  research, for everyone
• An intended experiment fails to accomplish one of
  the key defining features
• An experiment cannot be conducted for practical
  or ethical reasons
• A consumer of research needs to know what
  conclusions can be trusted

41
Two kinds of comparisons are used to make the
claim that a treatment is causally related to an
outcome Between-groups comparison X O O Withi
n-groups comparison O X O
Each of these comparisons is an imperfect
counterfactual. Why?
42

• Threats to Internal Validity
• Ambiguous temporal precedence
• Selection
• Attrition
• History
• Maturation
• Regression
• Testing
• Instrumentation

43
Ambiguous Temporal Precedence Temporal precedence
can be established in an experiment because
treatment precedes outcome. But, when treatment
is not possible, then logic and common sense can
sometimes dictate temporal precedence.
44
Selection Selection refers to any systematic
differences between groups that might account for
an observed effect X(S) O O The selection
variable is a confound. It can be effectively
eliminated by random assignment. R X O R O
45
Attrition Even if random assignment is used,
participants may drop out of the study, producing
unequal groups, a situation that has the same
inferential problems as selection R A
X(S) O R A O
How would this threat be ruled less plausible?
46
History History refers to any event that occurs
between the beginning of treatment and the
measurement of outcome that might have produced
the observed effect O X O H
How would this threat be ruled less plausible?
47
Maturation Maturation refers to changes in the
organism that occur regardless of treatment and
that may masquerade as a treatment
effect O X O M
How would this threat be ruled less plausible?
48
Maturation Maturation refers to changes in the
organism that occur regardless of treatment and
that may masquerade as a treatment effect O O O
O O O O O O X O O O O O O O O O M
49
O O O O O O O O O X O O O O O O O O O
50
Regression Regression (to the mean) occurs when
participants are selected because of their
extreme scores and those scores are unreliable.
The scores will regress toward the mean at the
second assessment O X O R
As the correlation between Time 1 and Time 2
decreases, the predicted score for Time 2
approaches the mean Z2 bZ1.
How would this threat be ruled less plausible?
51
Testing Testing refers to the possible change
that may occur just because participants have
been previously measured. These are often called
practice or fatigue effects. O X O T
How would this threat be ruled less plausible?
52
Instrumentation Change may occur because the
measurement changes over time, perhaps becoming
more or less reliable. O X O I
How would this threat be ruled less plausible?
53

• The Big Questions . . .
• What two major problems challenge the ability to
  make causal inferences?
• What are the essential ingredients in an
  experiment for making causal inferences?
• Whats missing in a quasi-experiment?
• What four questions guide control of threats in
  quasi-experiments?