Establishing a Cause-Effect Relationship

1
Establishing a Cause-Effect Relationship
2
Internal Validity
Is the relationship causal between...

• what you did and what you saw?
• your program and your observations?

alternative cause
alternative cause
Program
Observations
program-outcome relationship
What you do
What you see
alternative cause
alternative cause
Observation
In this study
3
Establishing Cause Effect
Temporal Precedence
4
Establishing Cause Effect
Temporal Precedence
Cause
Effect
then
time
5
Establishing Cause Effect
Temporal Precedence
Cause
Effect
then
time
Why is this important?
6
Establishing Cause Effect
Temporal Precedence
Cause
Effect
then
time
Why is this important?
Cyclical Functions
inflation
unemployment
7
Establishing Cause Effect
Temporal Precedence
Covariation of Cause and Effect
8
Establishing Cause Effect
Temporal Precedence
Covariation of Cause and Effect
if X, then Y if not X, then not Y
9
Establishing Cause Effect
Temporal Precedence
Covariation of Cause and Effect
if X, then Y if not X, then not Y
if program given, then outcome observed if
program not given, then outcome not observed
10
Establishing Cause Effect
Temporal Precedence
Covariation of Cause and Effect
if X, then Y if not X, then not Y
if program given, then outcome observed if
program not given, then outcome not observed
if more of program, then more of outcome
observed if less of program given, then less of
outcome observed
11
Establishing Cause Effect
Temporal Precedence
if X, then Y if not X, then not Y
Covariation of Cause and Effect
No Alternative Explanations
12
Establishing Cause Effect
Temporal Precedence
if X, then Y if not X, then not Y
Covariation of Cause and Effect
No Alternative Explanations
Program
Outcome
causes?
13
Establishing Cause Effect
Temporal Precedence
if X, then Y if not X, then not Y
Covariation of Cause and Effect
alternative cause
No Alternative Explanations
alternative cause
Program
Outcome
causes?
alternative cause
alternative cause
14
In typical outcome evaluation...

• is taken care of because you intervene before you
  measure outcome
• is taken care of because you control the
  intervention
• is the central issue of internal validity --
  usually taken care of through your design

Temporal Precedence
Covariation of Cause and Effect
No Alternative Explanations
15
Introduction to Validity
16
What is Validity?

• the best available approximation to the truth or
  falsity of a given inference, proposition,
  conclusion
• a set of standards by which research can be judged

17
The Causal Context
Theory
Observation
18
The Causal Context
Theory
Cause Construct
Observation
19
The Causal Context
Theory
Cause Construct
Observation
20
The Causal Context
Theory
Cause Construct
Effect Construct
cause-effect construct
Observation
21
The Causal Context
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
Observation
22
The Causal Context
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
operationalize
Program
Observation
In this study
23
The Causal Context
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
operationalize
operationalize
Program
Observations
Observation
In this study
24
The Causal Context
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
operationalize
operationalize
Program
Observations
What you do
What you see
Observation
In this study
25
The Causal Context
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
operationalize
operationalize
Program
Observations
program-outcome relationship
What you do
What you see
What you test
Observation
In this study
26
Conclusion Validity
Is there a relationship between...

• what you did and what you saw?
• your program and your observations?

Program
Observations
program-outcome relationship
What you do
What you see
Observation
In this study
27
Internal Validity
Is the relationship causal between...

• what you did and what you saw?
• your program and your observations?

alternative cause
alternative cause
Program
Observations
program-outcome relationship
What you do
What you see
alternative cause
alternative cause
Observation
In this study
28
Construct Validity
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct

• Can we generalize to the constructs?

Program
Observations
program-outcome relationship
What you do
What you see
Observation
29
External Validity
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct

• Can we generalize to other persons, places, times?

30
The Validity Questions are cumulative...
31
The Validity Questions are cumulative...
Is there a relationship between the cause and
effect?
In this study
32
The Validity Questions are cumulative...
In this study
Is the relationship causal?
Is there a relationship between the cause and
effect?
Conclusion
33
The Validity Questions are cumulative...
Can we generalize to the constructs?
In theory
Internal
Is the relationship causal?
Is there a relationship between the cause and
effect?
Conclusion
34
The Validity Questions are cumulative...
Can we generalize to other persons, places, times?
In theory
Can we generalize to the constructs?
Construct
Internal
Is the relationship causal?
Is there a relationship between the cause and
effect?
Conclusion
35
The Validity Questions are cumulative...
Validity
Can we generalize to other persons, places, times?
External
Can we generalize to the constructs?
Construct
Internal
Is the relationship causal?
Is there a relationship between the cause and
effect?
Conclusion
36
Threats to Validity
You want to make an inference...

• There is a relationship between the cause and
  effect
• The relationship is causal
• We can generalize to the constructs
• We can generalize to other persons, places, times

37
Threats to Validity
How could you be wrong in the inference?
Conclusion Validity

• there is a relationship but you dont see it
• there is no relationship but you do see one

38
Threats to Validity
How could you be wrong in the inference?
Internal Validity

• there is a causal relationship but you dont see
  it
• there is no causal relationship but you do see one

39
Threats to Validity
How could you be wrong in the inference?
Construct Validity

• you can generalize to constructs, but you
  conclude you cant
• you cant generalize to constructs but you
  conclude you can

40
Threats to Validity
How could you be wrong in the inference?
External Validity

• you can generalize to other contexts, but you
  conclude you cant
• you cant generalize to contexts but you conclude
  you can

41
Threats to Validity

• how you can be wrong in making your inference
• specific factors that can bias or distort your
  conclusions
• a list of common threats to the quality of your
  study
• a checklist you can use in planning your study

42
External Validity
43
The External Validity Question
Theory
What you think
Cause Construct
Effect Construct
cause-effect construct
44
The External Validity Question
Theory
Cause Construct
Effect Construct
cause-effect construct

• Can we generalize to other persons, places, times?

45
How Do We Generalize?Model I Sampling
specified persons, places , times
Population
46
How Do We Generalize?Model I Sampling
Population
draw sample
Sample
draw sample
47
How Do We Generalize?Model I Sampling
generalize back
generalize back
Population
Sample
48
How Do We Generalize?Model II Proximal
Similarity
Our Study
49
How Do We Generalize?Model II Proximal
Similarity
settings
Our Study
times
people
places
50
How Do We Generalize?Model II Proximal
Similarity
less similar
settings
Our Study
less similar
less similar
times
people
places
less similar
51
How Do We Generalize?Model II Proximal
Similarity
less similar
settings
Our Study
less similar
less similar
times
people
places
Gradients of Similarity
less similar
52
Threats to External Validity

• maybe it is just these people
• maybe it is just these places
• maybe it is just these times

Interaction of Selection and Treatment
Interaction of Setting and Treatment
Interaction of History and Treatment
53
How Can We Improve External Validity?
54
How Can We Improve External Validity?
random sampling
55
How Can We Improve External Validity?
random sampling
replicate, replicate, replicate
56
How Can We Improve External Validity?
random sampling
replicate, replicate, replicate
use theory
57
Single-Group Threats to Internal Validity
58
The Single Group Case
Two Designs
59
The Single Group Case
Two Designs
60
The Single Group Case
Two Designs
Measure Baseline
O
61
The Single Group Case
alternative explanations
Two Designs
alternative explanations
Measure Baseline
O
alternative explanations
62
Example

• Compensatory Education in Math for 1st Graders
• Pre-Post Single Group Design
• Measures (O) are standardized achievement tests
  (at start of grade 1 and start of grade 2 Forms
  A B)

63
History Threat

• any other event that occurs between pretest and
  posttest
• for example, kids pick up math concepts watching
  Sesame Street

64
Maturation Threat

• normal growth between pretest and posttest
• they would have learned these concepts anyway,
  even without program

65
Testing Threat

• the effect on the posttest of taking the pretest
• may have primed the kids or they may have
  learned from the test, not the program

66
Instrumentation Threat

• any change in the test from pretest and posttest
• for example, change due to different forms of
  test, not to program

67
Mortality Threat

• non-random dropout between pretest and posttest
• for example, kids challenged out of program by
  parents or teachers

68
Regression Threat

• group is a nonrandom subgroup of population
• for example, mostly low-math kids will appear to
  improve because of regression to the mean

69
Multiple-Group Threats to Internal Validity
70
The Central Issue

• when you move from single to multiple group
  research the big concern is whether ther groups
  are comparable
• usually this has to do with how you assign units
  (e.g., persons) to the groups (or select them
  into groups)
• we call this issue selection or selection bias

71
The Multiple Group Case
alternative explanations
Administer Program
Measure Outcomes
Measure Baseline
Do not Administer Program
Measure Outcomes
Measure Baseline
alternative explanations
72
Example

• Compensatory Education in Math for 1st Graders
• Pre-Post Program-Comparison Group Design
• Measures (O) are standardized achievement tests
  (at start of grade 1 and start of grade 2 Forms
  A B)

73
Selection Threats

• any factor other than the program that leads to
  posttest differences between groups
• for example, because of group differences, kids
  in one group watch Sesame Street more frequently
  and pick up more math concepts

74
Selection-History Threat

• any other event that occurs between pretest and
  posttest that the groups experience differently
• for example, kids in one group pick up more math
  concepts because they watch more Sesame Street

75
Selection-Maturation Threat

• differential rates of normal growth between
  pretest and posttest for the groups
• they are learning at different rates, even
  without program

76
Selection-Testing Threat

• differential effect on the posttest of taking the
  pretest
• the test may have primed the kids differently
  in each group or they may have learned
  differentially from the test, not the program

77
Selection-Instrumentation Threat

• any differential change in the test used for each
  group from pretest and posttest
• for example, change due to different forms of
  test being given differentially to each group,
  not due to program

78
Selection-Mortality Threat

• differential non-random dropout between pretest
  and posttest
• for example, kids drop out of the study at
  different rates for each group

79
Selection-Regression Threat

• different rates of regression to the mean because
  groups differ in extremity
• for example, program kids are disproportionately
  lower math scorers and consequently have greater
  regression to the mean

80
Social Interaction Threats to Internal Validity
81
What are Social Threats

• all are related to social pressures in the
  research context which can lead to posttest
  differences which are not directly caused by the
  treatment itself
• most of these can be minimized by isolating the
  two groups from each other, but this leads to
  other problems (e.g., hard to randomly assign and
  then isolate, may reduce generalizability)

82
Diffusion or Imitation of Treatment

• controls might learn about the treatment from
  treated people (e.g., kids in the school
  cafeteria)

83
Compensatory Equalization of Treatment
84
Compensatory Equalization of Treatment

• administrators give a compensating treatment to
  controls

85
Compensatory Rivalry

• controls compete to keep up with treatment group

86
Resentful Demoralization

• controls "give up" or get discouraged
• the screw you effect

87
The Two-Group Randomized Experiment
88
The Basic Design

• R X O
• R O

• note that a pretest is not necessary in this
  design -- Why?
• because random assignment assures that we have
  probabilistic equivalence between groups

89
The Basic Design

• R X O
• R O

• differences between groups on posttest indicate a
  treatment effect
• usually test this with a t-test or one-way ANOVA

90
Internal Validity

• R X O
• R O

history maturation testing instrumentation mortali
ty regression to the mean selection selection-hist
ory selection- maturation selection-
testing selection- instrumentation selection-
mortality selection- regression diffusion or
imitation compensatory equalization compensatory
rivalry resentful demoralization
91
Experimental Design Variations

• the posttest-only two group design is the
  simplest -- there are many variations
• to better understand what the variations try to
  achieve we can use the signal-to-noise metaphor

92
The Nonequivalent Groups Design
93
The Basic Design
N O X O N O O

• Key Feature
• nonequivalent assignment

94
What does nonequivalent mean?

• assignment is nonrandom
• researcher didnt control assignment
• groups may be different
• group differences may affect outcomes

95
Internal Validity

• N O X O
• N O O

• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?

history maturation testing instrumentation regress
ion to the mean selection mortality diffusion or
imitation compensatory equalization compensatory
rivalry resentful demoralization
96
Internal Validity

• N O X O
• N O O

• ?
• ?
• ?
• ?
• ?
• ?

selection-history selection-maturation selection-t
esting selection-instrumentation selection-regress
ion selection-mortality
97
The Bivariate Distribution
98
The Bivariate Distribution
Program Group has a 5-point pretest advantage
99
The Bivariate Distribution
Program Group scores 15-points higher on posttest
Program Group has a 5-point pretest advantage
100
Graph of Means
101
Possible Outcome 1

• ?
• ? (CG not growing)
• ?
• ?
• ?(PG moving away, CG level)
• ?more low-score PG dropouts

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality
102
Possible Outcome 2

• ?
• ?? (both growing)
• ?
• ?
• ?(wrong direction)
• ?more low-score dropouts

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality
103
Possible Outcome 3

• ?
• ? (in PG only)
• ?
• ?
• ??(in PG)
• ?more high-score PG dropouts not as likely

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality
104
Possible Outcome 4

• ?
• ? (in PG only)
• ?
• ?
• ??(in PG)
• ??more low-score PG dropouts

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality
105
Possible Outcome 5

• ?
• ?
• ?
• ?
• ?
• ?

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality