Causal Inference in Instructional Research

1
Causal Inference in Instructional Research
2
Discussion Focus

• Research goal of understanding (not forecasting)
• Quantitative inquiry (not qualitative)
• A common sense view of causation (no review of
  different definitions)
• Available ER courses and resources (not other
  departments or colleges)

3
Causation One Definition

• A causal relationship exists if the manipulation
  of a cause will result in the manipulation of an
  effect (Cook and Campbell, 1979, p. 36).
• See Cook and Campbell for a comparison of
  different philosophical positions on causation.

4
Logic of Causal Inference

• To infer that the correlation between two
  variables (or the difference in mean outcomes for
  two groups) is in part due to a causal effect,
  one must
• rule out alternative explanations (or rival
  hypotheses), aka
• eliminate all threats to internal validity,
  aka
• isolate the effect of one variable on the
  other, controlling for all other possible causes.

5
Role of Instrument Reliability and Validity in
Causal Inference

• Construct validity required to ensure that an
  empirical relationship reflects the putative
  cause and effect
• Reasonable instrument reliability required to
  minimize the threat to statistical validity
• ER courses e.g., EDF 5432

6
Overview of Study Designs for Causal Inference

• Experimental studies Researcher controls the
  treatment intervention and the random assignment
  of subjects to treatments.
• Quasi-experimental studies Researcher controls
  the treatment intervention, but cant assign
  subjects randomly. Statistical control may be
  used.
• Non-experimental (aka correlational) studies
  Researcher controls only the selection of
  variables to be used for statistical control.

7
Experimental Studies (I)

• Payoff of random assignment Control the threat
  of non-equivalent groups.
• Remaining threats to internal validity due
  primarily to focused inequities and to
  differential mortality.
• Reasons why an experiment may not be possible or
  desirable include, e.g., impossibility of
  manipulation (e.g., gender), ethical concerns
  (e.g., exposure to disease), and need for quick
  policy decisions.

8
Experimental Studies (II) Related ER Courses

• Research Design (EDF 5481)
• Statistical Analyses w/ ANOVA and ANCOVA
• EDF 5402 (classical)
• EDF 5401 (regression approach)
• EDF 5406 (multiple outcomes with MANOVA)
• Alternative analyses (e.g., nonparametric with
  EDF 5410)
• Note validity of causal inference not determined
  by analysis method

9
Quasi-Experimental Studies (I)

• Introduces a new threat compared to experiments
  nonequivalent groups
• A range of possible nonequivalent (ne) group
  designs from
• Posttest-only with ne groups
• One group pretest-posttest design
• Control group design with pretest and posttest
• Interrupted time series design

10
Quasi-Experimental Studies (II) Related ER
Courses

• Research design (EDF 5481)
• Statistical analysis
• ANOVA and ANCOVA designs with increased attention
  to statistical control for nonequivalent groups
  (EDF 5401 and 5402)
• Time series analysis (Stat department)

11
Non-experimental Studies (I)

• Bivariate correlation wo/ control
• Correlation does not imply causation (spurious
  correlation)
• Necessary but not sufficient condition for
  causation
• ER courses EDF 5400, 5410
• Bivariate correlation w/ control (e.g., partial
  correlation)

12
Non-experimental Studies (II) Statistical
Control with Multiple Regression

• Results in valid causal inference when correctly
  specified
• Threat to validity of causal inference is
  incorrect model specification (e.g., leaving out
  important variables)
• There is no empirical proof of correctness of a
  model
• Must rely on theoretical justification of model

13
Non-experimental Studies (III) Statistical
Control with Multiple Regression (Continued)

• Some limitations
• Only estimates the direct causal effects (i.e.,
  no mediation)
• No test of model consistency with data (i.e.,
  cant falsify a model)
• ER courses EDF 5401 (linear models)

14
Non-experimental Studies (IV) Structural
Equation Modeling (SEM)

• Hypothesize a causal model (see next slide). The
  hypothesized direct effects imply indirect
  effects.
• Test the fit of the model to data.
• If fit not acceptable (i.e., if the model is
  falsified), consider possible model revisions.
• Once acceptable fit found, describe direct,
  indirect, and total causal effects.

15
Path Diagram for Hypothesized Model
z
1
g
11
Aptitude
Motivation
b
31
( y
)
( x
)
1
1
g
31
Achievement
z
3
f
( y
)
b
12
3
21
g
b
22
32
SES
Study Habits
( x
)
2
( y
)
2
z
2
16
Non-experimental Studies (V) Structural Equation
Modeling (Cont.)

• Primary threat to validity of causal inference
  incorrect model specification
• No empirical test for correctness of model
• When the model fits the data, it can be concluded
  that the model is consistent with the data, not
  that the model has been proven true
• ER course EDF 5401 (EDF 5409 changing to 100
  HLM)

17
A COE Resource on Methods of Inquiry

• Go to College of Education web site and click on
  Inquiry Skills
• Help needed on revision and maintenance of site

18
Some Concluding Comments

• Strength of causal inference is a continuum, not
  a dichotomy
• Causal inference is possible with experimental,
  quasi-experimental, and non-experimental
  (correlational) studies
• The strength of causal inference
• Depends on study and analysis logic allowing for
  the elimination of alternative explanations,
• Does not depend on the analysis method