Quasi Experimental Approaches

1
Quasi Experimental Approaches

• COE/EDP 501
• Week 4

2
What makes it Quasi?

• The principle of random assignment is violated
  because
• It is not possible
• It is not practical
• It is not ethical
• Groups cannot be said to be equivalent
• Differences in outcome may be due to these
  confounding or nuisance variables

3
Impossible

• Categorical variables
• Classification by existing standards

4
Impractical

• Environmental Factors
• Available personnel, skill sets
• Existing beliefs, perspectives

5
Unethical

• Interfering with live-saving treatment
• Physically or psychologically traumatic

6
What do you do?

• Minimize for confounding
• Can the differences be compensated for?
• Can the odds be stacked against the study?
• Estimate effects of confounding
• How do treatment control differ?
• How might these affect outcome?
• Can they be factored out as covariates?

7
Regression to the Mean

• Identify differences
• Pre-testing
• Demographics
• Tracking events during the study
• Explore possibility of regression to the mean
• Use covariates to rule out other causal factors
  in outcomes

8
Regression to the Mean
9
Regression to the Mean
10
Regression to the Mean
11
Example 1 Career Counseling

• Kept groups as similar as possible by offering
  intervention later
• Similar groups on pre-test
• Cross-over interaction

12
Example 2 Share the Parenting

• Groups assigned to bias against finding a result
• Groups similar on pre-screening
• Adding covariates, the interaction didnt
  disappear

13
One-group Designs

• When only one sample is available
• Group must act as own control
• Extremely difficult to interpret
• Best approach Time-Series Design

14
One-group, Pre Post
15
One-group Time Series
16
One-group Time Series
17
One-group Time Series
18
One-Group Designs

• Very unsatisfactory
• May be the only option if
• Sample is very small
• Individual reaction is highly variable

19
Design Experiments

• Dissatisfaction w/ traditional studies of
  classroom intervention

20
Characteristics of Design Experiments

• Similar to one-group time-series
• Different in number, complexity of factors
• Intervention design
• Delivery method, style, interpersonal factors
• Physical resources, implementation decisions
• Different in number, complexity of outcomes
• Performance multiple open-ended tasks
• Social dynamics, shift of power
• Logistics, resource needs, costs
• Often reported in a narrative style, theory
  justification woven into methods data

21
Cycle of Discovery, Innovation Application
1. Design, implement, and document interventions
5. Design and develop tools, materials and methods
2. Synthesize and interpret results and identify
new insights and questions
3. Research on problems of learning, teaching,
implementation, and policy
4. Develop and test theory and knowledge about
teaching and learning

22
Advantages of Design Experiments

• Authenticity, or ecological validity
• Allows focus on complexity, processes, dynamics
• Flexibility natural day-to-day variations can be
  accommodated
• Immediate intervention, immediate response to
  problems opportunities
• Tends to be more client centered

23
Challenges Engineering vs. Research

• Initial design often based in intuition, not well
  or strongly motivated
• Whats working and how much work is it doing?
• Often atheoretical, limiting explanatory power,
  ability to inform other projects
• As design changes, factors and measures change
  narrative becomes disjointed

24
Design Experiments Arent Always Good Engineering

• Design underdetermined by data
• A problem may not suggest its solution
• Transfer from one group to another limited
• Often high overhead may be wasted on aspects
  that dont work