Research Design

1
Research Design

• Week 4
• Laura Christiansen

2
Introduction

• Structure of research
• Elements of design
• Group Assignments
• Measures/Observations
• Programs/Treatments

3
Design Notation

• Grouping Notations
• R random
• N non-equivalent
• C cutoff
• O observations
• X program/treatment
• Blank or /- notation

R O X O R O O

• N O1 X O2
• N O1 O2

• C O X O
• C O O

4
Research Design Types

• Experimental
• Random Assignment probabilistically equivalent
• Strongest internal validity
• Quasi-Experimental
• Multiple measures
• Multiple groups/control group

5
Experimental Designs

• Two-Group
• Signal Enhancers
• Factorial Designs
• Noise Reducers
• Randomized Block
• Covariance
• Hybrid Experimental
• Solomon Four-Group
• Switching Replications

6
Two-Group Experimental Designs
R X O R O

• No necessary pretests
• Probabilistically equivalent
• Strong internal validity against
• Single-group, (most) multiple-group
• Weak internal validity against
• Social interaction, selection mortality

7
Factorial Designs

• Factors Major independent variables
• Levels Subdivisions of factors
• Notation example 5x6
• of terms determines of factors 2
• of values determines levels one factor has 5,
  the other has 6
• Multiply to determine necessary groups 30

• R X11 O
• R X12 O
• R X21 O
• R X22 O

8
Factorial Designs Cont.

• Null Case
• No effect, no graphical slope
• Main Effects
• Consistent statistical difference between levels
• Lines still parallel
• Interaction Effects
• One factor depends on level of another
• Lines not parallel

9
Factorial Design Cont.

• Increasing number of factors
• Difficult to graph
• Increased number of groups
• Incomplete factorial designs
• Useful with control groups
• Leave out combinations

10
Randomized Block
R X O R O

• Separate into homogenous blocks
• Less variability/noise
• Pool estimates across blocks
• Data analysis strategy

R X O R O
R X O R O
11
Covariance
R O X O R O O

• Adjust post-test data for pre-test variability
• Covary post-test with pre-test
• Noise reduction
• Removes pre-post relationship

12
Solomon-Four Group

• Deals with testing threat
• Can view as 2x2 factorial
• Analyze difference between control and program
• Analyze difference between pre-test and
  non-pretest

R O X O R O O R X O R O
13
Switching Replications
R O X O O R O O X O

• 2 groups, 3 waves of measurement
• Helps mitigate social threats

14
Quasi-Experimental Designs

• Non-Equivalent Groups
• Regression Discontinuity
• Proxy Pretest
• Separate Pre-Post Samples
• Double Pretest
• Switching Replications

• Non-Equivalent Dependent Variables
• Pattern Matching
• Regression Point Displacement
• Interrupted Time Series

15
Non-Equivalent Groups

• N O X O
• N O O

• Groups are selected based on similarity
• Threat of selection
• Concerns regarding internal validity in analysis
• Popular in social research

16
Regression-Discontinuity

• C O X O
• C O O

• Assignment based on cutoff scores
• Targeted programs
• Pre-test/post-test measure equivalence not
  required
• Assumes no natural discontinuity at cutoff
• Less common in social research
• Comparable to randomized experimental IV for
  causal hypotheses
• Ethical benefits in some fields

17
Proxy-Pretest

• N O1 X O2
• N O1 O2

• Measure of O1 occurs after X
• Proxy variable estimate
• Two types
• Recollection ask for approximation
• Archived construct approximation
• Do not plan to use this

18
Separate Pre-Post Samples

• Pre-test and post-test data come from different
  groups
• Nonequivalence concerns
• Variants include random selection within
  non-equivalent groups

• N1 O
• N1 X O
• N2 O
• N2 O

19
Double Pretest

• N O O X O
• N O O O

• Stronger variant of Non-Equivalence Groups Design
• Detection of selection threats
• Detection of selection-maturation
• Also called dry run
• Simulates control group

20
Switching Replications

• N O X O O
• N O O X O

• 2 groups, 3 waves of measurement
• Equivalent to experimental variant
• Ethical benefits among quasi-experiments

21
Non-Equivalent Dependent Variables

• Targeted at specific outcome
• Single group divided in two
• Pre/post O2 tests act as control and gauge of
  maturity
• Minimize risk control being impacted by X
• Weak internal validity

22
Pattern Matching NEDV

• Variant of Non-Equivalent Dependent Variable
  design
• Incorporate multiple subgroups of single
  non-equivalence group
• Include theorization on affect of program
• Not concerned about lone impact as with NEDV
• Order of effects

23
Regression Point Displacement
N(n1) O X O N O O

• Compares test group to heterogeneous control
  groups
• Useful when dealing with multiple factors
• Judge post-test based on relation to regression
  line of control cases

24
Interrupted Time Series
N O O O O X O O O O

• Weak on internal validity
• Can be combined with other designs
• Ex Interrupted Time Series w/ Comparison Group
• Strengthens some benefits of Double Pretest