Introduction to Research Design

1
Introduction to Research Design
2
What is Research Design?

• the structure of research

3
Elements of a Design

• Observations or Measures
• Treatments or Programs
• Groups
• Assignment to Group
• Time

4
Observations or Measures

• symbolized with an "O". Subscripts are used to
  distinguish different combinations of measures
  only if this is necessary

5
Treatments or Programs

• symbolized with an "X". Subscripts are used to
  indicate different programs or combinations of
  programs

6
Groups

• each group of interest on its own line

7
Assignment to Groups

• R random assignment
• N nonequivalent groups
• C assignment by cutoff

8
Time

• left - to - right movement denotes the passage of
  time

9
Design Notation Example
R O X O R O O
10
Design Notation Example
R O X O R O O
Time
11
Design Notation Example
R O X O R O O
Os indicate different waves of measurement
12
Design Notation Example
Vertical allignment of Os shows that pretest and
posttest are measured at same time
R O X O R O O
13
Design Notation Example
X is the treatment
R O X O R O O
14
Design Notation Example
There are two lines, one for each group
R O X O R O O
15
Design Notation Example
R O X O R O O
R indicates the groups are randomly assigned
16
Design Notation Example
R O1 X O1, 2 R O1 O1, 2
Subscripts indicate subsets of measures
17
Design Notation Example
R O X O R O O

• pretest-posttest (before-after)...
• ...treatment versus comparison group...
• ...randomized experimental design

18
Types of Designs
19
Types of Designs
random assignment?
20
Types of Designs
random assignment?
yes
21
Types of Designs
random assignment?
yes
randomized or true experiment
22
Types of Designs
random assignment?
yes
no
randomized or true experiment
23
Types of Designs
random assignment?
yes
no
control group or multiple measures?
randomized or true experiment
24
Types of Designs
random assignment?
yes
no
control group or multiple measures?
randomized or true experiment
yes
25
Types of Designs
random assignment?
yes
no
control group or multiple measures?
randomized or true experiment
yes
quasi-experiment
26
Types of Designs
random assignment?
yes
no
control group or multiple measures?
randomized or true experiment
yes
no
quasi-experiment
27
Types of Designs
random assignment?
yes
no
control group or multiple measures?
randomized or true experiment
yes
no
quasi-experiment
non-experiment
28
Design Example
Posttest Only Randomized Experiment
29
Design Example
Posttest Only Randomized Experiment
R X O R O
30
Design Example
Pretest-Posttest Nonequivalent Groups
Quasi-Experiment
31
Design Example
Pretest-Posttest Nonequivalent Groups
Quasi-Experiment
N O X O N O O
32
Design Example
Posttest Only Non-Experiment
33
Design Example
Posttest Only Non-Experiment
X O
34
The Two-Group Randomized Experiment
35
The Basic Design

• R X O
• R O

• note that a pretest is not necessary in this
  design -- Why?
• because random assignment assures that we have
  probabilistic equivalence between groups

36
The Basic Design

• R X O
• R O

37
The Basic Design

• R X O
• R O

• differences between groups on posttest indicate a
  treatment effect
• usually test this with a t-test or one-way ANOVA

38
Internal Validity

• R X O
• R O

history maturation testing instrumentation mortali
ty regression to the mean selection selection-hist
ory selection- maturation selection-
testing selection- instrumentation selection-
mortality selection- regression diffusion or
imitation compensatory equalization compensatory
rivalry resentful demoralization
39
Experimental Design Variations

• the posttest-only two group design is the
  simplest -- there are many variations
• to better understand what the variations try to
  achieve we can use the signal-to-noise metaphor

40
Signal to Noise
What we observe can be divided into
41
Signal to Noise
What we observe can be divided into
signal
42
Signal to Noise
What we observe can be divided into
signal
noise
43
Signal to Noise
Experimental designs can take two approaches
44
Signal to Noise
Experimental designs can take two approaches
signal
focus on (enhance) the signal
45
Signal to Noise
Experimental designs can take two approaches
signal
focus on (enhance) the signal
46
Signal to Noise
Experimental designs can take two approaches
47
Signal to Noise
Experimental designs can take two approaches
noise
or reduce the noise
48
Signal to Noise
Experimental designs can take two approaches
noise
or reduce the noise
49
Signal to Noise
signal enhancers

• factorial designs
• covariance designs
• blocking designs

noise reducers
50
The Nonequivalent Groups Design
51
The Basic Design
N O X O N O O

• Key Feature
• nonequivalent assignment to groups

52
What does nonequivalent mean?

• assignment is nonrandom
• researcher didnt control assignment
• groups may be different
• group differences may affect outcomes

53
Internal Validity

• N O X O
• N O O

• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?
• ?

history maturation testing instrumentation regress
ion to the mean selection mortality diffusion or
imitation compensatory equalization compensatory
rivalry resentful demoralization
54
Internal Validity

• N O X O
• N O O

• ?
• ?
• ?
• ?
• ?
• ?

selection-history selection-maturation selection-t
esting selection-instrumentation selection-regress
ion selection-mortality
55
The Bivariate Distribution
56
The Bivariate Distribution
Program Group has a 5-point pretest advantage
57
The Bivariate Distribution
Program Group scores 15-points higher on posttest
Program Group has a 5-point pretest advantage
58
Graph of Means
59
Possible Outcome 1
Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality

• ?
• ? (CG not growing)
• ?
• ?
• ?(PG moving away, CG level)
• ?more low-score PG dropouts

60
Possible Outcome 2

• ?
• ?? (both growing)
• ?
• ?
• ?(wrong direction)
• ?more low-score dropouts

Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality
61
Possible Outcome 3
Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality

• ?
• ? (in PG only)
• ?
• ?
• ??(in PG)
• ?more high-score PG dropouts (not as likely)

62
Possible Outcome 4
Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality

• ?
• ? (in PG only)
• ?
• ?
• ??(in PG)
• ??more low-score PG dropouts

63
Possible Outcome 5
Selection-History Selection-Maturation Selection-T
esting Selection-Instrumentation Selection-Regress
ion Selection-Mortality

• ?
• ?
• ?
• ?
• ?
• ?

64
Internal Validity
Is the relationship causal between...

• what you did and what you saw?
• your program and your observations?

alternative cause
alternative cause
Program
Observations
program-outcome relationship
What you do
What you see
alternative cause
alternative cause
Observation
In this study