Statistical Analysis of the Two Group Post-Only Randomized Experiment

1
Statistical Analysis of the Two Group Post-Only
Randomized Experiment
2
Analysis Requirements
R X O R O

• Two groups
• A post-only measure
• Two distributions, each with an average and
  variation
• Want to assess treatment effect
• Treatment effect statistical (i.e., nonchance)
  difference between the groups

3
Statistical Analysis
4
Statistical Analysis
Control group mean
5
Statistical Analysis
Control group mean
Treatment group mean
6
Statistical Analysis
Control group mean
Treatment group mean
Is there a difference?
7
What Does Difference Mean?
8
What Does Difference Mean?
Medium variability
9
What Does Difference Mean?
Medium variability
High variability
10
What Does Difference Mean?
Medium variability
High variability
Low variability
11
What Does Difference Mean?
The mean difference is the same for all three
cases.
Medium variability
High variability
Low variability
12
What Does Difference Mean?
Medium variability
High variability
Which one shows the greatest difference?
Low variability
13
What Does Difference Mean?

• A statistical difference is a function of the
  difference between means relative to the
  variability.
• A small difference between means with large
  variability could be due to chance.
• Like a signal-to-noise ratio.

Which one shows the greatest difference?
Low variability
14
What Do We Estimate?
Low variability
15
What Do We Estimate?
Signal
Noise
Low variability
16
What Do We Estimate?
Signal
Difference between group means

Noise
Low variability
17
What Do We Estimate?
Signal
Difference between group means

Noise
Variability of groups
Low variability
18
What Do We Estimate?
Signal
Difference between group means

Noise
Variability of groups
_
_
XT - XC
_
_

SE(XT - XC)
Low variability
19
What Do We Estimate?
Signal
Difference between group means

Noise
Variability of groups
_
_
XT - XC
_
_

SE(XT - XC)

t-value
Low variability
20
What Do We Estimate?

• The t-test, one-way analysis of variance (ANOVA)
  and a form of regression all test the same thing
  and can be considered equivalent alternative
  analyses.
• The regression model is emphasized here because
  it is the most general.

Low variability
21
Regression Model for t-Test or One-Way ANOVA
yi ?0 ?1Zi ei
22
Regression Model for t-Test or One-Way ANOVA
yi ?0 ?1Zi ei
where

• yi outcome score for the ith unit
• ?0 coefficient for the intercept
• ?1 coefficient for the slope
• Zi 1 if ith unit is in the treatment group0 if
  ith unit is in the control group
• ei residual for the ith unit

23
In Graph Form...
24
In Graph Form...
Zi
0 (Control)
1 (Treatment)
25
In Graph Form...
Yi
Zi
0 (Control)
1 (Treatment)
26
In Graph Form...
Yi
Zi
0 (Control)
1 (Treatment)
27
In Graph Form...
Yi
?0 is the intercept y-value when z0.
Zi
0 (Control)
1 (Treatment)
28
In Graph Form...
Yi
?1 is the slope.
?0 is the intercept y-value when z0.
Zi
0 (Control)
1 (Treatment)
29
Why Is ?1 the Mean Difference?
Yi
?1 is the slope.
?0 is the intercept y-value when z0.
Zi
0 (Control)
1 (Treatment)
30
Why Is ?1 the Mean Difference?
Yi
Intuitive Explanation Because slope is the
change in y for a 1-unit change in x.
Change in y
Unit change in x (i.e., z)
Zi
0 (Control)
1 (Treatment)
31
Why Is ?1 the Mean Difference?
Yi
Change in y
Since the 1-unit change in x is the
treatment- control difference, the slope is the
difference between the posttest means of the two
groups.
Zi
0 (Control)
1 (Treatment)
32
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
33
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
34
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
35
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
36
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
37
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
yC ?0
38
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
yC ?0
For treatment group (Zi 1)
39
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
yC ?0
For treatment group (Zi 1)
yT ?0 ?1(1) 0
40
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
yC ?0
For treatment group (Zi 1)
yT ?0 ?1(1) 0
41
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
First, determine effect for each group
For control group (Zi 0)
yC ?0 ?1(0) 0
ei averages to 0 across the group.
yC ?0
For treatment group (Zi 1)
yT ?0 ?1(1) 0
yT ?0 ?1
42
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
43
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
44
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
treatment
yT ?0 ?1
yT
45
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC
46
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC (?0 ?1)
47
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC (?0 ?1) - ?0
48
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC (?0 ?1) - ?0
yT - yC ?0 ?1 - ?0
49
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC (?0 ?1) - ?0
?
?
yT - yC ?0 ?1 - ?0
50
Why ?1 Is the Mean Difference in
yi ?0 ?1Zi ei
Then, find the difference between the two groups
control
treatment
yC ?0
yT ?0 ?1
yT - yC (?0 ?1) - ?0
?
?
yT - yC ?0 ?1 - ?0
yT - yC ?1
51
Conclusions

• t-test, one-way ANOVA and regression analysis all
  yield same results in this case.
• The regression analysis method utilizes a dummy
  variable for treatment.
• Regression analysis is the most general model of
  the three.