Locating Variance: Post-Hoc Tests

1
Locating Variance Post-Hoc Tests
Developing Study Skills and Research Methods
(HL20107)
Dr James Betts
2
Lecture Outline

• Influence of multiple comparisons on P
• Tukeys HSD test
• Bonferroni Corrections
• Ryan-Holm-Bonferroni Adjustments.

3
Placebo
Placebo
Lucozade
Lucozade
Gatorade
Gatorade
Powerade
Powerade
4
Placebo
Placebo
Lucozade
Lucozade
Gatorade
Gatorade
Powerade
Powerade
5
Why not multiple t-tests?

• i.e.
• Placebo vs Lucozade
• Placebo vs Gatorade
• Placebo vs Powerade
• Lucozade vs Gatorade
• Lucozade vs Powerade
• Gatorade vs Powerade
• We accept significance and reject the null
  hypothesis at P?0.05 (i.e. a 5 chance that we
  are wrong)
• Performing multiple tests therefore means that
  our overall chance of committing a type I error
  is gt5.

6
Post-hoc Tests

• A popular solution is the Tukey HSD
  (Honestly Significant Difference) test
• This uses the omnibus error term from the ANOVA
  to determine which means are significantly
  different
• T (q)

Error Variance
v
n
7
(No Transcript)
8
(No Transcript)
9
q table for Tukeys HSD
10
(No Transcript)
11
(No Transcript)
12
Tukey Test Critique

• As you learnt last week, the omnibus error term
  is not reflective of all contrasts if sphericity
  is violated

Placebo
Lucozade

• So Tukey tests commit many type I errors with
  even a slight degree of asphericity.

Gatorade
Powerade
13
Solution for Aspherical Data

• There are alternatives to the Tukey HSD test
  which use specific error terms for each contrast
• Fishers LSD (Least Significant Difference)
• Sidak
• Bonferroni
• Many others
• e.g. Newman-Kewls, Scheffe, Duncan, Dunnett,
  Gabriel, R-E-G-W, etc.

14
(No Transcript)
15
(No Transcript)
16
Trial 2
Trial 4
Trial 1
Fishers LSD
Bonferroni
Trial 3
17
Bonferroni Correction Critique

• Correction of LSD values successfully controls
  for type I errors following a 1-way ANOVA

• However, factorial designs often involve a larger
  number of contrasts, many of which may not be
  relevant.

Recovery Supp. 1
Recovery Supp. 2
18
Solution for Factorial Designs

• An adjustment to the standard Bonferroni
  correction can be applied for factorial designs
• This Ryan-Holm-Bonferroni or stepwise method
  involves returning to the P values of interest
  from our LSD test
• These P values are placed in numerical order and
  the most significant is Bonferroni corrected
  (i.e. P x m)
• However, all subsequent P values are multplied by
  m minus the number of contrasts already
  corrected.

19
Summary Post-Hoc Tests

• A Tukey test may be appropriate when sphericity
  can be assumed
• Multiple t-tests with a Bonferroni correction are
  more appropriate for aspherical data
• Stepwise correction of standard Bonferroni
  procedures maintain power with factorial designs
• Best option is to keep your study simple
• Pre-planned contrast at a specific time point
• Summary statistics (e.g. rate of change, area
  under curve)
• Just make an informed based on the data
  available.

20
Further reading from this lecture

• Atkinson, G. (2001) Analysis of repeated
  measurements in physical therapy research
  Physical Therapy in Sport 2 p. 194-208
• Atkinson, G. (2002) Analysis of repeated
  measurements in physical therapy research
  multiple comparisons amongst level means and
  multi-factorial designs Physical Therapy in Sport
  3 p. 191-203

21
Compulsory reading for next weeks lecture

• Batterham A. M. Atkinson, G. (2005) How Big
  Does My Sample Need to Be? A primer on the Murky
  World of Sample Size Estimation Physical Therapy
  in Sport 6 p. 153-163.