Statistical Power And Sample Size Calculations

1
Statistical Power And Sample Size Calculations
Minitab calculations
Manual calculations
2
When Do You Need Statistical Power Calculations,
And Why?

• A prospective power analysis is used before
  collecting data, to consider design sensitivity .

3
When Do You Need Statistical Power Calculations,
And Why?

• A retrospective power analysis is used in order
  to know whether the studies you are interpreting
  were well enough designed.

4
What Is Statistical Power?Essential concepts

• the null hypothesis Ho
• significance level, a
• Type I error
• Type II error

5
Statistical Hypothesis Testing

• When you perform a statistical hypothesis test,
  there are four possible outcomes

6
Statistical Hypothesis Testing

• whether the null hypothesis (Ho) is true or false
  
• whether you decide either to reject, or else to
  retain, provisional belief in Ho

7
Statistical Hypothesis Testing
8
When Ho Is True And You Reject It, You Make A
Type I Error

• When there really is no effect, but the
  statistical test comes out significant by chance,
  you make a Type I error.
• When Ho is true, the probability of making a Type
  I error is called alpha (a). This probability is
  the significance level associated with your
  statistical test.

9
When Ho is False And You Fail To Reject It, You
Make A Type II Error

• When, in the population, there really is an
  effect, but your statistical test comes out
  non-significant, due to inadequate power and/or
  bad luck with sampling error, you make a Type II
  error.
• When Ho is false, (so that there really is an
  effect there waiting to be found) the probability
  of making a Type II error is called beta (ß).

10
The Definition Of Statistical Power

• Statistical power is the probability of not
  missing an effect, due to sampling error, when
  there really is an effect there to be found.
• Power is the probability (prob  1 - ß) of
  correctly rejecting Ho when it really is false.

11
Calculating Statistical PowerDepends On

• the sample size
• the level of statistical significance required
• the minimum size of effect that it is reasonable
  to expect.

12
How Do We Measure Effect Size?

• Cohen's d
• Defined as the difference between the means for
  the two groups, divided by an estimate of the
  standard deviation in the population.
• Often we use the average of the standard
  deviations of the samples as a rough guide for
  the latter.

13
Cohen's Rules Of Thumb For Effect Size
14
Calculating Cohens d
14
15
Calculating Cohens d
15
16
Calculating Cohens d from a t test
16
17
Conventions And Decisions About Statistical Power

• Acceptable risk of a Type II error is often set
  at 1 in 5, i.e., a probability of 0.2.
• The conventionally uncontroversial value for
  adequate statistical power is therefore set at
  1 - 0.2  0.8.
• People often regard the minimum acceptable
  statistical power for a proposed study as being
  an 80 chance of an effect that really exists
  showing up as a significant finding.

18
Next Week

• Statistical Power Analysis In Minitab

19
Statistical Power Analysis In Minitab

• Stat gt Power and Sample Size gt

20
Statistical Power Analysis In Minitab
Note that you might find web tools for other
models. The alternative normally involves
solving some very complex equations.

• 1-Sample Z
• 1-Sample t
• 2-Sample t
• 1 Proportion
• 2 Proportions
• One-Way ANOVA
• 2-Level Factorial Design
• Plackett-Burman Design
• Recall that a comparison of proportions equates
  to analysing a 22 contingency table.

21
Statistical Power Analysis In Minitab
Note that you might find web tools for other
models. The alternative normally involves
solving some very complex equations.
Simple statistical correlation analysis online
See Test 28 in the Handbook of Parametric and
Nonparametric Statistical Procedures, Third
Edition by David J Sheskin
22
Factors That Influence Power

• Sample Size
• alpha  
• the standard deviation

23
Using Minitab To Calculate Power And Minimum
Sample Size

• Suppose we have two samples, each with n  13,
  and we propose to use the 0.05 significance level
• Difference between means is 0.8 standard
  deviations (i.e., Cohen's d  0.8)
• All key strokes in printed notes

24
Using Minitab To Calculate Power And Minimum
Sample Size
Note that all parameters, bar one are
required. Leave one field blank. This will be
estimated.
25
Using Minitab To Calculate Power And Minimum
Sample Size

• Power and Sample Size
• 2-Sample t Test
• Testing mean 1  mean 2 (versus not )
• Calculating power for mean 1  mean 2
  difference
• Alpha  0.05 Assumed standard deviation  1
• Sample
• Difference Size Power
• 0.8 13 0.499157
• The sample size is for each group.

Power will be 0.4992
26
Using Minitab To Calculate Power And Minimum
Sample Size

• If, in the population, there really is a
  difference of 0.8 between the members of the two
  categories that would be sampled in the two
  groups, then using sample sizes of 13 each will
  have a 49.92 chance of getting a result that
  will be significant at the 0.05 level.

27
Using Minitab To Calculate Power And Minimum
Sample Size

• Suppose the difference between the means is 0.8
  standard deviations (i.e., Cohen's d  0.8)
• Suppose that we require a power of 0.8 (the
  conventional value)
• Suppose we intend doing a one-tailed test, with
  significance level 0.05.
• All key strokes in printed notes

28
Using Minitab To Calculate Power And Minimum
Sample Size
Select Options to set a one-tailed test
29
Using Minitab To Calculate Power And Minimum
Sample Size
30
Using Minitab To Calculate Power And Minimum
Sample Size

• Power and Sample Size
• 2-Sample t Test
• Testing mean 1  mean 2 (versus gt)
• Calculating power for mean 1  mean 2
  difference
• Alpha  0.05 Assumed standard deviation  1
• Sample Target
• Difference Size Power Actual Power
• 0.8 21 0.8 0.816788
• The sample size is for each group.

Target power of at least 0.8
31
Using Minitab To Calculate Power And Minimum
Sample Size

• Power and Sample Size
• 2-Sample t Test
• Testing mean 1  mean 2 (versus gt)
• Calculating power for mean 1  mean 2
  difference
• Alpha  0.05 Assumed standard deviation  1
• Sample Target
• Difference Size Power Actual Power
• 0.8 21 0.8 0.816788
• The sample size is for each group.

At least 21 cases in each group
32
Using Minitab To Calculate Power And Minimum
Sample Size

• Power and Sample Size
• 2-Sample t Test
• Testing mean 1  mean 2 (versus gt)
• Calculating power for mean 1  mean 2
  difference
• Alpha  0.05 Assumed standard deviation  1
• Sample Target
• Difference Size Power Actual Power
• 0.8 21 0.8 0.816788
• The sample size is for each group.

Actual power 0.8168
33
Using Minitab To Calculate Power And Minimum
Sample Size
Suppose you are about to undertake an
investigation to determine whether or not 4
treatments affect the yield of a product using 5
observations per treatment. You know that the
mean of the control group should be around 8, and
you would like to find significant differences of
4. Thus, the maximum difference you are
considering is 4 units. Previous research
suggests the population s is 1.64.
34
Using Minitab To Calculate Power And Minimum
Sample Size
35
Using Minitab To Calculate Power And Minimum
Sample Size
Power 0.83
Power and Sample Size One-way ANOVA Alpha 0.05
Assumed standard deviation 1.64 Number of
Levels 4 SS Sample
Maximum Means Size Power Difference 8
5 0.826860 4 The sample size is
for each level.
36
Using Minitab To Calculate Power And Minimum
Sample Size
To interpret the results, if you assign five
observations to each treatment level, you have a
power of 0.83 to detect a difference of 4 units
or more between the treatment means. Minitab
can also display the power curve of all possible
combinations of maximum difference in mean
detected and the power values for one-way ANOVA
with the 5 samples per treatment.
37
Next Week
Manual Calculations of Power
38
Sample Size Equations

• Five different sample size equations are
  presented in the printed notes.
• For obvious reasons, only one is explored in
  detail here.

39
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.

• Calculate an initial sample size using the
  following equation

recall
40
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.

• Calculate an initial sample size using the
  following equation

41
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.
42
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.
To obtain the adjusted sample size estimate,
consult the correction table in the printed
notes. n is the uncorrected sample size value
from the sample size equation. n is the
corrected sample size value. See the example
below.
43
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.
Additional correction for sampling finite
populations. The above formula assumes that the
population is very large compared to the
proportion of the population that is sampled. If
you are sampling more than 5 of the whole
population then you should apply a correction to
the sample size estimate that incorporates the
finite population correction factor (FPC). This
will reduce the sample size.
44
Determining The Necessary Sample Size For
Estimating A Single Population Mean Or A Single
Population Total With A Specified Level Of
Precision.
45
Example

• Objective Restore the population of species Y in
  population Z to a density of at least 30
• Sampling objective Obtain estimates of the mean
  density and population size of 95 confidence
  intervals within 20 () of the estimated true
  value.
• Results of pilot sampling 
• Mean ( )   25
• Standard deviation (s)  7

46
Example

• Given The desired confidence level is 95 so
  the appropriate Za from the table above is 1.96.
  The desired confidence interval width is 20
  (0.20) of the estimated true value. Since the
  estimated true value is 25, the desired
  confidence interval (B) is 25 x 0.20  5.

47
Example

• Calculate an unadjusted estimate of the sample
  size needed by using the sample size formula

Round 7.53 up to 8 for the unadjusted sample size.
48
Example

• To adjust this preliminary estimate, go to the
  sample size correction table and find n  8 and
  the corresponding n value in the 95 confidence
  level portion of the table. For n  8, the
  corresponding value is n  15.

49
Example
50
Example

• The corrected estimated sample size needed to be
  95 confident that the estimate of the population
  mean is within 20 (5) of the true mean is 15.

51
Example

• Additional correction for sampling finite
  populations.
• If the pilot data described above was gathered
  using a 1m x 10m (10 m2) quadrat and the total
  population being sampled was located within a 20m
  x 50m macroplot (1000 m2) then N  1000m2/10m2  1
  00.

52
Example

• The corrected sample size would then be

The new, FPC-corrected, estimated sample size to
be 95 confident that the estimate of the
population mean is within 20 (5) of the true
mean is 13.
53
Text

• Sample size calculations in clinical research
• edited by Shein-Chung Chow, Jun Shao, Hansheng
  Wang
• New York Marcel Dekker, 2003
• Long loan Robinson Books Level 4
• 610.72 SAM
• see also the conventional notes

54
Bibliography 1 of 2

• Also see Tutorial in Quantitative Methods for
  Psychology Volume 3, no 2 (2007) Special issue
  on statistical power.
• Editors note The Uncorrupted Statistical Power
• Statistical Power An Historical Introduction
• Understanding Power and Rules of Thumb for
• Determining Sample Sizes
• A Short Tutorial of GPower

55
Bibliography 2 of 2

• Computing the Power of a t Test
• Also see
• Non-central t Distribution and the Power of the t
  Test A Rejoinder
• Understanding Statistical Power Using Non-central
  Probability Distributions Chi-squared,
  G-squared, and ANOVA
• Power Estimation in Multivariate Analysis of
  Variance
• A Power Primer

56
Caveat

• It is well known that statistical power
  calculations can be valuable in planning an
  experiment. There is also a large literature
  advocating that power calculations be made
  whenever one performs a statistical test of a
  hypothesis and one obtains a statistically
  non-significant result.

57
Caveat

• Advocates of such post-experiment power
  calculations claim the calculations should be
  used to aid in the interpretation of the
  experimental results. This approach, which
  appears in various forms, is fundamentally
  flawed.

58
Caveat

• The paper documents that the problem is
  extensive and presents arguments to demonstrate
  the flaw in the logic.
• The abuse of power The pervasive fallacy of
  power calculations for data analysis, Hoenig JM,
  Heisey DM American Statistician, 551, 19-24,
  2001