Seth M' Noar, Ph'D'

1
Hypothesis Testing

• Seth M. Noar, Ph.D.
• Department of Communication
• University of Kentucky

2
Hypothesis Testing

• Widely used in the social sciences
• Statistical significance is one way to examine
  whether our findings are important statistically
• Numerous statistical tests allow us to test for
  statistical significance
• Logic for numerous statistical tests is the same,
  even though the tests are different

3
Hypotheses

• Research hypothesis (W M, p. 63)
• Null hypothesis (W M, p. 63)
• Two possibilities
• M1 equal to M2 (or different only to the extent
  to which we would expect based on sampling error)
• M1 unequal to M2
• We cannot directly test our research hypothesis.
  We test the null hypothesis.

4

• ARE THESE MEANS DIFFERENT???

5
Direction of tests

• We can also use directional or non-directional
  tests
• Two-tailed test Non-directional hypothesis
• One-tailed test directional hypothesis
• Note one-tailed test are more powerfully
  statistically.

6
Probability and Error

• We must choose a probability level (plt.05)
• The extent to which we risk error is directly
  related to the probability level
• Type 1 error rejecting null hypothesis when it
  was true
• However, we must balance this with statistical
  power probability of correctly rejecting null
  hypothesis

7

• Rejection regions for 2-tailed test, plt.05

8

• Rejection regions for 1-tailed test, plt.05

9

• Probability of type 1 error, plt.05

10
(No Transcript)
11

• Comparison of Type 1 2 errors, plt.05

12
t-test

• t-test is a significance test
• Focuses on mean differences
• When one would apply it
• 1 IV 2-level nominal (dichotomous)
• 1 DV continuous (must be interval)
• For example Do treatment and control
  participants differ at the end of a treatment
  program?

13
Logic of t-test

• M1 - M2
• SE diff
• SE diff standard error of the difference
  between means
• M1 M2 is straightforward.
• Larger the difference, larger the t value.
• SE diff is a bit more complex.

14
Standard Error of Difference

• Standard error of the difference between means
  takes into account
• Sampling error
• Does this by including sample size and
  variability in the formula.
• As t grows larger, associated values of
  probability grow smaller (higher chance of
  significance)
• Note Larger sample size often results in larger
  t values

15
Procedure

• Calculate t value
• Calculate degrees of freedom
• (n-1) (n-1) OR (n n 2)
• Look up critical value in Table
• Our calculated t-value must be greater than the
  value in the table to be statistically
  significant

16
(No Transcript)
17
Results

• If our t value is significant at the probability
  level we have chosen (plt.05), we reject the null
  hypothesis.
• This suggests that there are real differences
  between these means (e.g., they come from
  different populations).

18
Hypothesis Testing 2

• Seth M. Noar, Ph.D.
• Department of Communication
• University of Kentucky

19
ANOVA

• Single Factor ANOVA Analysis of Variance
• Focuses on mean differences
• When one would apply it
• 1 IV 2 or more nominal levels
• 1 DV continuous (must be interval)

20
ANOVA

• In many ways, similar to t test
• Major difference is that it can handle more than
  2 levels of IV
• For example Experiment with three groups
  Treatment, placebo, and no-treatment control.

21
Logic of ANOVA

• MS between (among)
• MS within
• Calculate between group variance
• Calculate within group variance
• Divide to get an F value
• We always expect within group variability
• However, we would only expect between group
  variability if the groups were different

22
Degrees of Freedom

• Degrees of Freedom
• Between (among) groups (numerator)
• Number of groups minus 1
• K-1 3 1 2
• Within groups (denominator)
• N-1 in each group, times number of groups
• N-1 5 1 4 x 3 12
• Similar to t look up critical value in table to
  examine significance
• In fact, t2 F (get same result whether applying
  t test or ANOVA)

23
(No Transcript)
24
Results

• If F value is significant at plt.05, we reject the
  null hypothesis.
• F (2, 12) 16.25, plt.05.
• However, if there are more than 2 levels of the
  IV, do we know where the differences lie?

25
Follow-up Tests

• Follow-up tests show us exactly where the
  differences lie.
• Tests include Tukey, Scheffe, Duncan, and Newman
  Keuls
• Some are more conservative, some more liberal
  (some control for type 1 error)
• Tukey HSD test is often used (controls for
  familywise error rate!)

26
Familywise error

• Individual significance tests may have an error
  rate of plt.05
• However, when you run multiple tests you get a
  greater error rate
• 5 x .05 .25
• This is referred to as familywise error rate
• Bonferroni procedure corrects familywise error
  rate back to original chosen rate
• .05 / 5 .01

27
Variations on ANOVA

• Repeated measures ANOVA
• Instead of multiple groups, we have multiple time
  points
• ANCOVA Analysis of covariance
• Same as ANOVA, except that we control for a
  variable called a covariate (e.g., stress, other
  meds)
• MANOVA Multivariate Analysis of Variance
• Allows for multiple DVs

28
Cautions related to significance

• Statistical significance is NOT clinical
  significance.
• Because two means are different statistically
  does NOT mean that the difference is meaningful
  or important.
• Statistical significance must be interpreted
  within the context of any given study.
• Measures of effect size are also very important
  here.

29
More cautions on significance

• Significant test controversy gets revisited every
  so often.
• Some have suggested abandoning traditional
  statistical tests (e.g., Jacob Cohen).
• Lines of argument include
• Over-reliance on dichotomous statistical tests to
  tell us whats important (p.06)
• Statistical tests are often misused
• Sensitivity to sample size
• Should rely more on effect sizes, confidence
  intervals, and meta-analysis

30
Effect Size (ES)

• Significance testing involves a dichotomous
  decision
• Either significant or not
• However, what about the magnitude of effect?
• Effect size magnitude of difference (or
  association) between variables
• Also called treatment magnitude

31
Effect Size (ES)

• Effect sizes range from -1 to 1
• Higher the value, stronger the effect size
  (similar to correlation)
• ES tends to be unaffected by sample size
• However, there are a variety of effect sizes that
  can be applied
• Each effect size is interpreted differently (and
  thus sometimes confusion results)

32
Effect Size

• Omega-squared Eta2 r d Size
• .01 .01 .10 .20 Small
• .06 .09 .30 .50 Medium
• .15 .25 .50 .80 Large
• (Cohen, 1988)

33
Effect Size

• Basic d effect size is this
• M1 - M2
• Pooled SD
• 4.5-3.5 .53
• 1.9

34
Other Effect Sizes

• r (correlation) and r2 (percent variance)
• Percent Those in the treatment group were 25
  more likely to improve as compared to those in
  the control group
• Odds ratios Those in the treatment group were
  twice as likely to improve as compared to those
  in the control group