Experimental Designs Review Conclusion Validity

1
Experimental DesignsReviewConclusion Validity
2
Single IV Designs

• The basic two-group design

Independent Variable Customer Hearing
Experimental group
Control group
Customers were deaf
Customers were hearing
3
Single IV Designs

• The basic two-group design
• Compare the effects of two levels of the IV on
  the DV
• Analysis t-test directly compares group means

4
Single IV Designs

• The multiple-group design

Independent Variable Type of Noise
Experimental group 2
Experimental group 3
Experimental group 1
White noise
Music
No noise
5
Single IV Designs

• The multiple-group design
• Allows you to see the effects of multiple levels
  of the IV on the DV
• One-way ANOVA tells you if there is a difference
  among groups
• Post-hoc comparisons tells you where the
  difference is

6
Multiple IV Designs

• Factorialdesigns

7
Multiple IV Designs

• Factorial designs allow you to see
• The effects of IV1 on your DV (main effect of
  IV1)
• The effects of IV2 on your DV (main effect of
  IV2)(and so on, through however many IVs you
  have)
• The joint, simultaneous effect of all IVs on your
  DV (interaction effect)
• The effect of one IV depends on the specific
  level of the other IV
• Often easiest to see and interpret with a graph

8
Multiple IV Designs

• Analysis of factorial designs
• Factorial ANOVA
• For independent groups
• For correlated groups
• For mixed groups
• If main effects arent qualified by an
  interaction, post hoc tests determine where
  differences occurred

9
Validity

• Method-Related Concerns
• Internal Validity
• External Validity
• Measure-Related Concerns
• Construct Validity
• Reliability
• Conclusion Validity

10
Conclusion Validity

• Degree to which the conclusions you reach about
  the relationships in your data are reasonable

11
Conclusion Validity

• Significance Testing
• HO ?1 ?2
• HA ?1 ? ?2, ?1 lt ?2, or ?1 gt ?2
• Significance level p lt .05
• In 100 replications of the experiment, HA could
  occur 5 times by chance
• If we find HA, we want to be able to infer that
  HA is because of our IV

12
Conclusion Validity

• If our experiment is one of those 5 times in 100
  that the results occurred by chance
• Actuality HO is true
• Our Conclusion HA is true
• We make a Type I error (?)
• Accepting the experimental hypothesis when the
  null hypothesis is true
• We control the probability of ? by setting our
  significance level

13
Conclusion Validity

• More extreme significance level requires larger
  inferential test statistic to be significant
• Actuality HA is true
• Our Conclusion HO is true
• We make a Type II error (?)
• Accepting the null hypothesis when the
  experimental hypothesis is true

14
Conclusion Validity
True state of affairs
Experimental hypothesis is true Null hypothesis is true
Correct Decision Type I error (?)
Type II error (?) Correct Decision
Experimental hypothesis is true
Your decision
Null hypothesis is true
15
Conclusion Validity
True state of affairs
Experimental hypothesis is true Null hypothesis is true
Correct Decision Type I error (?)
Type II error (?) Correct Decision
Experimental hypothesis is true
Your decision
Null hypothesis is true
16
Effect Size

• Statistical measure that conveys the magnitude of
  the effect produced by the IV
• ? and ? both depend on sample size effect sizes
  dont
• Cohens d
• r 2
• Makes them ideal for decisions about the
  importance of an effect
• Statistically significant or practically
  significant?

17
Statistically significant?Practically
significant?

• Statistical significance
• Difference between groups is due to some
  systematic influence (we assume HA) and not
  chance
• Degree of risk you are willing to take that you
  will reject a null hypothesis when it is actually
  true

18
Statistically significant?Practically
significant?

• Practical significance (meaningfulness)
• Is the difference between groups meaningful in
  some way?
• Results must be interpreted in the context of the
  experiment
• Results can be statistically significant without
  being practically significant
• Results can be practically significant without
  reaching statistical significance

19
Statistically significant?Practically
significant?