Validity and Reliability

1
Validity and Reliability
2
Generate Hypotheses

• Theory Alcohol impairs driving ability
• Step one generate testable hypotheses
• Hypothesis 1 Heavier drinkers take more risks
  while driving than lighter drinkers
• Hypothesis 2 Alcohol slows down reaction time
  which causes more accidents
• Hypothesis 3 Alcohol increases risk-taking while
  driving which causes more accidents

3
Testing Hypotheses

• Step 2 We conduct a controlled test
• We select a research design
• We identify the population and select a sample
• We define the variables of interest

4
Research Design

• Non-experimental vs Experimental designs
• Non-experimental observe a single group of
  subjects at one point in time
• Observational studies
• Survey Studies
• Experimental involves multiple groups or
  multiple observations across time
• Control standard against which the effects of
  the experimental condition is compared

5
Validity

• Scientific Validity refers to the concept of how
  correct is my statement or do I trust my
  results.

6
Types of Validity

• External Validity
• Can we generalize our findings from the
  experimental context to other people, in other
  places, at other times, in other situations?

7
External Validity

• Random Sampling
• Identify Population
• All people
• All adults
• All adults who drink alcohol
• All adults who drive after drinking alcohol
• Select Sample
• Select sample from population of interest
• Random
• Stratified representative of the population on
  key characteristics

8
External Validity
9
Types of Validity

• Internal Validity
• Can we be confident that the observed outcomes
  are due to (caused by) our experimental treatment
  and NOT to some other cause?
• How sure are we that manipulation of the
  independent variable caused the differences
  observed in the dependent variable

10
Internal Validity

• If some other factor could have caused the
  results (BAC just accidentally covaries with risk
  taking) this is known as a confound
• Confound uncontrolled and/or unmeasured
  characteristic(s) that accounts for the observed
  findings

11
Research Design

• Random Assignment
• Place subjects at random into different control
  and experimental conditions
• Goal is to ensure that potential confounds are
  equally represented in both groups
• Not always possible, so how do we ensure internal
  validity?

12
Internal Validity

• In our study, we put all of the heavy drinkers in
  the group that received alcohol and all of our
  light or non-drinkers in the group that received
  water
• To help subjects get in the mood, the bartender
  played loud and upbeat music during the drink
  alcohol condition. No music was played during
  the drink water condition.

13
Testing Hypotheses

• Observation a man leaves a bar and gets into a
  car accident
• Theory Alcohol impairs driving ability
• Hypothesis 1 Heavier drinkers take more risks
  while driving than lighter drinkers
• Hypothesis 2 Individuals who are intoxicated
  take more risks while driving than individuals
  who are sober

14
Testing Hypotheses

• Independent variable
• Predictor Variable
• Variable that is manipulated
• Dependent Variable
• Predicted variable
• Variable that depends on or is affected by the
  independent variable
• Operational definition
• Concrete description of how your variables will
  be measured

15
Testing Hypotheses

• Hypothesis 1
• IV level of drinking is defined using the
  quantity frequency index
• DV Risk-taking is defined as
• frequency of speeding (i.e., number of days per
  week that the individual drives 10 miles over
  the speed limit)
• Frequency of tailgating

16
We analyze the data and reject or accept our
hypothesis
17
Statistical Analysis
18
Statistical Analysis
19
Statistical Analysis
20
Statistical Analysis
21
Statistical Analysis
Variance Explained
22
Pirates and Global Warming
23
Testing Hypotheses

• Hypothesis 2
• IV Intoxication is defined as a BAC of .08 mg
  or greater
• DV Risk-taking is defined through use of a
  simulated driving task as
• amount of time spent speeding and
• number of times passing cars on a double yellow
  line

24
Statistical Analysis
25
Statistical Analysis
26
Construct Validity

• Related to external validity
• How well do my variables reflect what I want to
  measure
• How good are your operational definitions

27
Reliability

• Good science means hypotheses are not only
  testable but REPEATABLE

28
How to ensure reliability

• Test retest reliability
• One person can make the same observation several
  times
• Do they exhibit temporal consistency?
• Inter-rater reliability (inter-observer
  agreement)
• Do several judges or raters come to similar
  conclusions

29
Internal Consistency

• How well do items that should be related score
  similarly
• Items that are similar on a scale should covary.

30
Reliability

• More is better
• 10 ratings is more reliable than 5
• 20 ratings is more reliable than 10
• 5 Raters (judges) are better than 1
• 10 Raters (judges) are better than 5

31
Measurement

• In general the more reliable the measure the more
  valid the results
• Type of measurement scale can influence the
  reliability of your results

32
Measurement Scales

• Nominal Scale or Categorical Scale
• You score one or the other
• Examples
• Urine test positive or negative
• Sex male or female
• Pregnant yes or no
• Shoe type loafer, tennis shoe, hiking boot

33
Measurement Scales

• Ordinal scales
• Ordinal scales rank order things but tell you
  nothing about the distance between them
• Examples
• Foot race 1st, 2nd, 3rd, etc
• Olympic Medals Gold, Silver, Bronze

34
Measurement Scales

• Interval Scales
• Interval scales have numerical ordering
• Many Psychological tests use interval scales
• Examples
• IQ
• SAT
• More than one person can have the same score
• Distance between intervals not necessarily the
  same
• SAT is the difference between 560 and 600 the
  same as the distance between 760 and 800

35
Measurement Scales

• Ratio Scales
• All intervals have the same amount between them
• Examples
• Difference between 100 kg and 110 kg is the same
  as the difference between 40 and 50 kg
• Can express as ratios
• Example
• 40kg is half as heavy as 80 kg
• 20 kg is half as much as 40 kg

36
Measurement Scales

• General rules of scales
• The better the scale the easier it is to get
  reliability
• You can convert ration scales to interval to
  ordinal to categorical but you cannot go in the
  reverse
• Make your scales as reliable as possible

37
Measurement Scales

• Example of general rules
• You versus Tiger Woods in golf
• Measurement Scale categorical
• number of holes in one (yes hole in one, no,
  missed the cup)
• May take thousands of shots to determine who is
  the better golfer (who gets more holes in one on
  the long run)

38
Measurement Scales

• Example of general rules
• You versus Tiger Woods in golf
• Measurement Scale Ordinal
• Rank order shots closest to the hole
• You Tiger
• 25ft 27ft
• 100 ft 23ft
• 30 ft 32 ft
• 125 ft 27 ft
• 65 ft 29 ft
• Better but still harder to represent truth

39
Measurement Scales

• Example of general rules
• You versus Tiger Woods in golf
• Measurement Scale Interval or Ratio
• Rank measure the distance of the shots and get
  the total feet from the cup
• You Tiger
• 25ft 27ft
• 100 ft 23ft
• 30 ft 32 ft
• 125 ft 27 ft
• 65 ft 29 ft

40
Measurement Scales

• Test Measurement scales can affect reliability
• True false less reliable than ratings on a 5
  point likert scales
• 9 point likert scale more reliable than a 5 point
  scale
• Up to a point - adding more doesnt always add
  more, but it doesnt hurt (13 not necessarily
  better than 9, but never worse)

41
Measurement Scales

• Test Measurements can affect reliability
• Example 6 point scale of TV watching
• 1 6
• up to 0.5 hrs more than 2.5 hrs
• 1 6
• Up to 2.5 hrsmore than 4.5 hrs