Reliability

1
Reliability
2
Todays Objectives

• Define measurement error and describe how it
  interferes with psychological studies
• Know that reliability is the ratio of true
  variability to observed variability
• Name major sources of error variance
• Describe methods for assessing reliability

3
Reliability

• In psychometrics, reliability refers to
  consistency in measurement

4
Error Variance

• Score X is a reflection of the true score on the
  psychological construct being measured plus
  error.
• X T E

5
Error Variance

• If we wanted to measure spelling ability, the
  best technique would be to systematically go
  through the dictionary and have a person spell
  every word, which would be the TRUE score for
  spelling ability.
• But, that would be a terrible way to spend the
  summer.
• Better method would be to take a random sample of
  words from the dictionary.
• Reliability analysis is estimating how much error
  we would make by using the score from the shorter
  test as an estimate of spelling ability.

6
Error Variance

• The greater the proportion of variance attributed
  to the s E , the less reliable the test.

• s 2 s T 2 s E 2 s S 2

7
Error Variance

• A measurement is reliable if it reflects mostly
  the variability of the true score, relative to
  the error
• Reliability
• s T 2 /( s T 2 s E 2)
• Put differently, its the ratio of true
  variability to observed variability

8
Error Variance

• This item is probably an unreliable measure of a
  persons feelings toward iguanas.
• Why? Ample individual differences abound
  regarding feelings toward mating.

• Mating iguanas are particularly ugly.
• 012345
• Disagree completely Agree
  Completely

Reliability s T 2 /( s T 2 s E 2)
9
Measuring Reliability

• Suppose you measured the height of every member
  of your family with a crude stick.
• The goal is to reliably distinguish between
  individuals.
• What would be more reliable Measuring everyone
  once, or measuring each person 100 times and then
  taking the average across the 100 measurements as
  the final measure of each persons height?

10
Measuring Reliability

• Principle Numero Uno
• The more items on a test, the better the
  reliability.
• This is because errors will be randomly
  distributed across items, and sum to zero.
• For example, when given Flying iguanas are ugly
  as a test item, people who are for/against mating
  will be randomly for/against flying.

11
Measuring Reliability
12
Measuring Reliability
13
Sources of Error Variance

• Test construction
• Test administration
• Test scoring and interpretation

14
Sources of Error Variance

• Misinterpretation
• Social Desirability
• Misunderstanding the Instructions
• Attention Seeking
• Forgetting

15
Reliability Estimates

• Test-Retest Method
• The test is administered twice at two different
  points in time.
• Assumes that there will be no change in the
  quality or construct being measured.
• Test-retest reliability is best used for things
  that are stable over time, such as intelligence.

16
Reliability Estimates

• Parallel-Forms/Alternate Forms Method
• Parallel tests have identical true scores and
  identical error variances.
• Alternate forms are two forms of a tests whose
  items are intended to measure the same thing and
  do not differ from each other in any systematic
  way.
• E.g., Watson-Glaser Critical thinking test items
  
• Test A Terry don't worry about it. You'll
  graduate someday. You're a college student.
  Right? And all college students graduate sooner
  or later.
• Test B Charlie, don't worry about it. You'll get
  a promotion someday. You're working for a good
  company. Right?. And everyone who works for a
  good company gets a promotion sooner or later.

17
Reliability Estimates

• Split Half
• Correlate scores from equivalent halves of a
  single test administered once.
• Goal is usually to create parallel forms.
• Divide the items on the test into equivalent
  halves
• Correlate the scores between the two halves
• Adjust the result to take into account that if
  you were to reduce the items, reliability would
  decrease (Spearman-Brown formula)

18
Reliability Estimates

• Interrater Reliability Method
• Evaluate the consistency among judges who are
  rating the same behavior

19
Summary