PS28C Psychometrics

1
PS28CPsychometrics

• Lecture 3
• True Score Theory and Reliability

2
Overview of Lecture

• Will cover two related areas
• Classical True-Score Theory
• Reliability
• Will provide brief overview of true-score theory
• Describe the concept of reliability
• Describe types of reliability

3
True-Score Theory

• Clash between two approaches to viewing world
• World we see is real world versus world as ideal
  forms
• Classical true-score theory belongs to second
  view
• Classical true-score theory underlies reliability
  and validity

4
True-Score Theory

• All scales and measures do not perfectly capture
  events
• Observed scores are a combination of True Score
  and error
• Impossible to create pure measure as all measures
  are inherently flawed

5
True Score Theory

• Summarizes relationship between observed scores
  and true scores on tests
• Where X is the observed score
• T is the true score, and
• e is random error of measurement

6
Relation of Observed and True Scores
Error
True Score
7
Reliability

• Reliability is based on the idea of stability
• Stability can occur in several ways
• Across time
• Across different versions of test
• Across different people
• Within the same test

8
Reliability

• Each form of stability is linked to different
  type of reliability
• Stability mathematically assessed using
  correlation coefficients

9
Test-Retest Reliability

• Test-retest reliability is stability across time
• Administer same test to same people twice
• Correlate scores across sessions
• Problems of length of time between tests and
  carry-over effects

10
Parallel Form Reliability

• Stability across different versions of the same
  tests
• Two similar versions of the same test are
  administered to the same people twice
• Correlate scores from the two versions of the
  tests
• Problems of length of time between tests and
  carry-over effects

11
Inter-Rater Reliability

• Stability assessed across two or more raters
  rating the same person
• Both raters are asked to observe the behaviour or
  a characteristic of a specific person
• Pairs of ratings of same person are correlated

12
Internal Consistency Reliability

• Assesses stability within the same testing period
• Patterns of response should be consistent within
  a testing session
• Two approaches
• Divide items into equal halves
• Create all possible combinations of halves
• Correlate scores from the first and second halves
  of the test

13
Internal Consistency Reliability

• Creates the highest estimate of a tests
  reliability
• Can not be used for tests which are timed or the
  items become progressively more difficult

14
Mathematics of Reliability

• Reliability coefficient is another version of a
  correlation coefficient
• Can range from 0 to 1
• Can not be negative
• Level of reliability can be judged by the size of
  the reliability coefficient

15
General Guidelines for Evaluating Reliability

• High values of reliability coefficient indicate
  high levels of reliability
• 0.60 considered acceptable
• 0.80 considered good
• 0.90 or more considered excellent

16
Reliability Sets Maximum Variance of Analyses
17
Changing the Length of a Test

• Sometimes may want to lengthen or shorten a test
• In general, the longer the test the more reliable
  it will be
• Simple formula to estimate reliability after
  lengthening or shortening
• Assumes that only homogenous items added to
  measure

18
Changing the Length of a Test

• Where
• rtt is the predicted reliability
• n is the ratio to which the test is lengthened or
  shortened
• rtt is the original reliability

19
Example of Increasing the Length of a Test
20
Example of Shortening the Length of a Test