REVIEW I

1
REVIEW I

• Reliability scraps
• Index of Reliability
• Theoretical correlation between observed true
  scores
• Standard Error of Measurement
• Reliability measure
• Degree to which an observed score fluctuates due
  to measurement errors
• Factors affecting reliability
• A test must be RELIABLE to be VALID

2
REVIEW II

• Types of validity
• Content-related (face)
• Represent knowledge
• Use experts to establish
• Criterion-related
• Evidence of a statistical relationship w/ trait
  being measured
• Alternative measures must be validated w/
  criterion measure
• Construct-related
• Validates unobservable theoretical measures

3
REVIEW III

• Standard Error of Estimate
• Validity measure
• Degree of error in estimating a score based on
  the criterion
• Methods of obtaining a criterion measure
• Actual participation
• Experts
• Perform criterion
• Known valid test
• Interpreting r

4
Criterion-Referenced Measurement
Poor
Sufficient
Better
5
Guidelines for Writing Behavioral Objectives
(Mager , 1962)

• Identify the desired behavior/action by name
• Define the desired condition
• Specify the criteria of acceptable performance

6
Criterion-Referenced Testing

• Mastery Learning
• Standard Development
• Judgmental use experts
• Normative theoretically accepted criteria
• Empirical cutoff based on available data
• Combination expert norms typically combined

7
Advantages of Criterion-Referenced Measurement

• Represent specific, desired performance levels
  linked to a criterion
• Independent of the of the population that meets
  the standard
• If not met, specific diagnostic evaluations can
  be made
• Degree of performance is not important . . .
  reaching the standard is
• Performance linked to specific outcomes
• Individuals know exactly what is expected of them

8
Limitations of Criterion-Referenced Measurement

• Cutoff scores always involve subjective judgment
• Misclassifications can be severe
• Student motivation can be impacted
  frustrated/bored

9
Setting a Cholesterol Cut-Off
N of deaths
Cholesterol mg/dl
10
Setting a Cholesterol Cut-Off
N of deaths
Cholesterol mg/dl
11
Considerations with CRT

• The same as norm-referenced testing
• Reliability Consistency of measurement
• Validity Truthfulness of measurement

12
Statistical Analysis of CRTs

• Nominal Data
• Contingency Table Development (2x2 Chi2)
• Phi Coefficient (PPM for dichotomous variables)
• Chi-Square Analysis

13
CRT ReliabilityTest/Retest of a single measure
14
CRT Validity Use of a field test and criterion
measure
15
Figure 7.1 (a)FITNESSGRAM Standards (1987)
Below the criterion VO2max
Above the criterion VO2max
24 (4) 21 (4)
64 (11) 472 (81)
Did not achieve the standard on the run/walk
test
Did achieve the standard on the run/walk test
16
Figure 7.1 (b)AAHPERD Standards (1988)
Below the criterion VO2max
Above the criterion VO2max
130 (22) 23 (4)
201 (35) 227 (39)
Did not achieve the standard on the run/walk
test
Did achieve the standard on the run/walk test
17
Meeting Criterion-Referenced StandardsPossible
Decisions
 
Truly Below Criterion Truly Above Criterion
Did not achieve standard CorrectDecision FalsePositive
Did achieve standard FalseNegative CorrectDecision
18
Table 7.1Test-Retest Reliability Example
Day 2
Day 1 Did not achieve the standard Did achieve the standard
Did not achieve the standard 80 20 100
Did achieve the standard 50 250 300
130 270 400
P .825 K .576
Phi .586 C2 137.13, df 1, p lt .001
19
Table 7-2Criterion-Referenced Equivalence
Reliability Between the 1 Mile Run/Walk and PACER
Tests Total sample Boys Girls
Trial 1 Trial 1 Trial 1 Trial 1
P .76 .83 .66
K .51 .65 .33
Trial 2 Trial 2 Trial 2 Trial 2
P .71 .76 .65
K .43 .52 .30
20
Figure 7.3A theoretical example of the divergent
group method
21
Examples of Criterion Referenced Standards

• Cholesterol lt 240 mg / dl
• Systolic Blood Pressure lt 140 mmHg
• Diastolic Blood Pressure lt 90 mmHg
• FITNESSGRAM 1-mile run time for boy age 10 lt 1130

22
Criterion-referenced Measurement

• Find a friend
• Explain one thing that you learned today and
  share
• WHY IT MATTERS
• to you as a future professional