Generalizability Theory

1
Generalizability Theory
2
D Study P x (JO)
3
(No Transcript)
4
D Study P x (JO)
5
(No Transcript)
6
(No Transcript)
7
D Study P x (OJ)
8
(No Transcript)
9
D Study P x (OJ)
10
(No Transcript)
11
(No Transcript)
12
D Study (O x J)P
13
(No Transcript)
14
D Study (O x J)P
15
(No Transcript)
16
(No Transcript)
17
D Study JOP
18
(No Transcript)
19
D Study JOP
20
(No Transcript)
21
(No Transcript)
22
Relative Decisions
23
Absolute Decisions
24
The cartoon rating study had a People x
(CartoonsArtists) design (ignoring sex, which
some preliminary analyses suggest is safe). If
people are considered the object of measurement,
then the relevant error terms are
25
The corresponding generalizability coefficients
are
26
Only three of the five sources of variance are
substantial. This has important implications for
how error can best be reduced.
27
(No Transcript)
28
Object of Measurement People
29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
But, in generalizability, anything can be the
object of measurement. Depending on the research
question, people may be a source of error rather
than the object of inference.
33
Object of Measurement Artists
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
Object of Measurement CartoonsArtists
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
In the example problem of rating the
aggressiveness of kids on a playground, occasions
may be a quite sensible object of measurement
Can aggressiveness at different times of the day
be reliably distinguished?
42
Design PJO Object of Measurement O
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
Design P(JO) Object of Measurement O
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
Design J(PO) Object of Measurement O
51
(No Transcript)
52
(No Transcript)
53
(No Transcript)
54
Generalizability theory can be extended to
experimental designs. In these designs, people
are a source of error. The generalizability
theory approach shows how reliability and power
are related.
55
Experimental Design (Persons within Groups) x
Measurement Occasions
56
Design J(PG) Object of Measurement Groups
57
Source s2 G .20 PG .30 J .20 GJ .10 J(P
G) .20
58
(No Transcript)
59
Source s2 G .20 PG .10 J .20 GJ .30 J(P
G) .20
60
(No Transcript)
61
Source s2 G .20 PG .30 J .20 GJ .30 J(P
G) .20
62
(No Transcript)
63
In experimental research, calculating coefficient
alpha for a multi-item outcome measure can be
biased by the failure to remove treatment
variance.
Treatment
Control
Interitem correlations can be inflated (or
deflated) by treatment variance.
Q2
Q1
64
The proper analysis removes the treatment
variance from the reliability calculations. This
is easily accomplished in generalizability theory
by using PeopleGroups as the object of
measurement.
65
The problem is not limited to coefficient alpha.
Any correlational analysis on post-treatment
measures can be affected by the presence of
treatment effects.
66

• But wait, theres more . . .
• Cost can be included in a D study
• Multivariate generalizability theory