???????? (generalizability theory) ?????????? (performance assessment)

1
???????? (generalizability theory) ??????????
(performance assessment)

• ?????????????
• ?? ?
• murakou_at_orion.ocn.ne.jp

2
Contents

• ??????????
• ??????????????????????????

3
Contents

• ??????????
• ??????????????????????????

4
????????

• ??? (reliability) ?????????
• ?????
• ??????????????????
• ??????????????
• ????
• ?????,??????,Cronbach?a??

5
?????????

• ???????????????????

????????
????????
????????
? ????????????????????(????????????)
6
??????????????

• ???????????????????
• ???????????????????
• (?????????)???????????????a??
• ????????????????????????????

????????????? ????????????? ???????????????
7
???????? (generalizability theory) ??

• ??????????????????????????????2???????
• Generalizability (G) study
• ???? (ANOVA) ???????,???????????????????????
• Decision (D) study
• G study ?????????????,?????????????????????

8
??

• ? (facet)
• ???,??,?????????????.??????????,???????????????.
• ????? (object of measurement)
• ?????,?(??????).?????????????????.
• ?????(universe)
• ?????????????(??????).
• ??????????????(population)????????????????????.

9
??? (Brennan et al., 1996 ??)

• California Assessment Program (CAP)
  ??????????(????????????)
• 5???? (t)
• 3????? (r) ??????????
• ??????1?-4?
• ??????????? (p)

p x t x r ??????? (??????????) ????????????? p x
(r t) ???? (AB
?A?B????????????)
10
G study

• ???????????????? (?????MS????????)???
• ????????????????????
• p x t x r ???????,???????????????,p, t, r, pt,
  pr, tr, ptr e ? 7 ?.

? ? ??????????????????????(????1????????????????
???)
11
CAP???
??????????????????,?????????????????????? ?
???????(?????????????)
Variance components Estimates of s
Persons (p) 0.298
Tasks (t) 0.092
Raters (r) 0.003
pt 0.493
pr 0.000
tr 0.002
ptr e 0.148
????????(?????????)????.????????????,????????
????????????
???????????????????(A??B?????1?????,??2?B????A????
?)
????????????????????
??????????????
12
G study ????????

• ????????????????????????
• ???,????????????????????? ? ???(??????????????????
  ???????)
• ????????????????????(??????)

13
D study
G study ????? ?1?????????????(e.g.,
A????P???????a????)
?????????? ?5???????????2?????????????????

• D study ??????????
• ????????????????????????(??????)????
• ?????4???????????????????????????????????

14
p x t x r ??????????
Variance component in D study (??????????????)
Variance component in G study
?????????????
??????????????????(???????????)
15
2????????

• Dependability Coefficient F
• ????????????
• Generalizability Coefficient ?2 (or E?2)
• Person???????????????

16
F??2 ???

• ???? ? ??????????

??????????????
???????
??A(?) 1????(95?)2?????(90?)3???(80?) 55??
??(35?)56????(30?)
??B(?) 1????(55?)2?????(50?)3???(45?) 55??
??(5?)56????(0?)
????(????????????????).??????? ??????
???????
17
F??2 ???

• ???? ? ??????????

???????(e.g., 60?)????????
???????
??A(?) 1????(95?)2?????(90?)3???(80?) 55??
??(35?)56????(30?)
??B(?) 1????(55?)2?????(50?)3???(45?) 55??
??(5?)56????(0?)
???(????????60????????????).
???????
18
2???????????

• ????????????????? ??????????
• ????????????
• ??????????????
• Cronbach?a??? p x I ????????????????
• ????????????????(e.g., standard setting??)?,
  ??????????

19
CAP???
Variance components G studyEstimates D studyEstimatesnr3, nt5
Persons (p) 0.298 0.298
Tasks (t) 0.092 0.018
Raters (r) 0.003 0.001
pt 0.493 0.099
pr 0.000 0.000
tr 0.002 0.000
ptr e 0.148 0.010
E?2 0.73 F 0.70
3 ?????,5 ???????????????????? (?????????3?????,5?
???????????????)
20
?????????????????

• ??????????(???)????????????

nt ? nr ??????????,?????
Efficient ?????????!
21
CAP???

• Brennan (1996) ??

??????????????????????????? ??????
???????????????????? ??????
22
D study ????????

• 3?????,5????????????????????
• ???????????????????????.
• ??????????????????????(????????,??????,???????????
  ????).

23
???????????1

• Nested ????
• G study, D study ??????????
• Nested?????????Balanced design
• ??????????????
• ??????????? ? ????????
• ??? (Searle et al., 1992),????Analogous ANOVA
  (Brennan, 2001) ????
• ???????????(??)????

24
???????????2

• ???????????
• ???????????
• ???? (fixed factor ) ?????????????????????????????
  
• ?????(2003)
• ?????????????????????????????????????,???????????
  ???????????

????????????,random factor??????,??????????????
25
???????????3

• ????????? (see Brennan, 2001)
• ??????????????????????????
• ???????????????????
• ??????
• G study SAS?varcomp, mixed
• D study SAS??? -gt http//flash.lakeheadu.ca/boc
  onno2/gtheory/gtheory.html
• GENOVA (Brennan) http//www.education.uiowa.edu/
  casma/GenovaPrograms.htm

26
Contents

• ??????????
• ??????????????????????????

27
???????????????

• ????????????????????????,???????????????,????????
  ????!?
• ???????? (inter-rater reliability) ??????????????
• ????????????????.?????????!?

????,?????????.???
28
???????????,?????????!
Brennan (1996) ??????????
????????????,?????????????????!
29
CAP???reprise

• Brennan (1996) ??

??????????????????????????? ??????
??????????????????!(????????????)
???????????????????? ??????
30
Inter-rater reliability??????

• ????,????????2?????????????????
• ???????????

???
p x t ??????????,?????????????(Brennan, 2000)
???????????????????????
???????????????????????
31
????????????
Ruiz-Primo et al. (1993)
o occasion ????(5??)
Source VC () Source VC ()
p 0.07(4) ro 0.00
r 0.00 to 0.00
t 0.00 prt 0.00
o 0.01(1) pro 0.01
pr 0.01(1) pto 1.16(59)
pt 0.63(32) rto 0.00
po 0.00 prto e 0.08(4)
rt 0.00
???2??????????????????????????
E?2 0.04, F 0.04
32
?????????

• ?? (2008)?????????????
• ?????????
• ?????????????
• ???(?????????????????)?????? r 0.00 0.15 (!)

33
???

• ???????????????,?????????????????????
• ?????????
• ???,?????????????????????? p x t
  ???(?????????????????????????)
• ?????????????
• ???????????????????
• ?????????????????????????
• ???(occasion??)?????????????

34
?? 1

• ????????
• ?????
• Cronbach, Gleser, Nanda, Rajaratnam (1972). The
  dependability of behavioral measurements.
• ??????
• Brennan (1992). Elements of geeneralizability
  theory (rev. ed.).
• Shavelson Webb (1991). Generalizability theory
  A primer.
• ????????
• Brennan (2001). Generalizability theory.
• ??????
• ???????(???)
• ?????(Linn?,??????) ?3?

35
?? 2

• ??????????????????????
• ????
• Brennan (1996). Generalizability of performance
  assessments. In Phillips (Ed.). Technical issues
  in performance assessments.
• Cronbach et al. (1997). Generalizability analysis
  for performance assessments of student
  achievement or school effectiveness. Educational
  and Psychological Measurement.
• ??
• Ruiz-Primo et al. (1993). On the stability of
  performance assessments. Journal of Educational
  Measurement.
• Shavelson et al. (1993). Sampling variability of
  performance assessments. Journal of Educational
  Measurement.
• Webb et al. (2000). The dependability and
  interchangeability of assessment methods in
  science. Applied Measurement in Education.