Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System - PowerPoint PPT Presentation

1 / 20

About This Presentation

Title:

Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System

Description:

... factor: 0.526. Reporter: Tsung-Yao Lee (???) Date: 2005/9/10 ... Precision and Efficiency. CAT platform (PC or PDA) does not cause significant difference on ... – PowerPoint PPT presentation

Number of Views:37

Avg rating:3.0/5.0

Slides: 21

Provided by: cda64

Category:

more less

Transcript and Presenter's Notes

Title: Design and Evaluation of an XMLBased PlatformIndependent Computerized Adaptive Testing System

1
Design and Evaluation of an XML-Based
Platform-Independent Computerized Adaptive
Testing System

Author Rong-Guey Ho and Yung-Chin Yen
Source IEEE Transaction on Education, VOL. 48,
NO. 2, May 2005
SCI 10/20 50 Impact factor 0.526
Reporter Tsung-Yao Lee (???)
Date 2005/9/10

2
Outline

Introduction
IRT-based CAT
XML-based CAT system
Experiment
Result
Conclusion
Comment

3
Introduction

CAT can be tailored to the ability of each
examinee
Based on item response theory (IRT)
Benefit
Improve the accuracy of testing
Save testing time
Require fewer items per examinee
Using XML to describe the item bank, one is able
to solve the problem while implementing a CAT
system on a different platform (PC, PDA, Handheld
device)

4
Item Response Theory (IRT)
ai item discrimination bi item difficulty ci
pseuso-guessing
ai 2.0, bi 0.0, ci 0.25 ? -3.0 to 3.0
5
Adaptive strategy

1) An initial ability is estimated in some way
2) The initial ability estimate is used to select
an appropriate item from the item bank.
3) The item is scored as either correct or
incorrect, and the examinees ability level is
reestimated
4) If the estimate is sufficiently precise or a
prespecified criterion has been reached, the
procedure is terminated. Otherwise, the estimate
is further refined by the return to Step 2

6
XML-Based CAT System
7
(No Transcript)
8
Workflow 1/3

Step 1) The item parameters of each item in the
item bank will be sent from the item retrieval
subsystem to the CAT subsystem as soon as the
system is activated.
Step 2) The system first detects the type of
platform
Step 3) Assuming the examinee is of median
ability, the system will pick an initial item of
medium difficulty and send its item number to the
item retrieval subsystem.

9
Workflow 2/3

Step 4) CAT subsystem extracts the XML-format
item from the bank and sends it to the XSL
implementation procedure.
Step 5) XSL implementation procedure generates
the HTML-format item by applying the
corresponding XSL template
Step 6) displays HTML-format item on the screen

10
Workflow 3/3

Step 7) After the answer has sent, the system
will check the answer and reevaluate current
ability
Step 8) The system will then check whether the
stopping condition is met. If the stopping
condition is not met, the next step will return
to Step 3).
Step 9) If the stopping criterion is met, the
system shows his or her final estimation of
ability on the screen and sends the response data
to response database for further analysis.

11
Experiment design

Participants
Fifty senior high school students, 95.92 had at
least one year of computer experience.
Only 28 had ever taken tests on the computer,
and only 22 of them had experience with a PDA.
Item bank
84 items from the vocabulary sections of CEEC
The value of IRT parameters of each item was
calibrated using the BILOG program following a
3-p logistic model based on 5000 high school
students
Properties in item bank

12
Procedure

Group A (n25) PC-version CAT taken first
PDA-version CAT taken second.
Group B (n25) PDA-version CAT taken first
PC-version CAT CAT taken second.
PC-CAT was administered on Intel Pentium III PC
with 128M RAM
PDA-CAT was on HP Jornada 565 and COMPAQ
iPAQH3850.

13
Precision and Efficiency
Ability
Time
Item no.

CAT platform (PC or PDA) does not cause
significant difference on
accuracy (ability)
efficiency (time, item no.)

14
Participants Response Toward Testing Environment
15
Conclusion

CAT on PDA is practical
The results may differ when experiment is of a
different age or other domains
Probably is only a result of the examinees
curiosity regarding PDA

16
Future work

Revising this system to a platform-independent,
on-line practice system
Implementing this system on other domains
Expanding the diversity of the CAT administration
platform

17
Comment

??????????,??XML????????????
???????????,??????SCORM????
???XML???????,??????SCORM????

18
IRT

IRT?Load?1980???????????(item response
theory,??IRT),????????????????,????????,??????????
?????????
1.???(Difficulty index)???????????????????????,?
???????
2.???(Pseudoguessing parameter)?????????????????
,??????????????
3.???(item-descriminating power)?????????,??????
?????????????

19
(No Transcript)
20
(No Transcript)

Write a Comment

User Comments (0)