Computerized Adaptive Testing: developments in theory and practice

1
Computerized Adaptive Testing developments in
theory and practice
Prof. dr. Cees Glas
2
Background information

• Research line Computerized Adaptive Testing
• Computerized Adaptive Testing Theory and
  Practice, Wim J. van der Linden Cees A.W. Glas
  (2002).
• Linear models for optimal test design Wim J.
  van der Linden (2005)
• Research line in Educational Surveys
• PISA Cycle 2009, Core B Background questionnaires

3
Adaptive Testing

• Target the difficulty of the items to the ability
  level of the students
• Motives
• Optimization of measurement precision
• Shorter tests without loss of precision
• More interesting tests for students
• More informative tests for students and teachers
• Flexibility testing on demand

4
Adaptive Testing is based on IRT(Item response
Theory Models)
5
Motive for using IRT

• Possibility of using incomplete designs
• Not everybody needs to respond to the same set of
  items
• Yet measurement is on a common scale
• Possibility of creating efficient statistically
  optimal designs

6
New developments in IRT

• Response formats polytomous items, continuous
  responses
• Multidimensional models
• Item shells, item cloning
• Modeling variability in item parameters
• Cognitive models

7
Item and Test Information

• Item and test information function
• In CAT items are selected to maximize information
  at the estimated ability of examinee.

8
Adaptive Item Selection
Information
9
Adaptive Item Selection Contd
Information
Item 1
10
Adaptive Item Selection Contd
Test
Item 1
Item 2
Information
11
Adaptive Item Selection Contd
Test
Information
Item 3
Item 2
Item 1
12
Item and Test Information Contd
Test
Information
Items
Ability
13
Item parameters must be known through
pre-testing Item Bank Calibration

• Problem how to collect proper pre-test data for
  precise estimation of item parameters
• On-line calibration

14
CAT with Content Constraints

• Adaptive individualized testing
• Psychometrically optimal
• Test content specifications
• Psychometrically optimal within content
  constraints and practical constraints
• Discrete optimization problem

15
CAT with Content Constraints

• Law School Admission Test
• content constraints
• item type constraints
• word count constraints
• answer key constraints
• gender / minority orientation
• clusters of items (testlets)
• some items contain clues to each other

16
CAT with Content Constraints

• Constraints are imposed by Linear - Programming
  techniques
• For every item i a variable is defined

17
Test assembly model
Item i is selected for the test or not.
18
Test assembly model
Item i is selected for the test or not.
At most 5 items on statistics
Items 12 and 35 contain clues to each other
Time available is 60 minutes
19
Test assembly model
Maximize information in the test
Item i is selected for the test or not.
At most 5 items on statistics
Items 12 and 35 contain clues to each other
Time available is 60 minutes
20
Exposure Control

• For reasons of efficiency
• For reasons of security

21
CAT without Item Exposure Control
Exposure Rate
rmax.25
Item
22
CAT with Sympson-Hetter
Exposure Rate
rmax.25
Item
23
Background information

• Packages for IRT calibration
• Bilog, Multilog, Parscale, Testfact
• Conquest, OPLM
• Commercial software packages for discrete
  optimization
• CPLEX (ILOG)
• AIMMS modeling software
• OPL Studio

24
CAT in Practice

• High stakes tests
• ETS, ACT, LSAC
• Problems with item bank security
• Public item banks lead to undesirable strategies
• New item types may be a solution

25
CAT in Practice

• Low stakes tests
• Selection and Placement, pupil monitoring
  systems less problematic
• Cognitive and diagnostics models in development
• Sub-optimal item selection
• Cito CAT for young children

26
Conclusion

• Computer adaptive testing gives a lot of
  possibilities of optimizing a survey study
• But it is not simple
• Logistic task software and expertise
• Scientific task measurement and survey experts