Assessing Local Item Dependence in Building Explanation Tasks

1
Assessing Local Item Dependence in Building
Explanation Tasks
------An Application of the Multidimensional
Random Coefficients Multinomial Logit Item Bundle
Model

• Han Bao Dr. Robert J. Mislevy
• University of Maryland, College Park

We would like to thank Nancy Butler Songer and
Amelia Wenk Gotwals of the BioKIDS project for
permission to use the data in the example, and
their time, patience, and insights in helping us
understand the substantive aspects of the
modeling described here. We are grateful to
Larry Hamel and Cathy Kennedy for lots of help
along the way.
AERA April 2005
2
Item Bundle MRCML Model

• Assumption Bundle Independence implies that a
  bundle of items are expected to be dependent and
  are assumed local independence across bundles.
• Item Bundle MRCML Model --- Item bundle nested in
  Multidimensional Random Coefficient Multinomial
  Logit (MRCML) model

3
BioKIDS Assessment Summary

• Content Areas
• Biodiversity
• Simple Machine
• Inquiry Skills
• Hypothesis/Predictions
• Building Explanations from Evidence
• Interpreting data
• Re-express data

4
Two Structurally Similar Examples from BioKIDS
BioKIDS04
5
Two Structurally Similar Examples from BioKIDS
BioKIDS05
6
Analysis Design
Biodiversity Combined Inquiry

• I. Dimensionality
• One-Dimension----------
• Two-Dimension----------
• Five-Dimension---------

7
Analysis Design
II. Dimensionality

• Parameterization of Not bundled Model of
  BioKIDS04 (Claim)
• Parameterization of Not Bundled Model of
  BioKIDS04 (Evidence)

8
Analysis Design

• Parameterization of Bundled Model of BioKIDS04

Note item bundle parameter ( ,
, , , ) , for
example represent for step difficulty
parameter associated with response category of
evidence equals 1 conditioning on claim equals 0,
etc.
9
Results of Chi-Square

• BioKIDS04
• BioKIDS05
• Note Chi-Square critical value is 15.0863

10
Results of Deviance
11
Conclusions

• A multidimensional item bundle analysis suggest
  that the BioKIDS Fall 2003 Pretest exhibits item
  local dependence in at least one cluster of items
  based on a common stimulus situation.
• Taking dimensionality and idiosyncratic features
  of test format into consideration makes the
  analysis of local item independence more accurate
  and meaningful.
• Modeling such data using the MRCML item bundle
  model can deal with item dependence and
  dimensionality simultaneously and reduce
  distortions in item parameter estimates as well
  as proficiency estimates.
• Thank you !