Detection of Differential ItemTest Functioning DIFDTF Using IRT

1
Detection of Differential Item/Test Functioning
(DIF/DTF) Using IRT

• Stephen Stark and Oleksandr Chernyshenko
• University of Illinois at Urbana-Champaign

2
Why Study DIF/DTF Using IRT

• Researchers are often interested in comparing
  cultural, ethnic, or gender groups.
• Meaningful comparisons require that measurement
  equivalence holds.
• Classical test theory methods confound bias
  with true mean differences IRT does not.
• In IRT terminology, item/test bias is referred to
  as DIF/DTF

3
Defining DIF and DTF

• DIF refers to a difference in the probability of
  endorsing an item for members of a reference
  group (e.g., US workers) and a focal group (e.g.,
  Chinese workers), having the same standing on
  theta.
• DTF refers to a difference in the test
  characteristic curves, obtained by summing the
  item response functions for each group.
• DTF is perhaps more important for selection
  because decisions are made based on test scores,
  not individual item responses.

4
Examples of DIF
5
Procedures for Detecting DIF/DTF

• DIF
• Parametric
• Lords Chi-Square
• Likelihood Ratio Test
• Signed and Unsigned Area Methods
• Nonparametric
• SIBTEST
• Mantel-Haenszel
• DTF
• Parametric
• Rajus DFIT Method
• Nonparametric
• SIBTEST

6
Detecting DIF Using Lord's Chi-Square
vi is a vector of the differences in the
estimated item parameters for the ith item
between the focal and reference groups Si is the
variance-covariance matrix for the differences in
item parameter estimates Lords Chi-Square is
sensitive to both uniform and nonuniform DIF.
7
Detecting DIF Using Lord's Chi-Square

• Estimate item parameters and covariances for
  focal and reference groups separately.
• Obtain linking constants, A and K, for putting
  the focal and reference parameters on a common
  metric.
• Compute Lords chi-square to identify DIF items
  using the reference and transformed focal group
  parameters and their covariances.
• Once the DIF items have been identified, reequate
  the focal and reference group metrics using only
  the non-DIF items.
• Repeat steps 2 through 4 until the same items are
  identified on consecutive trials.
• This procedure is implemented in the program
  ITERLINK.

8
Using ITERLINK

• ITERLINK is an interactive program that performs
  iterative linking for the 2PL and 3PL models
  using Lords Chi-Square.
• Creates three output files
• ITERLINK.DBG
• DIF results and linking constants across
  iterations
• PAIRDIF.DBG
• Summary of DIF results
• User-named file
• Contains transformed focal parameters

9
ITERLINK.DBG
Bonferroni Corrected p .05 / items
Yes DIF No No DIF
10
PAIRDIF.DBG
Check that BILOG .cov and .3PL files were read
correctly
11
PAIRDIF.DBG
12
Example of DTF for 50-Item Test
Most focal group members expected to score about
3 points higher
13
Detecting DTF Using the DFITD4 Program

• Parametric procedure that detects DTF by
  comparing test characteristic curves.
• Determines whether DIF cancels or cumulates to
  produce DTF.
• Linking coefficients, item parameters, and
  thetas are required.
• Note What we refer to as the reference group,
  Raju calls the focal group

14
JCL File for DFITD4
15
Output File for DFITD4
16
Detecting DIF/DTF Using SIBTEST

• Nonparametric method that can be used to examine
  individual items or groups of items
• Assumes only monotonicity
• Requires only item response data
• Works well with fairly small samples (250)
• Several variations exist
• Original SIBTEST Uniform DIF
• Crossing SIBTEST Nonuniform DIF
• PolySIB Uniform DIF, polytomous data
• MultiSIB Uniform DIF, multiple dimensions

17
Using SIBTEST

• SIBTEST consists of two executable files
• SIBIN.EXE interactive, creates input file
• SIBTEST.EXE performs DIF/DTF analyses
• Choose E for either, R for reference, or F
  for focal group
• Detailed discussion of running SIBIN and SIBTEST
  is presented on the web

18
SIBTEST DIF Output