Measurement Invariance and DIF in Comparative Research

1
Measurement Invariance and DIF in Comparative
Research

• Jerry Welkenhuysen-Gybels, PhD
• Business Decision
• Katholieke Universiteit Leuven

2
What is DIF?

• Observation Mean of group A on item I (indicator
  of latent trait W) different from mean of group B
• Interpretation Group A and B differ with respect
  to the latent trait W

3
What is DIF? (Continued)

• If interpretation is correct, then controlling
  for W (matching criterion) should remove
  relationship between group variable and item
  score
• Conclusion Item is measurement invariant across
  the groups

Item I
Trait W
Group
4
What is DIF? (Continued)

• If relationship between group and item remains
  after controlling for the latent trait W, then
  Differential Item Functioning (DIF)
• Interpretation Item measures something in
  addition to W that is differentially related to
  the group variable

Item I
Trait W
Group
5
DIF example

• Question from Scholastic Aptitude Test (SAT)
• Decoy relates to Duck as
• Spider to Web
• Lure to Fish
• Lasso to Rope
• Net to Butterfly
• Boys and girls with same verbal ability level had
  different probabilities of answering the question
  correctly
• Answering the question also requires knowledge of
  hunting and fishing

6
DIF detection techniques

• Nonparametric no assumption about the functional
  form of the relationship between latent trait W
  and item score
• Observed/latent variable Is W treated as a
  latent variable or as and observed variable
  (calculated as the sum of the item scores)

7
Observed - Parametric

• Cumulative logit model

Uniform DIF
Nonuniform DIF
8
Latent - Parametric

• Cumulative logit model with random effect

9
Observed - Nonparametric

• Mantel-Haenszel statistic
• Calculates the association between I an group (on
  the basis of a ij partial contingency table) at
  each level w of W and sums across the levels of W
• Can only detect uniform DIF

10
Use in substantive research

• Cumulative logit model (with random effect) has
  high power to detect any kind of DIF (but also
  inflated type I error rate), also if parametric
  assumption is not met
• Mantel-Haenszel statistic has highest power for
  the detection of uniform DIF

11
Issues for future research

• Observation In international research strict
  equivalence is difficult to obtain
• Comments
• More attention to scale construction
• Is strict equivalence necessary?
• Comparison of means Yes
• Use of item/scale as response or explanatory
  variable and interest in size of coefficient
  absence of uniform DIF suffices
• Use of item/scale as response or explanatory
  variable and interest in direction of
  coefficient item I needs to be significantly
  related to W and in the same direction across the
  groups

12
Issues for future research (Continued)

• Simulation studies show that the performance of
  DIF detection techniques decreases when W
  distributions of groups differ
• Techniques with either inflated type I error or
  low power
• DIF detection across (much) more than two groups
  have received few attention