Developments and Applications of the Zinnes

1
Developments and Applications of the Zinnes
Griggs Paired Comparison IRT Model

• Stephen Stark and Fritz Drasgow
• University of Illinois at Urbana-Champaign
• Department of Industrial/Organizational
  Psychology

2
Overview

• Description of Zinnes Griggs (1974) Model
• Item Response and Information Functions
• Stimulus and Person Parameter Estimation
• EM algorithm and Bayes Modal Estimation
• Simulation studies and results
• Summary and Conclusions

3
Application of the Zinnes and Griggs
(1974)Probabilistic Unfolding Model

• Idea Rater has an ideal point that represents
  his/her perception of an employees level of job
  performance
• Task On each trial, the rater chooses the
  statement j or k that more accurately depicts the
  employees level of performance
• Assumptions raters perceptions of an
  employees performance and the performance
  described by the behavioral statements are
  normally distributed with means of

4
Equation for IRFsAn item (i) consists of a pair
of statements (stimuli)
m0 Person
mj Stimulus j
mkStimulus k
5
IRF for Stimulus-Pair j 17, k 18(m17
5.6, m18 3.8)
mj gt mkMonotonically Increasing
mj gt mkMonotonically Increasing
Slope steeper when mj - mk large
Slope steeper when mj - mk large
6
IRF for Stimulus-Pair j 21, k 22 (m21
2.2, m22 6.4)
mj lt mkMonotonically Decreasing
7
IRF for Stimulus-Pair j 7, k 8(m7 4.2,
m8 4.2)
mj mkFlat IRF P0.5Random Responding
8
Analytical Solution for Item InformationObtained
by Differentiating Equation for IRFs
9
Item Information for Stimulus-Pair j 21, k
22(m21 2.2, m22 6.4)
Information increases as mj - mk increases
Peaks at (mj mk) / 2 e.g., at m0 4.3
10
Item Information for Stimulus-Pairj 7, k
8(m7 4.2, m8 4.2)
mj mkFlat IRF P0.5Random Responding
Provides NO Information
11
Plot of Maximum Item Information vs. Absolute
Difference in SME ratings for all Items
Maximum Information when mj and mkmaximally
distant symmetric about m0 90 Max occurs when
mj - mk 2
12
Approaches to Stimulus Parameter Estimation

• Joint Maximum Likelihood (JML)
• Simultaneously estimate stimulus (structural)
  parameters and person (incidental) parameters
• Stimulus parameter estimates NOT consistent do
  not approach parameter values as sample size
  increases
• Marginal Maximum Likelihood (MML)
• Integrate out incidental parameters ( m0 )
• Compute Bayes Posterior densities
• Assume prior distribution (e.g., normal)
• Parameter estimates ARE consistent

13
MML Estimation of Stimulus Parameters using an EM
Algorithm

• Expectation (E-) Step
• Distribute the data for each rater (a 1, , N)
  probabilistically over Gaussian quadrature points
  Xh, (h 1, q )
• Discrete values of an assumed prior distribution
  in this case, a normal N(5,1)
• Compute pseudo-counts
• expected number of times stimulus j was
  preferred to stimulus k across raters
• expected total number of stimulus pairs
  attempted

14
E-Step Computing Pseudo-Counts

• dai 1 if stimulus-pair i was administered 0
  otherwise
• ua vector of responses for rater a
• P(Xh ua ) Bayes posterior density at
  quadrature point h

• P(ua Xh) Conditional probability of response
  pattern (likelihood of the data)
• A(Xh) Ordinate of the prior density function

15
Maximization (M-Step)

• Log likelihood equation to be maximized

• Expressions for first partial derivatives

16
Maximization Accomplished Numerically

• Numerical Recipes subroutine DFPMIN
• Computes parameter estimates iteratively
• Requires at least as many iterations as
  parameters to be estimated
• Provides approximate inverse Hessian values
• Use diagonal elements to get standard errors

17
Stimulus Parameter Estimation Simulation Study

• Nine Experimental Conditions
• 200, 400, 800 ratings
• 10, 20, 40 stimulus-pairs (items)
• Generating Parameters
• Estimated from job performance data
• Starting values
• Assumed rater can differentiate between poor (2),
  average (5), and excellent (8) job performance by
  examination of behavioral statements
• Real world situation, use SME ratings
• No repetition of stimuli

18
Stimulus Parameters Estimates10 Stimulus-Pairs,
200 Ratings
Similar stimuli Bias small
Disparate stimuli Bias large
Emp. SD lt Est. SE Estimated SEs Conservative
19
Stimulus Parameters Estimates10 Stimulus-Pairs,
800 Ratings
Similar stimuli Bias ? as N ?
Disparate stimuli Bias nearly same
20
Regression of Parameter Estimates Toward Mean as
Consequence of Bayesian Estimation
21
Interpolated Surface PlotBias in Stimulus
Parameter Estimates, 40 Stimulus-Pairs, 800
Ratings
BIAS
1.3

0

8
-1.3
6
8
6
4
mk
mJ
4
2
2
22
Latent Trait Estimation

• Bayes Modal Estimation
• Method of Golden Sections (Numerical Recipes)
• No derivatives needed

• Simulation study, 3 Nonadaptive Tests
• 10, 20, 40 stimulus-pairs (items)
• 100 replications at values of m0, 2.0, 2.2, 8.0
  
• SME ratings used as generating stimulus parameters

23
Bias in the Average Latent Trait Estimates10,
20, and 40 Item Nonadaptive Tests
24
Root Mean Square Error of Latent Trait
Estimates10, 20, and 40 Item Nonadaptive Tests
25
Summary and Conclusions

• Stimulus Parameter Estimation
• Good recovery of generating parameters for most
  pairings
• Exception Pairings involving very disparate
  stimuli
• Repetition might help
• Accurate SEs if 400 or more ratings
• Generally conservative
• Latent Trait Estimation
• Bias decreases as test length increases (20 items
  good)
• RMSE smallest near midpoint of scale, where test
  information greatest