Classical Test Theory: A Bayesian Perspective

1
Classical Test TheoryA Bayesian Perspective

• Robert J. Mislevy
• University of Maryland
• November 1, 2001

2
Topics

• Review of the (normal) CTT model
• Frequentist and likelihood inference about
  individual students
• The full Bayesian model
• Bayesian inference about individual students,
  given higher-level parameters
• Collateral information about students
• MCMC estimation

3
Notation for CTT

• qi True score of Person i
• eij Independent measurement error, Person i on
  Test j
• Xij Observable score of Person i on Test j, qi
  eij
• Xs are parallel tests.
• m Population (true-score) mean of qs
• Population (true-score) variance of qs
• tq Population precision, ,
• Measurement (Error) variance ie, of xs given
  qs
• te Measurement precision, ,

4
Some Basic Formulas

• Expectation of Person is scores is qi ie,
  E(Xij qi) qi
• Errors are independent, and they are unbiased
  ie,
• "i"j E(eij qi) E(eij) 0.
• Variance of Person is scores is ie,
  Var(Xij qi)
• Observed score mean true score mean ie,
• E(Xij) E(qi) m
• Observed variance true variance error
  variance ie,
• Assumptions about moments, but not about
  distributions.

5
Reliability (1)

• Typically, Xi is taken as the estimate of qi.
  (more on this later)
• Reliability
• Correlation of two sets of observed scores is r.
• Correlation of observed score true score is Ör.

6
Reliability (2)

• Spearman-Brown formula. If the reliability of X
  is r, the reliability of a double-length test
  (X1 X2 )/2 is
• More generally, if the reliability of X is r, the
  reliability of a test of length n times as long
  is

7
Frequentist inference about individual students
(1)

• No assumption about forms of distributions
  inference just depends on first two moments
  (means, variances, covariances) and independence
  assumptions.
• Any Xij, by itself, is an unbiased estimate of
  qi.
• The standard error of measurement of Xij as an
  estimate of qi is se, the standard deviation of
  Xijs around qi.
• Note that this standard error corresponds to a
  precision of te

8
Frequentist inference about individual students
(2)

• Suppose n tests are given to each student.
  Denote by
• the average score of Student i.
• is an unbiased estimate of qi.
• The standard error of measurement of as an
  estimate of qi is se / Ön, the standard deviation
  of around qi.
• This standard error corresponds to a precision
  of nte .

9
Likelihood inference about individual students (1)

• Form of measurement-model distribution must be
  specified to carry out likelihood inference.
• Well assume , with
  known.
• The likelihood function for qi induced by
  observation of xij, or viewed
  as a
• function of q given xij, is
• The MLE is xij.
• The standard error is se
• again precision is te.

se
xij
10
Likelihood inference about individual students (2)

• Suppose n tests are given to each student.
• Again each , with
  known.
• The likelihood function for qi induced by
  observation of n xijs, or
  , is
• The MLE is
• The standard error is se /Ön
• The precision is nte.

se/Ön
11
The full Bayesian model

• A single-plate MSBNx approximation
• A BUGS graph
• Recursive expression of the joint probabilities

12
A single-plate approximation (a la MSBNx)

• Addresses one person
• Includes the q and all test scores X for that
  person
• q population distribution structure implicit, in
  q prior
• Measurement distribution implicit, in conditional
  probability distribution for observable test
  scores
• Reliability doubly implicit, as function of
  variances (precisions) implicit in distributions
  for q and Xq .

13
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Distributions for all parameters variables
• Addresses all tests and all people
• Plates for people and tests within people
• People modeled as exchangeable, test scores as
  conditionally exchangeable given qis.

14
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Whats explicit in MSBNx for a single examinee

15
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Distribution parameters

16
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Hyperparameters, or parameters of the
  distributions of the parameters in the priors.
  We will assign them fixed, mild values--i.e.,
  inference will be conditional on their values.

17
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• The measurement model For Person i, normal
  distribution for test scores Xij, with mean qi,
  and precision te (i.e., variance )
• xi,jdnorm(thetai,taue)

18
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Normal prior on qs, with unknown mean m and
  precision tq.
• thetaidnorm(mu,tautheta)

19
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Normal prior on m, with given parameters m0 and
  precision t0.
• mudnorm(muzero,tauzero)

20
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Gamma prior on t0, with given parameters a0 and
  b0. -- conjugate prior for precision in a normal
  distribution. Centered around a0 / b0 , more
  spread as b0 gt0, more concentrated as b0 gt .
• tauthetadgamma(atheta,btheta)

21
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Gamma prior on te, with given parameters ae and
  be. This too is a conjugate prior for precision
  in a normal distribution.
• tauedgamma(ae,be)

22
The full Bayesian model Recursive expression of
the joint probabilities
The measurement model Normal distribution for
test score given person is true error variance
(1/precision) Normal distribution for true
scores Normal prior for population true score
mean Gamma prior for population true score
precision Gamma prior for error term precision
23
Bayesian inference about individual students

• Bayesian inference about the mean of a normal
  distribution when variances are known
• Kelleys formulas
• Mean and variance decompositions

24
Bayesian inference about the mean of a normal
distribution when variances are known (1)

• The distribution were interested in is N(m,t).
• t is known want to draw inferences about m.
• Two sources of information about m
• Prior. Conjugate prior for mean of a normal
  distribution when variance is known is also the
  normal distribution say N(m0,t0), with both m0
  and t0 known. m0 is center t0 small for diffuse
  prior, big for tight prior.
• Data. Will observe draws from N(m,t). One draw
  likelihood is N(x,t). N draws likelihood is
  N(x,nt). (Note written in BUGS precision
  format.)

25
Bayesian inference about the mean of a normal
distribution when variances are known (2)

• The posterior for m is normal too. For one x,

Prior N(m0,t0)
Likelihood N(x,t)
Posterior
Posterior mean is a weighted average of prior
mean and mean of the data. Weight of prior mean
is proportional to prior precision. Weight of
data mean is proportional to data precision.
26
Bayesian inference about the mean of a normal
distribution when variances are known (3)

• The posterior for m is normal too. For one x,

Prior N(m0,t0)
Likelihood N(x,t)
Posterior
Posterior precision is the sum of prior precision
and data precision.
27
Bayesian inference about the mean of a normal
distribution when variances are known (4)

• For n xs, the posterior for m is normal too.

Prior N(m0,t0)
Likelihood N(x,nt)
Posterior
28
Bayesian inference about the mean of a normal
distribution when variances are known (5)
A pictorial example with a prior with less
information than the data
m0
x
Prior N(m0,t0)
Likelihood N(x,t)
Posterior
29
Bayesian inference about the mean of a normal
distribution when variances are known (5)
A pictorial example with a prior with more
information than the data
m0
x
Prior N(m0,t0)
Likelihood N(x,t)
Posterior
30
Kelleys formulas One test score

• Population distribution is N(m,tq), with m and tq
  known.
• Well assume , with
  known.
• The likelihood function for qi induced by
  observation of xij is
• Posterior for qi is normal too

31
Kelleys formulas n test scores

• Population distribution is N(m,tq), with m and tq
  known.
• Well assume , with
  known.
• The likelihood function for qi induced by
  observation of n xijs for Examinee i is
  
• Posterior for qi is normal too
• with .