Classical Test Theory: A Bayesian Perspective Part 3: MCMC

1
Classical Test TheoryA Bayesian Perspective
(Part 3 MCMC)

• Robert J. Mislevy
• University of Maryland
• March 6, 2003

2
Topics

• Review of the full Bayesian model
• Overview of Gibbs Sampling in the CTT full
  Bayesian model
• Review of inference about qs when population
  parameters are known.
• Inference about the population mean
• Inference about the population variance
• Inference about the error variance
• Inference about reliability

3
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• The measurement model For Person i, normal
  distribution for test scores Xij, with mean qi,
  and precision te (i.e., variance )
• xi,jdnorm(thetai,taue)

4
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Normal prior on qs, with unknown mean m and
  precision tq.
• thetaidnorm(mu,tautheta)

5
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Normal prior on m, with given parameters m0 and
  precision t0.
• mudnorm(muzero,tauzero)

6
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Gamma prior on t0, with given parameters a0 and
  b0. -- conjugate prior for precision in a normal
  distribution. Centered around a0 / b0 , more
  spread as b0 gt0, more concentrated as b0 gt .
• tauthetadgamma(atheta,btheta)

7
A full Bayesian model BUGS DAG
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Tests j
Persons i
bq

• Gamma prior on te, with given parameters ae and
  be. This too is a conjugate prior for precision
  in a normal distribution.
• tauedgamma(ae,be)

8
Gibbs Sampling in the CTT model

• Bayesian inference about individual students qis
  when draws of population parameters are treated
  as known (normal version of Kelleys formulas)
• Estimation of population mean, based on qi draws,
  drawn population variance, and normal hyperprior
• Estimation of population variance, based on qi
  draws, drawn population mean, and gamma
  hyperprior
• Estimation of error variance, based on qi draws
  and gamma hyperprior
• Calculation of r each cycle

9
Full conditional for qi
m0
m
ae
Xij
te
qi
t0
be
aq
tq
Always known
Tests j
Persons i
bq

• The measurement model For Person i, normal
  distribution for test scores Xij, with mean qi,
  and precision te (i.e., variance )
• xi,jdnorm(thetai,taue)

Previous draw treated as known
10
Full conditional for qi

• Full conditional for qi means that we treat as
  known
• and as always, xi.
• Other qs and hyperparameters are not relevant.
• Full conditional for qi is normal, a la Kelley
• Take a draw, .

11
Full conditional for m
m0
m
ae
Xij
te
qi
t0

be
aq
tq
Tests j
Previous draws treated as known
Persons i
bq
Always known
12
Full conditional for m

• Full conditional for m means that we treat as
  known
• and as always, m0 and
  t0.
• xs and other hyperparameters are not relevant.
• For n qt1s, the full conditional for m is
  normal
• where .
• Take a draw, m t1.

13
An aside on Bayesian estimation of precision (1)

• In the CTT setup, we are using the conjugate
  priors for both the precision parameters
  corresponding to population and error variance.
• These are gamma(a,b) priors. Notation for the
  parameters varies among BUGS, Gelman, and
  elsewhere.
• We take a brief detour to take a look at
  estimation of precision with a gamma prior.

14
An aside on Bayesian estimation of precision (2)

• The range of gamma(a,b) is the real line.
• agt0, bgt0.
• The mean of gamma(a,b) is a/b, and the spread is
  greater as b increases (details follow soon).
• The gamma(a,b) distribution is a natural
  conjugate prior for the precision in the normal
  distribution with mean known (as is appropriate
  in Gibbs sampling).
• There are fairly simple relationships among
  (a,b), the mean variance of gamma(a,b), and
  sample size and sums-of-squares in the
  observation of a sample from a normal
  distribution with a known mean.

15
An aside on Bayesian estimation of precision (3)
16
An aside on Bayesian estimation of precision (4)

• For basing priors on previous data, use third
  column.
• For basing priors on other belief, use second or
  third.
• A reasonable mild prior is gamma(.5, 1).

17
An aside on Bayesian estimation of precision (5)

• If...
• Prior t Gamma(a,b)
• Likelihood y N(m,t) with m known
• So sufficient statistics are n and SSS(m-yi)2
• Then
• Posterior t Gamma(a n/2, b SS/2).

18
An aside on Bayesian estimation of precision (6)

• Prior t Gamma(5,5)
• Data n10,
• y(2,2,2,2,2,-2,-2,-2,-2,-2)
• i.e., SS40 thus about gamma(5,20)
• Posterior t Gamma(10,25).

normalized
19
An aside on Bayesian estimation of precision (7)
20
Full conditional for tq
m0
m
ae
Xij
te
qi
t0

be
aq
tq
Tests j
Previous draws treated as known
Persons i
bq
Always known
21
Full conditional for tq

• Full conditional for tq means that we treat as
  known
• and as always, aq and
  bq.
• xs and other hyperparameters are not relevant.
• For n students, the full conditional for tq is
  gamma
• where .
• Take a draw, tq t1.

22
Full conditional for te
m0
m
ae
Xij
te
qi
t0

be
aq
tq
Tests j
Previous draws treated as known
Always known
Persons i
bq
23
Full conditional for te

• Full conditional for te means that we treat as
  known
• and as always, each xi, ae and
  be.
• Other parameters and hyperparameters are not
  relevant.
• For n students, the full conditional for te is
  gamma
• where --i.e.,
  pooled within.
• Take a draw, te t1.

24
MCMC inference for r

• In each Gibbs cycle t, calculate rt
• Can monitor in BUGS with density, statistics,
  history, etc.