EDMS 738, Fall 2004 Bayesian Inference with Measurement Models

1
EDMS 738, Fall 2004Bayesian Inference with
Measurement Models

• Key ideas of the course
• Key ideas of probability-based reasoning
• Assignments
• Final project

2
Spring 2002, Fall 2003, Fall 2005Foundations of
Assessment
3
Fall 2002, Spring 2005Cognitive Psychology and
Assessment
4
Spring 2003, Fall 2004Bayesian Inference and
Measurement Models
5
Spring 2004, Spring 2006Theory-Based Task Design
Theory-Based Task Design
6
Class Web Site http//www.education.umd.edu/EDMS/
mislevy/BayesianInferenceFall2004/index.html

• Calendar
• Overheads
• Class news
• Assignments
• Examples
• Data sets
• Supplemental readings
• Links of interest

7
Key ideas of the course

• Foundations of assessment, i.e., as evidentiary
  reasoning
• Bayesian inference
• How these concepts interact

8
Key ideas of the course (1)

• Foundations of assessment, i.e., as evidentiary
  reasoning
• Where measurement models fit in the assessment
  argument
• Bayesian inference
• Key concepts in Bayesian inference
• Some recent machinery for Bayesian inference
• Discrete Bayes nets
• Markov Chain Monte Carlo estimation

9
Key ideas of the course (2)

• How the ideas interact with foundations of
  assessment
• The role of probability-based reasoning in
  assessment
• Probability-based reasoning with measurement
  models
• Bayesian estimation with measurement models
• Unified framework for thinking about a wide range
  of measurement models
• Provide you with some tools First cutnot
  especially efficient procedures
• Provide you with insight into the relationships
  among measurement models
• IRT, classical test theory, factor analysis,
  latent class models, generalizability can all be
  seen as variations on the same theme. Their
  different terminologies, notation, and computer
  programs are historical, due in part to the
  people interested in them, including their
  philosophies and purposes, and in part to the
  difficulty of writing estimation algorithms.

10
Key Ideas of Probability-based reasoning (1)

• Probability as machinery for reasoning
• Probability as degree of belief
• Probability with multiple, interrelated,
  variables
• Conditional probability
• Independence / conditional independence
• Bayes Theorem

11
Key Ideas of Probability-based reasoning (2)

• Relating probability models to the real world
• Frames of discernment
• Interpretations of probability
• Long-run frequencies
• logical degree of belief
• Subjective degree of belief
• Games of chance as paradigms for degree of belief
• Exchangeability
• Vis a vis random sampling
• Conditional exchangeability
• Independence and conditional independence

12
The structure of evidentiary arguments

• Toulmin Wigmore on structuring arguments
• Deductive, inductive, and abductive reasoning,
  vis a vis Bayesian inference
• Relating probability models to ed/psych
  measurement
• Measurement models are an essential link in an
  assessment argument, but by no means sufficient
  in and of themselves.
• ECD overview
• Psychometric principles / Role of
  probability-based reasoning

13
Bayesian inference (1)

• Basic ideas
• The full Bayesian model
• Reasoning from what you believe or learn about
  any variable in the system to any other.
• Relationship to prior/likelihood/posterior

14
Bayesian inference (2)

• Methods of computation
• Definitional / exact inference
• Conjugate priors
• Normal-normal
• Beta-binomial
• Sampling-based methods
• Common distributions
• Recurring structures

15
Graphical models / Bayes nets

• Essential idea
• Simplifications for exact inference with many
  variables
• if they have favorable structures
• Structures depend on theory, how we choose to
  express our knowledge, what we observe.
• Computation
• Some details and pointers to the literature
• Example from Mislevy 95
• ERGO, MSBNx, HUGIN, Netica

16
Sampling-based methods

• MCMC estimation
• Gibbs sampling as special case
• Little bit on Metropolis sampling
• Model-checking sensitivity analysis
• BUGS computer program
• Free download
• http//www.mrc-bsu.cam.ac.uk/bugs/welcome.shtml

17
Application to measurement models (1)

• Regression
• Classical test theory (special emphasis)
• Latent class models
• Balance-beam example from Mislevy 94
• Missing data
• Factor analysis
• Spearmans insight
• Similarity to regression

18
Application to measurement models (2)

• IRT
• Estimating person abilities via MSBNx
• Estimating item parameters via BUGS
• Possible extensions (esp. class projects)
• Ordered response categories (relationship to
  factor analysis)
• LLTM
• Rater effects
• Unfolding models
• Mixture models

19
Application to measurement models (3)

• Generalizability theory
• Hierarchical models
• Cognitive diagnosis
• Agouti example
• Simplified mixed-number subtraction example
• NetPass example?

20
Assignments

• Readings
• Assignments every week or two Problems,
  questions, work through examples
• Im giving 3,2,1,0 grades
• Self graded for details (class discussion, sample
  solutions)
• Class presentation of your project example

21
Final Project (1)

• Analysis of a data set of your choosing
• E.g., Project, job, research paper, data from
  another class
• Go for a simple version of the kind of model
  youre interested in
• Outline the evidentiary argument
• What variables did test designer decide upon?
• What decisions about exchangeability of students,
  tasks?
• Written description of problem, modeling
  approach, results
• Directed acyclic graph
• BUGS estimation

22
Final Project (2)

• Outline due 4 weeks before paper is due
• Opportunity for us to touch base before you get
  too deeply into analysis
• Could be earlier for those with interest or work
• Short (10-20 minute) presentation to class about
  your plans
• Overview of the test/study/situation
• Directed acyclic diagram
• Preliminary specification of full Bayesian model