Variational Bayes Model Selection for Mixture Distribution

1
Variational Bayes Model Selectionfor Mixture
Distribution
Authors Adrian Corduneanu Christopher M. Bishop
Presented by Shihao Ji Duke University Machine
Learning Group Jan. 20, 2006
2
Outline

• Introduction model selection
• Automatic Relevance Determination (ARD)
• Experimental Results
• Application to HMMs

3
Introduction

• Cross validation
• Bayesian approaches
• MCMC and Laplace approximation
• (Traditional) variational method
• (Type II) variational method

4
Automatic Relevance Determination (ARD)

• relevance vector regression
• Given a dataset , we
  assume is Gaussian

Likelihood
Prior
Posterior
Determination of hyperparameters
Type II ML
5
Automatic Relevance Determination (ARD)

• mixture of Gaussian
• Given an observed dataset
  , we assume each data point is drawn
• independently from a mixture of Gaussian
  density

Likelihood
Prior
Posterior
VB
Determination of mixing coefficients
Type II ML
6
Automatic Relevance Determination (ARD)

• model selection

Bayesian method
,
Component elimination if
,
i.e.,
7
Experimental Results

• Bayesian method vs. cross-validation

600 points drawn from a mixture of 5 Gaussians.
8
Experimental Results

• Component elimination

Initially the model had 15 mixtures, finally was
pruned down to 3 mixtures
9
Experimental Results
10
Automatic Relevance Determination (ARD)

• hidden Markov model
• Given an observed dataset
  , we assume each data sequence is
• generated independently from an HMM

Likelihood
Prior
Posterior
VB
Determination of p and A
Type II ML
11
Automatic Relevance Determination (ARD)

• model selection

Bayesian method
,
State elimination if ,
Define -- visiting frequency
where
12
Experimental Results (1)
13
(No Transcript)
14
Experimental Results (2)
15
(No Transcript)
16
Experimental Results (3)
17
(No Transcript)
18
Questions?