Fitting Gaussian Mixtures by Handsfree RJMCMC

1
Fitting Gaussian Mixtures by Hands-free RJMCMC

• Daniel Eaton
• April 20, 2006

2
Mixture of Gaussians

• A ubiquitous model
• Used frequently in density estimation and for
  classification
• Often, number of mixture components, K, chosen
  arbitrarily, so we will estimate that as well
• We will be Bayesian, and not commit ourselves to
  a point estimate of K

Two possible explanations for the data
3
Mixture of Gaussians

• Posterior lives on
  so we will use reversible jump to sample from it
• Can write down posteriors over conditioned
  on model index ( of components) K
• Use conjugate priors for parameters
  (weights/Dirichlet, mean/Normal,
  precision/Wishart)

4
Automatic reversible jump

• Reversible Jump MCMC samplers are difficult for
  non-experts to create
• A general, automatic sampler would just need to
  know a model specification and have some means to
  evaluate probabilities pointwise, up to a
  constant
• Green (1997), Dellaportas Papageorgiou (2006)
• Gaussian mixtures by split/merge, birth/death
• Green (2003), Hastie (2005)
• Automatic reversible jump in general models

5
AutoRJ

• Peter Green (2003)
• Assume we know within-model means and covariances
• Assume we are now in model A, and will propose a
  jump to model B
• Squash and translate the current state vector (in
  A) so that it looks like a plausible state under
  B
• Dimension increasing sample some additional
  components from a Normal distribution
• Dimension decreasing discard as many components
  as necessary

6
AutoRJ
Case Unchanging dimension
Model A
Normalized
Model B
The Proposal
7
AutoRJ

• Major caveats
• Must know, or have good estimates of the
  within-model mean covariance
• Furthermore, these must be meaningful quantities
• ie. the within-model parameters should be
  distributed like Gaussians

8
AutoMix

• David Hastie, 2005
• Within-model distributions are modeled as
  mixtures of Gaussians
• These mixtures are estimated automatically using
• An adaptive random walk metropolis algorithm to
  draw within-model samples
• A mixture fitting algorithm that simultaneously
  fits the number of components their parameters
  (based on MDL criteria)
• All the user provides is a function which can
  evaluate within-model log probability, pointwise,
  up to a constant

9
AutoMix for Gaussian Mixtures

• Superficially, it appears that Hastie achieved
  the automatic sampler goal
• But, there is a major problem
• Automix expects all parameters to have support
  everywhere on the real line
• Not the case for a Gaussian mixture
• Mixing weights lie on the K-1 dimensional simplex
• Covariance matrices must be SPD
• Solution reparameterize and hack

10
AutoMix for Gaussian Mixtures

• Restrict our attention to 2-D case
• Parameterize covariance matrix by two eigenvalues
  and angle of first eigenvector
• If both eigvals strictly positive, then matrix
  will be PD
• Must restrict range of angle parameter
• Introduce notion of cyclic variables (RW on the
  cycle)
• Must ensure the eigvals always positive
• Introduce reflecting barrier at origin
• This only works because neither hack breaks the
  proposal symmetry

11
AutoMix for Gaussian Mixtures

• Tragically, there is no simple hack for the
  mixing coefficients
• SoftMax-type approach doesnt work since multiple
  parameter settings lead to the same weights
• Only thing to do is return Infinity for the log
  likelihood in case AutoMix proposes an impossible
  set of weights
• Only perform walk on K-1 weights
• Is the downfall of AutoMix for this application

12
Results

• Poor
• Tested it on synthetic data (Gaussian mixtures)
  and old faithful dataset
• Intended to compare results with Dellaportas
  Papageorgiou (2006)
• On synthetic data, software is
  good at recovering the
  generating parameters

13
AutoMix for Gaussian Mixtures

• But, trans-dimensional moves are effectively
  never accepted
• Sampler gets stuck in dimension it started in
• Worsens as number of components increases (Kgt3)
• AutoMix never proposes a valid set of weights
• Also, we expect label switching, but adaptive
  random walk algorithm never finds the extra modes
• Overall, its doubtful such an algorithm would
  ever properly explore the within-model posteriors
• Alas, I will end my cynicism here

14
Questions, Comments?
Zzzz zzz

• AutoRJ and AutoMix software can be obtained at
  www.davidhastie.me.uk
• Maybe it will work for your model!