Kernel Stick-Breaking Process

1
Kernel Stick-Breaking Process

• D. B. Dunson and J. Park

Discussion led by Qi An Jan 19th, 2007
2
Outline

• Motivation
• Model formulation and properties
• Prediction rules
• Posterior Computation
• Examples
• Conclusions

3
Motivation

• Consider a problem of estimating the conditional
  density of a response variable using a mixture
  model, , where Gx is an
  unknown probability measure indexed by x.
• The problem of defining priors for random
  probability measures on Gx has received
  increasing attention in recent year. For example,
  DP, DDP.

4
One model

• In DDP, the atoms can vary with x according to a
  stochastic process while the weights are fixed
• Dunson et al propose a model to allow the weights
  to vary with predictors
• while this model lacks reasonable
  marginalization and updating properties.

5
Model formulation

• Introduce a countable sequence of mutually
  independent random components
• The kernel stick-breaking process (KSBP) can be
  defined as follows

6
About the model

• The model for Gx is a predictor-dependent mixture
  over an infinite sequence of basis probability
  measures, Gh located at Gh.
• Bases located close to x and having a smaller
  index, h, tend to receive higher probability
  weight.
• KSBP accommodates dependency between Gx and Gx

7
Special cases

• If K(x,G)1 for all and GhDP(aG0), it is a
  stick-breaking mixture of DP.
• If K(x,G)1, and , we obtain
  GxG, with G having a stick-breaking prior.
• If and , we obtain a
  Pitman-Yor process.

8
Properties

• Let , we can
  obtain
• The correlation between measures

First moment
No dependency on V and G
Second moment
It can be proven and
the value ?1 in the limit as x ?x
where
9
Alternative representation
The KSBP has an alternative representation
The moments and correlation coefficient has the
form
10
Truncation

• For stick-breaking Gibbs sampler, we need to make
  truncation approximation
• Author proves that the residual weights decrease
  exponentially fast in N and an accurate
  approximation may be obtained for moderate N

The approximated model can be expressed as
11
Prediction rules

• Consider a special case in which

The model can be equivalently expressed as
12
Prediction rules

• Define and
  is a subset of the integers between 1 and n
• It can be proven that the probability that
  subjects i and j belong to the same cluster is

The predictive distribution is obtained by
marginalization
where
and denote the set of possible
r- dimensional subsets of 1,,s that include i
13
Posterior Computation
From the prior, we can obtain
1, sample Si 2, sample CSi when Si0 (assign
subject I to a new atom at an occupied
location) 3, sample ?h
14
4, sample Vh 5, sample Gh using a
Metropolis-Hastings step or Gibbs step if H is a
set of discrete potential locations
First sample and then, alternate between (i)
Sampling (Aih,Bih) from their conditional
distribution (ii) Updating Vh by sampling from
conditional posterior
15
Simulated examples
16
(No Transcript)
17
(No Transcript)
18
Conclusions

• This stick-breaking process is useful in setting
  in which there is uncertainty in an uncountable
  collection of probability measures
• The process can be applied in predictor dependent
  clustering, dynamic modeling and spatial data
  analysis, besides the density regression.
• The KSBP formulation can be applied to many tools
  developed for exchangeable stick-breaking
  processes with minimal modification.
• A predicator dependent urn scheme is obtained,
  which generalizes the Polya urn scheme