Gibbs%20Sampling%20Methods%20for%20Stick-Breaking%20priors

1
Gibbs Sampling Methods for Stick-Breaking priors

• Hemant Ishwaran and Lancelot F. James
• 2001

Presented by Yuting Qi ECE Dept., Duke
Univ. 03/03/06
2
Overview

• Introduction
• Whats Stick-breaking priors?
• Relationship between different priors
• Two Gibbs samplers
• Polya Urn Gibbs sampler
• Blocked Gibbs sampler
• Results
• Conclusions

3
Introduction

• Whats Stick-Breaking Priors?
• Discrete random probability measures
• pk random weights, independent of Zk,
• Zk are iid random elements with a distribution H,
  where H is nonatomic.
• .
• Random weights are constructed through
  stick-breaking procedure.

4
Introduction (contd)

• Steak-breaking construction
• , i.i.d. random variables.
• N is finite set VN1 to guarantee .
• pk have the generalized Dirichlet distribution
  which is conjugate to multinomial distribution.
• N is infinite
• Infinite dimensional priors include the DP,
  two-parameter Poisson-Dirichlet process
  (Pitman-Yor process), and beta two-parameter
  process.

5
Pitman-Yor Process,

• Two-parameter Poisson-Dirichlet Process
• Discrete random probability measures
• Qn have a GEM distribution
• Prediction rule (Generalized Polya Urn
  characterization)
• A special case of Stick-breaking random measure

6
Generalized Dirichlet Random Weights

• Finite stick-breaking priors GD
• Random weights pp1,..,pN constructed from a
  finite Stick-breaking procedure
• is a Generalized Dirichlet distribution (GD).
  The density for p is

f(p1,..,pN)f(pN pN-1,, p1) f(pN-1 pN-2,,
p1)f(p1)
7
Generalized Dirichlet Random Weights

• Finite dimensional Dirichlet priors
• A random measure
• with weights, p(p1,,pN)Dirichlet(?1,, ?N),
• p has a GD distribution w/ ak?k, bk?k1?N.
• Connection all random measures based on
  Dirichlet random weights are Stick-breaking
  random measure w/ finite N.

8
Truncations

• Finite Stick-breaking random measure can
  be a truncation of .
• Discard the N1, N2, terms in , and
  replace pN with 1-p1--pN-1.
• Its an approximation.
• When as a prior is applied in Bayeisan
  hierarchical model,
• the Bayesian marginal density under the
  truncation is

9
Truncations (contd)

• If n1000, N20, ? 1, then 10(-5)
  

10
Polya Urn Gibbs Sampler

• Stick-breaking measures used as priors in
  Bayesian semiparametric models,
• Integrating over P, we have
• Polya Urn Gibbs sampler
• (a)
• (b)

11
Blocked Gibbs Sampler

• Assume the prior is a finite dimensional
  , the model is rewritten as
• Direct Posterior Inference

Iteratively draw values
Each draw defines a random measure
Values from joint distribution of
12
Blocked Gibbs Algorithm

• Algorithm
• Let denote the set of current m
  unique values of K,

13
Comparisons

• In Polya Urn Process, in one Gibbs iteration,
  each data inquires existing m clusters a new
  cluster one by one. The extreme case is each data
  belongs to one cluster, ie, of cluster equals
  to of data points.
• In Blocked Gibbs sampler, in one Gibbs iteration,
  all n data points inquire existing m clusters
  N-m new different clusters. Thats the infinite
  un-present clusters in Polya Urn process is
  represented by N-m clusters in Blocked Gibbs
  sampler. Since of data points is finite, once
  Ngtn, N possible clusters are enough for all data
  even in the extreme case where each data belongs
  to one cluster.
• In this sense, Blocked Gibbs sampler is
  equivalent to Polya Urn Gibbs sampler.

14
Results

• Simulated 50 observations from a standard normal
  distribution.