Empirical Bayes approaches to thresholding

1
Empirical Bayes approaches to thresholding

• Bernard Silverman, University of Bristol
• (joint work with Iain Johnstone, Stanford)
• IMS meeting 30 July 2002

2
Finding needles or hay in haystacks

• Archetypal problem n noisy observations
• Sequence ?i may well be sparse, but not
  necessarily
• Assume noise variance 1 no restriction in
  practice

3
Examples

• Wavelet coefficients at each level of an unknown
  function
• Coefficients in some more general dictionary
• Pixels of a nearly, or not so nearly, black
  object or image

4
Needles or hay?

• Needles rare objects in the noise
• if the observation was 3, would be inclined to
  think it was straw

• Hay common objects
• non-sparse signal
• if the observation was 3, would be inclined to
  think it was a nonzero object

A good method will adapt to either situation
automatically
5
Thresholding

• Choose a threshold t
• If Yi is less than t, estimate ?i 0
• If Yi is greater than t, estimate ?i Yi
• Gain strength from sparsity if sparse a high
  threshold gives great accuracy
• Data-dependent choice of threshold is essential
  to adapt to sparsity a high threshold can be
  disadvantageous for a dense signal

6
Aims for a thresholding method

• Adaptive to sparse and dense signals
• Stable to small data changes
• Tractable, with available software
• Performs well on simulations
• Performs well on real data
• Has good theoretical properties
• Our method does all these!

7
Bayesian Formulation

• Prior for each parameter is a mixture of an atom
  at zero (prob 1-w) and a suitable heavy-tailed
  density ? (prob w)
• Posterior median is a true thresholding rule
  denote its threshold by t(w)
• Small w ? large threshold, so want small w for
  sparse signals, large for dense

8
Other possible thresholding rules

• hard or soft thresholding with the same threshold
  t(w)
• posterior mean not a strict thresholding rule
• Posterior probability of non-zero gives
  probability that pixel/coefficient/feature is
  really there
• threshold if this prob is lt 0.5
• threshold for some larger prob?
• Mean and Bayes factor rules generalize to complex
  and multivariate case

9
Empirical Bayes Data-based choice of w

• Let g convolution of ? and normal density
• Marginal log likelihood of w is
• computationally tractable to maximize
• automatically adaptive gives large w if a large
  number of Yi are large and vice versa

10
Example

• Six signals of varying sparsity
• Each has 10000 values arranged as an image for
  display
• Independent Gaussian noise added
• Excellent behaviour of the MML automatic
  thresholding method is borne out by other
  simulations, including in the wavelet context

11
(No Transcript)
12
(No Transcript)
13
Root mean square error plotted against threshold
14
Root mean square error plotted against threshold

• Much lower RMSE can be obtained for sparse
  signals with suitable thresholding
• Best threshold decreases as density increases
• The MML automatic choice of threshold is excellent

15
Estimates obtained with optimal threshold
16
Theoretical Properties

• Characterize sparseness by n-1??ip ? ? p
  for some small p gt 0
• Among all signals with given energy (sum of
  squares), the sparsest are those with small lp
  norm
• For signals with this level of sparsity, best
  possible estimation MSE is O(? p log ?(2-p)/2)

17
Automatic adaptivity

• MML thresholding method achieves this best mean
  square error rate, without telling it p or ?, all
  the way down to p0
• Price to pay is an additional O(n-1 log3 n) term
• Result also works if error is measured in q-norm,
  for 0 lt q ? 2

18
Adaptivity for standard wavelet transform

• Assume MML method is applied level by level
• Assume array of coefficients lies in some Besov
  class with 0 lt p ? 2 allows for a very wide
  range of function classes, including very
  inhomogeneous
• Allow mean q-norm error
• Apart from an O(n-1 log4 n) term, achieve minimax
  rate regardless of parameters.

19
Conclusion to this part

• Empirical Bayes thresholding has great promise as
  an adaptive method
• Wavelets are only one of many contexts where this
  approach can be used
• Bayesian aspects have not been considered much in
  practical contexts if you want 95 posterior
  probability that a feature is there, you just
  increase the threshold