Patch-based Image Interpolation: Algorithms and Applications

1
Patch-based Image Interpolation Algorithms and
Applications

• Xin Li
• Lane Dept. of CSEE
• West Virginia University

2
Where Does Patch Come from?

• Neuroscience receptive fields of neighboring
  cells in human vision system have severe
  overlapping
• Engineering patch has been under the disguise of
  many different names such as windows in digital
  filters, blocks in JPEG and the support of
  wavelet bases,

Cited from D. Hubel, Eye, Brain and Vision,
1988
3
Patch-based Image Models

• Local models
• Markov Random Field (MRF) and higher-order
  extensions (e.g., Field-of-Expert)
• Transform-based PCA, DCT, wavelets
• Nonlocal models
• Bilateral filtering (Tomasi et al. ICCV1998)
• Texture synthesis via Nonparametric resampling
  (EfrosLeung ICCV1999)
• Exemplar-based inpainting (Criminisi et al.
  TIP2004)
• Nonlocal mean denoising (Buades et al.
  CVPR2005)
• Total Least-Square denoising (HirakawaParks
  TIP2006)
• Block-matching 3D denoising (Dabov et al.
  TIP2007)

4
A Bayesian Formulation of Image Interpolation
Problem
Image prior (e.g., sparsity-based)
Likelihood (our focus here)
Unobservable data
Observable data
Model class (e.g., local vs. nonlocal)
5
A Simple Extension of BM3D
Hard thresholding
3D transform of similar patches
Basic idea combine BM3D with progressive
thresholding (Guleryuz TIP2006)
6
Interpolation of LR Images
x
y
bicubic
NEDI1
this work
31.76dB 32.36dB 32.63dB
34.71dB 34.45dB 37.35dB
28.70dB6 27.34dB 28.19dB
18.81dB 15.37dB 16.45dB
1X. Li and M. Orchard, New edge directed
interpolation, IEEE TIP, 2001
7
Go Back to Biology
rods
cone
Spatially random distribution of rod/cone cells
keeps aliasing artifacts out of our vision
8
Interpolation of Nonuniformly-sampled Images
x
y
KR
this work
DT
29.06dB 31.56dB 34.96dB
DT- Delauney Triangle-based (griddata under
MATLAB) KR- Kernal Regression-based (Takeda et
al. IEEE TIP 2007)
28.46dB 31.16dB 36.51dB
26.04dB 24.63dB 29.91dB
17.90dB 18.49dB 29.25dB
9
Modeling Spatial Randomness

• Extensively studied in geostatistics and
  environmental statistics (e.g., spatial
  distribution of animals and plants)
• Mathematically modeled by homogeneous Poisson
  process (density parameter?)
• Lack of positional differentiation
• Lack of scale differentiation
• Empirically there exist quadrant-based and
  distance-based randomness metrics

10
Monte-Carlo Based Optimization
The lower energy the more random
Iterative procedure randomly pick two locations
(one black and the other white), if swapping them
decreases the energy, accept it otherwise accept
it with some probability
11
Importance of Locations
after optimization
In biological world evolution development
before optimization
Identical reconstruction algorithm only differ
on sampling locations
12
Application intoCompressive Imaging
Random Sampling Pattern
S
interpolation
channel
quantization
sensor node
How is it different from conventional image
coding system? No bits are spent on coding the
location information (randomno cost).
13
Coding Results
R0.21bpp
ours PSNR27.85dB SSIM0.8750
SPIHT PSNR28.82dB SSIM0.8637
original
R0.81bpp
ours PSNR28.10dB SSIM0.9182
SPIHT PSNR22.98dB SSIM0.7512
original
14
Error Resilience Results
15
Conclusions

• A good image prior is useful to many processing
  tasks involving incomplete or noisy observation
• As we move from local to nonlocal models, the
  location of sampling points becomes important
  location (address) and intensity (data) are the
  same thing cited from T. Kohonen
  Self-Organization and Associative Memory
• Image processing is at the intersection of
  science and engineering- will BM3D lead to a new
  class of SOM?