1
Locally Adaptive Patch-based Image and Video
Restoration
Thank you for participating inand contributing
to our mini-symposium on

• Session I Today (Mon) 1030 100
• Session II Wed
• Same Time, Same Room

2
Local Adaptivity Patch-Based Approaches

• State of the Art Performance A Convergence of
  Ideas Extremely Popular

3
Patch-based methods have become so popular in
fact .
Patchy the Pirate
4
Multi-dimensional Kernel Regression for Video
Processing and Reconstruction

• Peyman Milanfar
• EE Department
• University of California, Santa Cruz

Joint work with Hiro Takeda (UCSC), Mattan
Protter and Michael Elad (Technion), Peter van
Beek (Sharp Labs of America)
SIAM Imaging Science Meeting, July 7, 2008
5
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

6
Summary

• Motivation
• Existing methods make strong assumptions about
  signal and noise models.
• Develop universal, robust methods based on
  adaptive nonparametric statistics
• Goal
• Develop the adaptive Kernel Regression framework
  for a wide class of problems, including video
  processing producing algorithms competitive with
  state of the art.

7
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

8
Kernel Regression Framework

• The data model

Zero-mean, i.i.d noise (No other assump.)
The sampling position
The number of samples
A sample
The regression function

• The specific form of
• may remain unspecified for now.

9
Local Approximation in KR

• The data model
• Local representation (N-terms Taylor expansion)
• Note
• With a polynomial basis, we only need to estimate
  the first unknown,
• Other localized representations are also
  possible, and may be advantageous.

Unknowns
10
Optimization Problem

• We have a local representation with respect to
  each sample
• Minimization

The regression order
N1 terms
The choice of the kernel function is open, e.g.
Gaussian.
This term give the estimated pixel value at x.
11
Locally Linear Estimator

• The optimization yields a pointwise estimator
• The bias and variance are related to the
  regression order and the smoothing parameter
• Large N ? small bias and large variance
• Large h ? large bias and small variance

The smoothing parameter
Kernel function
The weighted linear combinations of the given data
Equivalent kernel function
The regression order
12
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

13
(2D) Data-Adaptive Kernels
Classic kernel
Data-adapted kernel

• Take not only spatial distances, but also
  radiometric distances (pixel value differences)
  into account
• Data-adaptive kernel function
• Yields locally non-linear estimators

14
Simplest Case Bilateral Kernels
Spatial kernel
Radiometric kernel
.

.

.

Low noise case
15
Bilateral Kernel (High Noise)
Spatial kernel
Radiometric kernel
Effectively useless
.

.

.

High noise case
16
Better Steering Kernel Method
Local dominant orientation estimate based on
local gradient covariance
H. Takeda, S. Farsiu, P. Milanfar, Kernel
Regression for Image Processing and
Reconstruction, IEEE Transactions on Image
Processing, Vol. 16, No. 2, pp. 349-366, February
2007.
17
Steering Kernel
Steering matrix

• Kernel adapted to locally dominant structure
• The steering matrices scale, elongate, and rotate
  the kernel footprints locally.

Local dominant orientation estimation
Elongate
Rotate
Scale
18
Steering Kernel Function
Steering matrix

• Steering matrix estimation
• Naive estimate
• (Compact) Singular value decomposition

With regularization
19
Steering Kernel (Low Noise)

• Kernel weights and footprints

Steering kernel as a function of xi with x held
fixed
Steering kernel as a function of x with xi
and Hi held fixed
Footprints
Weights
Low noise case
20
Steering Kernel (High Noise)

• Kernel weights and footprints

Steering kernel as a function of xi with x held
fixed
Steering kernel as a function of x with xi
and Hi held fixed
Footprints
Weights

• Steering approach provides stable weights even in
  the presence of significant noise.

High noise case
21
Some Related (0th-order) Methods

• Non-Local Means (NLM)
• A. Buades, B. Coll, and J. M. Morel. A review of
  image denoising algorithms, with a new one.
  Multiscale Modeling Simulation, 4(2)490-530,
  2005.
• Optimal Spatial Adaptation (OSA)
• C. Kervrann, J. Boulanger Optimal spatial
  adaptation for patch-based image denoising. IEEE
  Trans. on Image Processing, 15(10)2866-2878, Oct
  2006.

SKR
NLM
OSA
22
Adaptive Kernels for Interpolation

• When there are missing pixels
• We cannot have the radiometric distance.
• Using a pilot estimate, fill the missing
  pixels
• Classic kernel regression
• Cubic or bilinear interpolation

?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
?
23
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

24
Kernel Regression in 3-D

• Setup is similar to 2-D, but..
• Data samples come from various (nearby) frames
• Signal structure is now in 3-D
• We can perform
• Denoising
• Spatial Interpolation
• Frame rate upconversion
• Space-time super-resolution

Temporal gradients
Spatial gradients
25
Kernel Regression in 3-D Cont.

• Two ways to proceed
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Roughly warp the data to neutralize large
  motions
• Implicitly capture sub-pixel motions in 3-D
  Kernel
• Motion-Aligned Steering Kernel (MASK)
• Estimate motion with subpixel accuracy
• Accurately warp the kernel (instead of the data)

26
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

27
AIMS Kernel in 3-D

• Steering kernel visualization examples

Isosurface
A plane structure
Steering kernel weights
A tube structure
28
AIMS Motion Compensation

• Large displacements make orientation estimation
  difficult.

• By neutralizing the large displacement, the
  steering kernel can effectively spread again.

The local kernel effectively spread along the
local motion trajectory.
The local kernel effectively spread along the
local motion trajectory.
Small motions
Large motions
Important The compensation does not require
subpixel accurate motion estimation, nor does it
require interpolation
29
AIMS Contains Implicit Motion
Small motion vector
Optical flow equation
Space-time gradients of roughly compensated data
Assuming the patch moves with approximate
uniformity
(Eigenvalues of C)
Homogeneous Optical Flow Vector
30
AIMS Summary

• AIMS is a two-tiered approach.
• Neutralize whole-pixel motions.
• 3-D SKR with implicit subpixel motion
  information

Steering matrices estimated from the motion
compensated data in 3-D.
31
A Simple Example of AIMS
Original EIA image
Low resolution frame (9 frames with large motions)
Lanczos (single frame)
Deblurred using BTV
AIMS with motion compensation
AIMS without motion compensation
32
Foreman Example
Input video (QCIF 144 x 176 x 28)
Lanczos (frame-by-frame upscaling)
AIMS
Factor of 2 upscaling
33
Spatial Upscaling Example
Input (200 x 200)
Upscaled image by AIMS (multi-frame, 5 frames),
400x400
34
Spatiotemporal Upscaling
Single frame steering kernel regression (400 x
400 x 20)
Input video (200 x 200 x 20)
Spatiotemporal classic kernel regression (400 x
400 x 40)
AIMS regression (400 x 400 x 40)
35
Outline

• Background and Motivation
• Classic Kernel Regression
• Data-Adaptive Regression
• Regression in 3-D
• Adaptive Implicit-Motion Steering Kernel (AIMS)
• Motion-Aligned Steering Kernel (MASK)
• Conclusions

36
Motion-Aligned Steering Kernel

• Motion is explicitly estimated to subpixel
  accuracy
• Kernel weights are aligned with the local motion
  vectors using warping/shearing
• The warped kernel acts directly on the data
• Handles large and/or complex motions

Accurate, explicit motion estimates
2-D motion-steered (spatial) kernel
1-D (temporal) kernel
37
Intuition Behind the MASK
2-D motion-steered (spatial) kernel
1-D (temporal) kernel
38
The Shapes of MASK

• Spreads along spatial orientations and local
  motion vectors.

Slices of MASK kernels
Local data
39
Implementation of MASK

• Block diagram

Spatial Steering matrix
Input video
Spatial gradient estimation
Spatial orientation estimation
Output video
MASK
Dense motion estimation
Temporal gradient estimation
Local motion
Note The choice of motion estimation algorithm
is not restricted to gradient-based
methods
40
A Comparison of AIMS and MASK

• Spin Calendar video

Input video (200 x 200 x 20)
AIMS (400 x 400 x 40)
MASK (400 x 400 x 40)
41
A Comparison of AIMS and MASK

• Foreman video

Input video (QCIF 144 x 176 x 28)
AIMS BTV deblurring (CIF 288 x 352 x 28)
MASK BTV deblurring (CIF 288 x 352 x 28)
42
Conclusions

• We extended the 2-D kernel regression framework
  to 3-D.
• Illustrated 2 distinct approaches
• AIMS Avoids subpixel motion estimation, needs
  comp. for large motions
• MASK Needs subpixel motion estimation, deals
  directly with large motions
• Which is better? Depends on the application.
• The overall 3-D SKR framework is simultaneously
  well-suited for spatial, temporal, and
  spatiotemporal
• upscaling, denoising, blocking artifact removal,
  superresolution
• not only in video but in general 3-D data sets.
• Future work
• Integration of deblurring directly in the 3-D
  framework
• Computational complexity