Image SuperResolution Using the MumfordShah Functional

1
Image Super-Resolution Using the Mumford-Shah
Functional

• Todd Wittman
• Department of Mathematics
• University of Minnesota
• Lotus Hill Vision Conference
• EZhou, China
• August 2005

2
The Super-Resolution Problem

• Intuitive Definition Given a low-resolution
  image(s), produce an aesthetically pleasing
  high-resolution image.
• Mathematical Definition ???
• Single-image super-resolution (interpolation)
• Multiple-image super-resolution
• Many applications web browsing, HDTV, satellite
  imaging, medical diagnosis, surveillance video,
  recognition

3
The Goal of Computer Vision
What the heck are we trying to do here?
-Song-Chun Zhu
Low-level Vision
Mid-level Vision
High-level Vision
Understanding
Image
Super-resolution Denoising/Deblurring Registration
Compression
Segmentation Inpainting
Detection Recognition Scene Categorization
How do we extract the information from an image?
How do we quantify the information content in an
image?
My Goal If we have a good model to measure
information, then we can use it to interpolate a
high-resolution image.
4
Linear Interpolation Filters

• The most common interpolation filters are
  essentially a weighted average of nearby pixels.

Known
Interpolated

• Blurs edges, oversmooths texture, aliasing
  (staircasing), ringing artifacts.
• May make sense for interpolation of a general
  data set, but not a good model for visual
  information.

5
The Mumford-Shah Model
O
G
Given an initial image u0 defined on a domain O,
we seek a better image u with corresponding edge
set G by minimizing the following energy
Total length of edges
Smoothness away from edges
Matching to original image
Mumford-Shah,89
6
Single-Image Interpolation
Given an image u0 defined on a domain D and a
zoom factor Kgt1, we want to produce an image u on
a finer lattice . (Note K1 just
produces the original lattice D.)
For square lattices and , this reduces
to filling in pixels in between pixels.
Zoom by factor K3
Inpaint
Interpolant Image u Lattice O3
Original Image u0 Lattice D
Separate by K-12 pixels
Unknown Pixels
Interpolation is a special case of inpainting.
Bertalmio-Bertozzi-Sapiro,01
7
Can we remove the columns?
Mumford-Shah inpainting was solved by Jackie Shen.
Original Mask over columns
MS Inpainting
Click for movie.
Texture is still a problem.
Shen-Esedoglu,02
8
Mumford-Shah Interpolation
We only need to match to the original image u0 on
the original domain D.
Numerically, we approximate the edge set G with
an edge canyon z.
1
0
e
e
G
The gamma convergence formulation is
Shen-Esedoglu,02, Tsai-Yezzi-Willsky,01
9
Mumford-Shah Interpolation
10
Multiple-Image Super-Resolution
Given a sequence of N low-resolution images
For a zoom factor Kgt1, we want to
produce a single high-resolution image
containing more information
than any one image in the sequence.
Our first step is to register the images to the
base high-resolution space OK.
Register by matching selected landmarks under
projective transformations
D1
D2
OK
OK
Im cheating! Landmarks are selected from
high-resolution versions. (This is common in the
super-resolution literature, e.g. Baker-Kanade.)
11
Mumford-Shah Super-Resolution
On average, match the original images only on the
mapped domains
Numerically, the gamma convergence formulation
changes just slightly
The Euler-Lagrange equations become
12
Super-Resolution of a Video Sequence
Result on 5 frames from a slow-moving camera.
The image gets fuzzier on the ends where there is
less information.
13
Color Super-Resolution
For color images, we can process each channel
separately.
R
G
B
Click for movie.
14
The Power of Multiple Images
Our results are better than any single-image
interpolation scheme.
Limits of SR The super-resolution seems to
break around zoom K4.
15
Useful By-Products
Super-segmentation Keeping track of the edge
set, we also segment the image.
Parameters are adjusted for interpolation, not
segmentation.
Super-denoising Denoising is difficult to do
alongside interpolation.
Adjusting parameters to remove noise will
increase blur.
16
Video Super-Resolution

• In a video sequence, we repeatedly align to the
  ith frame.
• But the first and last frame may be very
  different.
• We expect nearby frames in a sequence to resemble
  each other (Temporal Markov Property).
• We can adjust the matching weight ß to reflect
  this.

To super-resolve the ith frame of a video 1.)
Align the images to the ith frame. 2.) Set the
weights ßi to a Gaussian centered at the ith
frame.
ß
ui1
ui-1
ui
17
Video Super-Resolution
We can super-resolve a video by aligning the
images to each frame.
Low-resolution video
Mumford-Shah super-resolution
Click for movie.
Click for movie.
18
Comparison to Learning-Based Methods
Compare to strengths and weaknesses of LBSR
methods that Dr. Zhouchen Lin discussed.

• Mumford-Shah SR
• Relatively slow (minutes)
• Dependent on parameters
• Oversmooths texture
• Low zoom factor
• Deterministic
• Generic, can be tweaked to task (medical, text,
  binary, etc.)

• Learning-Based SR
• Very fast (seconds)
• Dependent on training set / prior knowledge
  (eigenface)
• May interpolate texture (depending on training
  set)
• High zoom factor, but often requires
  high-resolution image to start with
• Somewhat unpredictable
• Not appropriate for medical image processing

Both methods require precise image
alignment. Solution Cheat!
19
Further Issues

• Single-image interpolation
• Corners and curves (Shahs Elastica with hinges)
• Quantization (thresholding, piecewise constant
  Mumford-Shah segmentation)
• Shape priors (Chen-Tagare-etc.)
• Texture (Portilla-Simoncelli, Zhu-Mumford)
• Error analysis (Chen-Kang)
• Multiple-image super-resolution
• Automatic landmark detection (RANSAC)
• Incorporate registration into MS framework
• Video de-interlacing (suggested by David Martin)
• 3D super-resolution (suggested by Jackie Shen)

20
Thats All Folks!

• Big thanks to
• Jackie Shen for his gamma convergence code,
  intellectual and moral support, and guiding me
  through China.
• Song-Chun Zhu for inviting me here and paying for
  this trip.
• You the audience for listening to me.
• Send comments to wittman_at_math.umn.edu
• www.math.umn.edu/wittman