Multiframe Resolution Enhancement

1
Multi-frame Resolution Enhancement

• by
• Bahadir K. Gunturk

2
Outline

• Idea of Superresolution

• Application areas

• Different Superresolution techniques

• Future of Superresolution

3
Imaging System
Resolution is reduced during imaging process !

• Nonzero physical dimensions of each sensor
  element
• Nonzero aperture time
• Optical blurring (out-of-focus, ...)
• Motion, sensor noise, ...

High Resolution
Low Resolution
4
Superresolution

• Inverse Problem
• Find HR given LR 1, LR 2, LR 3, ...
• Model the imaging system
• Estimate h1, h2, h3, ...
• LR 1, LR 2, LR 3, ... are not identical

HR
h1
h2
h3
LR 1
LR 3
LR 2
5
Applications

• Aerial/Satellite Imaging

• Medical Imaging

• High-Definition TV (HDTV) Displays

• Digital Camera

• Scanner resolution enhancement

• Extracting/Printing still image from video
  sequence

• Security/Surveillance systems, forensic science,
  ...

6
An Example

• LR images sent from Mars Pathfinder

• Superresolution applied

7
Superresolution Methods

• Bayesian Methods
• probabilistic
• model the error in h1, h2, h3, ... estimation as
  a random variable
• find the most likely HR that will produce LR 1,
  LR 2, LR 3,
• Maximum Likelihood, Maximum A Posteriori
  Probability

HR
h1
h2
h3
NOISE
LR 1
LR 3
LR 2
8
Superresolution Methods

• Iterative Methods
• deterministic
• assumes h1, h2, h3, ... are estimated accurately
• compute LR 1, LR 2, LR 3, from an initial HR
  estimate
• back-project the error between the computed and
  the observed LR images

HR
h1
h2
h3
Computed from HR estimate
Observed LR
LR 1
LR 3
LR 2
9
What if there is compression?

• MPEG compression
• DCT of LR 1, LR 2, LR 3, are taken
• DCT coefficients are quantized

HR
h1
h2
h3
LR 1
LR 3
LR 2
DCT
Quantize
Transmit
D 1
D 3
D 2
10
What if there is compression?

• MPEG compression
• DCT of LR 1, LR 2, LR 3, are taken
• DCT coefficients are quantized

HR
An example DCT coefficient d1 52 Quantize
round(52/10)5 Send 5 Receive 51050 Bounds
45ltd1lt55
h1
h2
h3
LR 1
LR 3
LR 2
DCT
. d1
D 1
D 3
D 2
11
What if there is compression?

• MPEG compression
• DCT of LR 1, LR 2, LR 3, are taken
• DCT coefficients are quantized

HR
An example DCT coefficient d1 52 Quantize
round(52/10)5 Send 5 Receive 51050 Bounds
45ltd1lt55
h1
h2
h3
LR 1
LR 3
LR 2
DCT
D 1
D 3
D 2
12
What if there is compression?

• DCT-domain Superresolution
• compute H1, H2, H3, from h1, h2, h3,
• compute D 1, D 2, D 3, from an initial HR
  estimate
• back-project the error between the computed and
  the observed DCT coefficients

HR
H1
h1
h2
h3
error if computed DCT coefficient is larger than
the upper bound or smaller than the lower bound
LR 1
LR 3
LR 2
DCT
D 1
D 3
D 2
13
Review

• Motivation for Superresolution

• 2 Main Superresolution Approaches
• Bayesian Methods
• Iterative Methods

• Superresolution if there is compression
• - http//users.ece.gatech.edu/yucel/DCTSRResults
  .html

14
Future of superresolution?

• Do not expect miracles from Superresolution
• Loss of information Total information obtained
  from multiple LR images may not be enough to
  fully recover the HR image

• Have we reached saturation in Superresolution?
• It is still an active research area to find fast
  and robust methods
• Hardware implementation
• 3-D Superresolution
• - http//ic-www.arc.nasa.gov/ic/projects/bayes-gr
  oup/super-res/

15
The End
Bahadir K. Gunturk gte714q_at_prism.gatech.edu user
s.ece.gatech.edu/gte714q Dr. Yucel Altunbasak
yucel_at_ece.gatech.edu users.ece.gatech.edu/yucel