Amit Agrawal and Ramesh Raskar

1
Resolving Objects at Higher Resolution from a
Single Motion-Blurred Image

• Amit Agrawal and Ramesh Raskar
• Mitsubishi Electric Research Labs (MERL)
• Cambridge, MA, USA

2
Can we recover more detail by letting objects
blur?
Text
Object Moves
Low Res Camera Blurred Image
Low Res Camera Static Object
Text 10 pixels wide
Blurred Text 30 pixels wide
10 pixels
30 pixels
Our approach Resolution enhanced
deblurring Resolution enhancement factor 3
Traditional Motion Deblurring Recovers 10 pixels
3
Applications
Aerial Imaging
Electronic Toll Booths
Goal Automatic license plate number recognition
4
Overview

• Exploiting motion blur for super-res
• Linear system
• No hallucination, image priors or training data
• Motion deblurring is ill-posed
• Practical Solution Flutter Shutter Camera
• Resolution enhanced deblurring
• Using Flutter Shutter Camera

5
Sensor point spread function (PSF)
20 pixel object
Ground Truth Static Scene (High Resolution Camera)
Decimation Matrix High Resolution -gt Low
Resolution
6
Exploiting motion blur for super-res
20 pixel object
10 pixels
Ground Truth Static Scene (High Resolution Camera)
Static Scene (Low Resolution Camera)
Traditional Motion Deblurring
7
10
?
20
Blurring Matrix A
Sharp Image x
Blurred Image b
Traditional Motion Deblurring
8
10
20
Traditional Motion Deblurring
Blurring Matrix A
Blurred Image Low Resolution
Sharp Image x
?
20
20
0 0 0 1 1 0 0 0
Decimation Matrix D
Sharp Image x
Blurring Matrix A High Resolution
Blurred Image Low Resolution
Resolution Enhanced Deblurring
9
Deblurring is ill-posed
K D A Combined deblurring resolution
enhancement matrix Typical Case Condition
number of K 1018
10
Overview

• Exploiting motion blur for super-res
• Linear system
• No hallucination, image priors or training data
• Motion deblurring is ill-posed
• Practical Solution Flutter Shutter Camera
• Raskar, Agrawal, Tumblin, SIGGRAPH 2006
• Resolution enhancement deblurring
• Using Flutter Shutter Camera

11
Blurring Convolution
Frequency Spectrum Magnitude Sinc Function
?
Traditional Camera Shutter is OPEN Box Filter
12
Coded Exposure
Traditional
Deblurred Image
Deblurred Image
Matlab Lucy
Image of Static Object
13
Implementation
LCD Shutter in front of lens becomes opaque and
transparent
14
License Plate Retrieval
15
Related Work

• Super-Resolution Motion Deblurring
• Sondhi, Proc. IEEE 1972
• Irani Peleg, CVGIP91, CVPR92
• Baker Kanade, PAMI02
• Ben-Ezra et al. PAMI 2005
• Bascle et al. ECCV96
• Tappen et al. SCTV03
• Shechtman et al. ECCV02
• Coding Modulation in Cameras/Projectors
• HDR, Nayar Mitsunaga, CVPR00
• Multiplexed Illumination, Schechner et al.
  ICCV03
• Flutter shutter camera, Raskar et al. SIGGRAPH06
• Coded aperture, Hiura Matsuyama, CVPR98
• Coded aperture with lens, light field modulation,
  SIGGRAPH07
• With Jack Tumblin, Ankit Mohan and Ashok
  Veeraraghavan (UMD)

16
Overview

• Exploiting motion blur for super-res
• Linear system
• No hallucination, image priors or training data
• Motion deblurring is ill-posed
• Practical Solution Flutter Shutter Camera
• Raskar et al. SIGGRAPH 2006
• Resolution enhancement deblurring
• Using Flutter Shutter Camera

17
Super-resolution with motion blur Jitter Camera
M. Ezra, A. Zomet S.K. Nayar,
Video Super-Resolution Using Controlled Subpixel
Detector Shifts, PAMI
2005
Z. Lin H.Y Shum,
Fundamental Limits of Reconstruction-Based
Super-resolution Algorithms under Local
Translation,
PAMI 2004
18
Experimental Setup
Resolution Chart is moved horizontally along the
tracks
19
Flutter Shutter Shutter is OPEN and CLOSED
20
Unblurred Width
Recovered at triple resolution
Recovered at normal resolution
Blurred Width
Deblurred Image Upsampled
Input Blurred Photo (cropped)
Deblurred Image
Super-Resolution Deblurring Result
21
Input Image Patch
Deblurred Patch
Our approach
22
Future Work

• Exploiting Motion Blur
• 2D Motion Blur Camera shake
• Super-resolution with RGB Bayer mosaic
• Coded exposure video with Pointgrey Dragonfly2
  camera
• Focus blur
• Coded Aperture
• Handling motion and focus blur simultaneously
• Fluttering Codes
• Binary vs Continuous
• Suitable for other applications

23
Acknowledgments

• Jack Tumblin, Northwestern University
• Hideaki Nii, University of Tokyo
• MERL
• Fatih Porikli, Jay Thornton
• Joseph Katz, Keisuke Kojima
• Haruhisa Okuda, MELCO

24
Super-res from single motion blurred photo

• Exploiting single photo motion blur
• Letting objects blur in 1D
• Solving simple linear system
• More blur more resolution but with increased
  noise
• Practical implementation
• Coded Exposure (Flutter Shutter) Camera
• Makes deblurring well-posed
• Results on 1D resolution enhancement
• Using Flutter Shutter Camera

25
Coded Exposure (Flutter Shutter) Camera
Motion Deblurring Resolution Enhancement
Motion Deblurring
Google Flutter Shutter
26
Extra Slides
27
Captured Blurred Photo
Blurred Image Patch
Traditional Deblurring
Upsampled by three times
Resolution Enhancement Deblurring
28
Are all codes good?
All ones
Alternate
Random
Our Code
29
Need to consider zero padded codes!!!
30
(No Transcript)
31
(No Transcript)
32
Input Image
33
Rectified Crop
Deblurred Result
34
Shutter
open
Exposure Time
closed
Continuous blur
Blurred image Traditional camera
Code 1101101010001001100011
35
Comparison
36
Inverse Filter stable
Inverse Filter Unstable
37
Traditional Motion Deblurring
Captured Blurred Image
Deblurred Image
Deblurring is at the same resolution
38
Traditional Camera Shutter is OPEN
39
Our Camera Flutter Shutter
40
Shutter is OPEN and CLOSED