Computational Photography

1
Computational Photography
Ramesh Raskar Mitsubishi Electric Research
Labs, Cambridge MA
http//www.merl.com/people/raskar/photo
2
Traditional film-like Photography
Detector
Lens
Pixels
Image
3
Computational Photography
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
4
Computational Photography
Programmable Lighting
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
Scene
5
Computational Photography
Programmable Lighting
Light Sources
Novel Cameras
a. Multi-flash Illum
GeneralizedSensor
Generalized Optics
Processing
c. Image Fusion
b. SAMPd. Camera Array
e. Gradient Sensing f. Flutter Shutter
Scene
6
Computational Photography
Programmable Lighting
Light Sources
Novel Cameras
a. Multi-flash Illum
GeneralizedSensor
Generalized Optics
Processing
c. Image Fusion
b. SAMPd. Camera Array
e. Gradient Sensing f. Flutter Shutter
Scene
7
Depth Edge Camera
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Depth Discontinuities
Internal and externalShape boundaries, Occluding
contour, Silhouettes
14
Result
Photo
Canny Intensity Edge Detection
Our Method
15
(No Transcript)
16
Shadows Clutter Many Colors
Highlight Shape Edges Mark moving parts Basic
colors
17
Computational Photography
Programmable Lighting
Light Sources
Novel Cameras
a. Multi-flash Illum
GeneralizedSensor
Generalized Optics
Processing
c. Image Fusion
b. SAMPd. Camera Array
e. Gradient Sensing f. Flutter Shutter
Scene
18
SAMP Camera(Single Axis Multiple Parameters)
Parameters vary in focus, exposure, and aperture.
19
Multiparameter Camera
20
A Night Time Scene Objects are Difficult to
Understand due to Lack of Context
Dark Bldgs
Reflections on bldgs
Unknown shapes
21
Enhanced Context All features from night scene
are preserved, but background in clear
Well-lit Bldgs
Reflections in bldgs windows
Tree, Street shapes
22
Night Image
Background is captured from day-time scene using
the same fixed camera
Result Enhanced Image
Day Image
23
But, Simple Pixel Blending Creates Ugly
Artifacts
24
Traffic Context Enhancement
Frame N
Gradient field
Mixed gradient field
TimeAveraged importance mask
Processed binary mask
Final result
Gradient field
Daytime image
Frame N-1
25
Camera Arrays Foreground Segmentation
26
Extracting Foreground (Without Blue Screen)
I
?
?F
New Background
Composite
27
Computational Photography
Programmable Lighting
Light Sources
Novel Cameras
a. Multi-flash Illum
GeneralizedSensor
Generalized Optics
Processing
c. Image Fusion
b. SAMPd. Camera Array
e. Gradient Sensing f. Flutter Shutter
Scene
28
Gradient CameraSensing Difference between
Neighboring Pixels
Short Exposure
Goal High Dynamic Range
Long Exposure
29
High Dynamic Range Images Sensing Pixel
Difference with Locally Adaptive Gain
Scene
Intensity camera saturation map
Gradient camera saturation map
Intensity camera fails to capture rangeGradients
saturate at very few isolated pixels
30
Figure 2 results
Problem Motion Deblurring
Input Image
31
Blurred Taxi
Image Deblurred by solving a linear system. No
post-processing
32
Application Aerial Imaging
Sharpness versus Image Pixel Brightness
Long Exposure The moving camera creates smear
Shutter Open
Time
Shutter Closed
Short ExplosureAvoids blur. But the image is
dark
Shutter Open
Time
Shutter Closed
Time
Solution Flutter Shutter
Shutter Open
Goal Capture sharp image with sufficient
brightness using a camera on a fast moving
aircraft
Time
Shutter Closed
33
Fluttered Shutter Camera
Raskar, Agrawal, Tumblin Siggraph2006
Ferroelectric shutter in front of the lens is
turnedopaque or transparent in a rapid binary
sequence
34
Coded Exposure Photography Assisting Motion
Deblurring using Fluttered Shutter
Raskar, Agrawal, Tumblin (Siggraph2006)
Short Exposure
Traditional
MURA
Coded
Shutter
Captured Photos
Deblurred Results
Result has Banding Artifacts and some spatial
frequencies are lost
Decoded image is as good as image of a static
scene
Image is dark and noisy
35
Sample Projects

• Illumination
• Multi-flash Imaging Comprehensible Cartoons
• Optics and Camera Array
• SAMP Camera Background Segmentation
• Image Fusion (Day-Night, Visible-IR)
• Camera Array
• Smart Sensors
• Gradient Camera Dynamic Range
• Flutter Shutter Motion Deblurring

36
Computational Photography
Ramesh Raskar, MERL
Novel Cameras
GeneralizedSensor
Programmable Lighting
Generalized Optics
Light Sources
Processing
a. Multi-flash Illum
c. Image Fusion
b. SAMPd. Camera Array
e. Gradient Sensing f. Flutter Shutter