Computational Cameras: Convergence of Optics and Software

1
Computational CamerasConvergence of Optics and
Software

• Shree K. Nayar

Computer Science Columbia University
Support NSF, ONR, Packard Foundation
2
detector
lens
image
Traditional Camera
3
detector
detector
lens
new optics
compute
image
Traditional Camera
Computational Camera
4
Wide Angle Imaging
5
Whats the Mirrors Shape ?
(with Simon Baker, ICCV 98)
camera
scene
lens
z
r
viewpoint
6
(courtesy RemoteReality)
7
(No Transcript)
8
(No Transcript)
9
System with a Flexible Field of View
(with Sujit Kuthirummal, 2007)
10
System with a Flexible FOV
11
Flexible Field of View
12
Computing the 3D Mirror Shape
13
Estimating 3D Mirror Shapes for a Sequence
Captured Video
Estimated Mirror Shapes
14
Field of View
15
Resolution
16
Mapping to an Equi-Resolution Image
Captured Image IC
Vertical Stretching
Horizontal Stretching
Horizontally Stretched Image IH
Equi-Resolution Image
17
Mapping to a Rectangular Image
Rectangular Image
18
(No Transcript)
19
Street Monitoring
20
Street Monitoring
21
Panning Up
22
Birthday
23
(No Transcript)
24
Image Stitching
AutoStitch (Brown and Lowe 2003)
25
Seamless Mosaic
26
Image Stitching with Parallax
27
Mosaic
28
Hockney Style Collage
Place Furstenberg, by David Hockney, 1985
29
Scene Collage with Parallax
Find SIFT Features in Images (Lowe 2004)
Match Features using F Matrix and RANSAC
Find Image Layering with Least Fragmentation
(with Yoshi Nomura and Li Zhang 07)
30
Scene Collage
31
Photo Browsing with Scene Collages
PhotoWalker (Tanaka et al. 03) Photo Tourism
(Snavely et al. 06)
32
Nested Scene Collages
33
(No Transcript)
34
(No Transcript)
35
Camera Modules
(with Yoshi Nomura and Li Zhang 07)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Dynamic Scene Collage
40
(No Transcript)
41
Dynamic Scene Collage
42
(No Transcript)
43
(No Transcript)
44
detector
detector
lens
new optics
compute
image
Traditional Camera
Computational Camera
detector
new optics
compute
controller
Programmable Imaging
45
Camera with a Lens
Scene
Aperture
Lens
Image Detector
46
Lensless Camera with Volumetric Aperture
Scene
Image Detector
(with Assaf Zomet, CVPR 2006)
47
Single Aperture Layer
Scene
Single Layer Aperture
Image Detector
f
48
Multiple Aperture Layers
Scene
Multi-Layered Aperture
j1
j2 . .
Image Detector
Pixel Brightness
Scene
Transmittance Functions
49
Initial Implementation LCD Attenuator
Camera without Lens
LCD Aperture
LCD Controller
50
What Mappings are Possible?

• 1D camera
• Not all mappings M are feasible

image
mapping
scene
51
Spatially Varying Zoom
Scene
MX and MX are impossible with 3 layers
52
Fov 1
Fov 3
Attenuating Layers
Pinhole
Pinhole
Pinhole
Image Detector
53
(No Transcript)
54
detector
detector
lens
new optics
compute
image
Traditional Camera
Computational Camera
detector
projector
new optics
compute
compute
image
controller
camera
Programmable Imaging
Programmable Flash
55
(No Transcript)
56
Active Refocusing with a Single Image
Acquired Image
57
Acquired Image
(with Francesc Moreno and Peter Belhumeur 07)
58
Refocusing
Acquired Image
59
(No Transcript)
60
Fast Multispectral Imaging
(with J. Park, M. Lee, M. Grossberg)
61
Multiplexed Illumination
62
Image 1
Image 2
Image 2
time
Image 1
R
G
B
Measured Data
63
Ground Truth
Computed
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
detector
lens
image
Traditional Camera
68
(No Transcript)