Course 3: Computational Photography

1
Course 3 Computational Photography
Ramesh Raskar Mitsubishi Electric Research
Labs Jack Tumblin Northwestern University
Course WebPage http//www.merl.com/people/raska
r/photo
2
Course 3 Computational Photography
Course WebPage http//www.merl.com/people/raskar
/photo
Course Evaluation http//www.siggraph.org/course
s_evaluation
3
Welcome

• Understanding Film-like Photography
• Parameters, Nonlinearities, Ray-based concepts
• Image Processing and Reconstruction Tools
• Multi-image Fusion, Gradient domain, Graph Cuts
• Improving Camera Performance
• Better dynamic range, focus, frame rate,
  resolution
• Future Directions
• HDR cameras, Gradient sensing, Smart
  optics/lighting

4
Goals

• Review of 30 recent papers
• Understand computational aspects of cameras
• Discuss issues in traditional cameras
• Explore alternative imaging methods
• Learn vision and optics techniques
• Discuss image processing and reconstruction tools
• What we will not cover
• Film Cameras, Novel view rendering (IBR), Color
  issues, Traditional image processing/editing

5
Speaker Jack Tumblin
Jack Tumblin is an Assistant Professor of
Computer Science at Northwestern University.
His interests include novel photographic
sensors to assist museum curators in historical
preservation, computer graphics and visual
appearance, and image-based modeling and
rendering. During his doctoral studies at Georgia
Tech and post-doc at Cornell, he investigated
tone-mapping methods to depict high-contrast
scenes. His MS in Electrical Engineering
(December 1990) and BSEE (1978), also from
Georgia Tech, bracketed his work as co-founder of
IVEX Corp., (gt45 people as of 1990) where his
flight simulator design work was granted 5 US
Patents. He is an Associate Editor of ACM
Transactions on Graphics, was a member of the
SIGGRAPH Papers Committee (2003, 2004), and in
2001 was a Guest Editor of IEEE Computer Graphics
and Applications. http//www.cs.northwestern.edu/
jet
6
Speaker Ramesh Raskar
Ramesh Raskar is a Senior Research Scientist at
MERL. His research interests include
projector-based graphics, computational
photography and non-photorealistic rendering. He
has published several articles on imaging and
photography including multi-flash photography for
depth edge detection, image fusion,
gradient-domain imaging and projector-camera
systems. His papers have appeared in SIGGRAPH,
EuroGraphics, IEEE Visualization, CVPR and many
other graphics and vision conferences. He was a
course organizer at Siggraph 2002, 2003 and 2004.
He is a panel organizer at the Symposium on
Computational Photography and Video in Cambridge,
MA in May 2005. He is a member of the ACM and
IEEE. http//www.merl.com/people/raskar/raskar.ht
ml
7
Opportunities

• Unlocking Photography
• How to expand camera capabilities
• Digital photography that goes beyond film-like
  photography
• Opportunities
• Computing corrects for lens, sensor and lighting
  limitations
• Computing merges results from multiple images
• Computing reconstructs from coded image samples
• Cameras benefit from computerized light sources
• Think beyond post-capture image processing
• Computation well before image processing and
  editing
• Learn how to build your own camera-toys

8
Traditional Photography
Detector
Lens
Pixels
Image
9
Computational Photography Optics, Sensors and
Computations
GeneralizedSensor
Generalized Optics
Computations
Ray Reconstruction
4D Ray Bender
Upto 4D Ray Sampler
10
Computational Photography
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
11
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
12
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
Scene 8D Ray Modulator
13
Computational Photography
Novel Illumination
Light Sources
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
Display
Scene 8D Ray Modulator
Recreate 4D Lightfield
14
Computational Photography
Novel Illumination
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
4D Incident Lighting
4D Ray Bender
Ray Reconstruction
Upto 4D Ray Sampler
4D Light Field
Display
Scene 8D Ray Modulator
Recreate 4D Lightfield
15
A Teaser Dual Photography
Projector
Photocell
Scene
16
A Teaser Dual Photography
Projector
Photocell
Scene
17
A Teaser Dual Photography
Projector
Photocell
Scene
18
A Teaser Dual Photography
Projector
Photocell
Camera
Scene
19
The 4D transport matrix Contribution of each
projector pixel to each camera pixel
projector
camera
scene
20
The 4D transport matrix Contribution of each
projector pixel to each camera pixel
projector
camera
scene
Sen et al, Siggraph 2005
21
The 4D transport matrix Which projector pixel
contribute to each camera pixel
projector
camera
?
scene
Sen et al, Siggraph 2005
22
Dual photographyfrom diffuse reflections
the cameras view
Sen et al, Siggraph 2005
23
A Brief History of Images
1558
Camera Obscura, Gemma Frisius, 1558
24
A Brief History of Images
1558
1568
Lens Based Camera Obscura, 1568
25
A Brief History of Images
1558
1568
1837
Still Life, Louis Jaques Mande Daguerre, 1837
26
A Brief History of Images
1558
1568
1837
Silicon Image Detector, 1970
1970
27
A Brief History of Images
1558
1568
1837
1970
Digital Cameras
1994
28
Dream of A New Photography

• Old New
• People and Time Cheap Precious
• Each photo Precious Free
• Lighting Critical Automated
• External Sensors No Yes
• Stills / Video Disjoint Merged
• Exposure Settings Pre-select Post-Process
• Exposure Time Pre-select Post-Process
• Resolution/noise Pre-select Post-Process
• HDR range Pre-select Post-Process

29
Survey

• How many of you are photographers ?
• How many of you are photo-artists ?
• How many of you are involved in camera
  design/development ?
• How many do active programming ?
• Field of work Academics? Industry ? Research ?
  Art ?

30
Schedule
830 Introduction (Raskar) 840
Photographic Signal Film-like Photography
(Tumblin) 910 Image Processing Tools
(Raskar) 940 Improving Film-like
Photography (Tumblin) 1015 Break 1030 Image
Reconstruction Techniques (Raskar) 1115 Smart
Lights and Beyond Photography (Tumblin) 1145
Smart Optics and Sensors (Raskar) 1205
Discussion
Course Page http//www.merl.com/people/raskar/ph
oto