Computational Illumination

1
Computational Illumination
Course WebPage http//www.merl.com/people/raska
r/photo/
Ramesh Raskar Mitsubishi Electric Research Labs
2
Computational Illumination
3
Computational Photography
Illumination
Novel Cameras
GeneralizedSensor
Generalized Optics
Processing
4D Light Field
4
Computational Illumination
Novel Illumination
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
Programmable 4D Illumination field time
wavelength
4D Light Field
5
Edgerton 1930s
Not Special Cameras but Special Lighting
6
Edgerton 1930s
Multi-flash Sequential Photography
Stroboscope (Electronic Flash)
Time
Flash
Shutter Open
7
Smarter Lighting Equipment
What Parameters Can We Change ?
8
Computational IlluminationProgrammable 4D
Illumination Field Time Wavelength

• Presence or Absence, Duration, Brightness
• Flash/No-flash
• Light position
• Relighting Programmable dome
• Shape enhancement Multi-flash for depth edges
• Light color/wavelength
• Spatial Modulation
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• Exploiting (uncontrolled) natural lighting
  condition
• Day/Night Fusion, Time Lapse, Glare

9
Computational Illumination

• Presence or Absence, Duration, Brightness
• Flash/No-flash
• Light position
• Multi-flash for depth edges
• Programmable dome (image re-lighting and matting)
• Light color/wavelength
• Spatial Modulation
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• General lighting condition
• Day/Night

10
Denoising Challenging Images

• Available light
• nice lighting
• noise/blurriness
• color

11

• Flash
• details
• color
• flat/artificial

Flash
12
Elmar Eisemann and Frédo Durand , Flash
Photography Enhancement via Intrinsic
RelightingGeorg Petschnigg, Maneesh Agrawala,
Hugues Hoppe, Richard Szeliski, Michael Cohen,
Kentaro Toyama. Digital Photography with Flash
and No-Flash Image Pairs

• Denoise no-flash image using flash image

13

• Transfer detail from flash image to no-flash
  image

original lighting details/sharpness color
14
Cross-Bilateral Filter based Approach
15
Flash
Ambient
16
Build Exposure HDR image

• Multiple images with different exposure
• Debevec Malik, Siggraph 97
• Nayar Mitsunaga, CVPR 00

Increasing Exposure
17
Build Flash HDR image
Flash Intensity
18
Build Flash-Exposure HDR image
Flash Intensity
Agrawal, Raskar, Nayar, LiSiggraph05
Exposure
19
Capturing HDR Image
Varying Exposure time
Varying Flash brightness
Varying both
20
Flash and Ambient Images Agrawal, Raskar,
Nayar, Li Siggraph05
Result
Reflection Layer
Flash
Ambient
21
Intensity Gradient Vectors in Flash and Ambient
Images
Same gradient vector direction
Flash Gradient Vector
Ambient Gradient Vector
Ambient
Flash
No reflections
22
Reflection Ambient Gradient Vector
Different gradient vector direction
Flash Gradient Vector
Ambient
Flash
With reflections
23
Reflection Ambient Gradient Vector
Intensity Gradient Vector Projection
Residual Gradient Vector
Flash Gradient Vector
Result Gradient Vector
Ambient
Flash
Result
Residual
24
Flash Matting
25
Computational Illumination

• Presence or Absence, Duration, Brightness
• Flash/No-flash
• Light position
• Relighting Programmable dome
• Shape-Detail enhancement Multi-flash for depth
  edges
• Spatial Modulation
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• General lighting condition
• Day/Night

26
Synthetic LightingPaul Haeberli, Jan 1992
27
Table-top Computed Lighting for Practical Digital
Photography

• Ankit Mohan, Jack Tumblin
• Northwestern University

Bobby Bodenheimer Vanderbilt University
Cindy Grimm, Reynold Bailey Washington University
in St. Louis
28
(No Transcript)
29
Sketch Your Desires, Optimize
Target
Result
30
Debevec et al. 2002 Light Stage 3
31
Image-Based Actual Re-lighting
Debevec et al., SIGG2001
Light the actress in Los Angeles
Film the background in Milan, Measure incoming
light,
Matched LA and Milan lighting.
Matte the background
32
Computational Illumination
Novel Illumination
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
Programmable 4D Illumination field time
wavelength
4D Light Field
33
Quest for 8D Capture ..
34
Non-photorealistic Camera Depth Edge Detection
and Stylized Rendering using Multi-Flash Imaging

• Ramesh Raskar, Karhan Tan, Rogerio Feris, Jingyi
  Yu, Matthew Turk
• Mitsubishi Electric Research Labs (MERL),
  Cambridge, MA
• U of California at Santa Barbara
• U of North Carolina at Chapel Hill

35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
Depth Discontinuities
Internal and externalShape boundaries, Occluding
contour, Silhouettes
40
Our Method
Canny
41
(No Transcript)
42
Result
Photo
Canny Intensity Edge Detection
Our Method
43
A New Problem
Shadows Clutter Many Colors
Highlight Shape Edges Mark moving parts Basic
colors
44
Multi-light Image Collection

• Fattal, Agrawala, Rusinkiewicz Sig2007

Input Photos
ShadowFree,Enhanced surface detail, but Flat
look
Some Shadows for depth but Lost visibility
45
Multiscale decomposition using Bilateral
Filter, Combine detail at each scale across all
the input images.
Fuse maximum gradient from each photo,
Reconstruct from 2D integration all the input
images. Enhanced shadows
46
Computational Illumination

• Presence or Absence
• Flash/No-flash
• Light position
• Multi-flash for depth edges
• Programmable dome (image re-lighting and matting)
• Spatial Modulation (Intra-flash 2D Modulation)
• Camera flash Projector
• Synthetic Aperture Illumination
• Dual Photography
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• General lighting condition
• Day/Night

47
Synthetic aperture imaging illumination
Levoy et al 2004
48
What does synthetic aperture illumination look
like?
49
Underwater confocal imagingwith and without SAP
50
Dual Photography
Projector
Photocell
Scene
51
Dual Photography
Projector
Photocell
Scene
52
Dual Photography
Projector
Photocell
Scene
53
Dual Photography
Projector
Photocell
Camera
Scene
54
The 4D transport matrix Contribution of each
projector pixel to each camera pixel
projector
camera
scene
55
The 4D transport matrix Contribution of each
projector pixel to each camera pixel
projector
camera
scene
Sen et al, Siggraph 2005
56
The 4D transport matrix Which projector pixel
contribute to each camera pixel
projector
camera
?
scene
Sen et al, Siggraph 2005
57
Dual photographyfrom diffuse reflections
the cameras view
Sen et al, Siggraph 2005
58
Direct Global Sep
59
Computational Illumination

• Presence or Absence
• Flash/No-flash
• Light position
• Multi-flash for depth edges
• Programmable dome (image re-lighting and matting)
• Light color/wavelength
• Spatial Modulation
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• General lighting condition
• Day/Night

60
Computational Illumination

• Presence or Absence
• Flash/No-flash
• Light position
• Multi-flash for depth edges
• Programmable dome (image re-lighting and matting)
• Light color/wavelength
• Spatial Modulation
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• Natural lighting condition
• Day/Night Fusion, Time Lapse Imaging, Glare
  removal

61
A Night Time Scene Objects are Difficult to
Understand due to Lack of Context
Dark Bldgs
Reflections on bldgs
Unknown shapes
62
Enhanced Context All features from night scene
are preserved, but background in clear
Well-lit Bldgs
Reflections in bldgs windows
Tree, Street shapes
63
Night Image
Background is captured from day-time scene using
the same fixed camera
Result Enhanced Image
Day Image
64
But, Simple Pixel Blending Creates Ugly
Artifacts
65
Pixel Blending
Our MethodIntegration of blended Gradients
66
Day of the year
67
The Archive of Many Outdoor Scenes (AMOS) Images
from 1000 static webcams, every 30 minutes
since March 2006.
Variations over a year and over a day
Jacobs, Roman, and Robert Pless, WUSTL CVPR 2007,
68

• Analysing Time Lapse Images
• PCA
• Linear Variations due to lighting and seasonal
  variation
• Decompose (by time scale)
• Hour haze and cloud for depth.
• Day changing lighting directions for surface
  orientation
• Year effects of changing seasons highlight
  vegetation
• Applications
• Scene segmentation.
• Global Webcam localization. Correlate timelapse
  video over a month from unknown camera with
• sunrise sunset (localization accuracy 50
  miles)
• Known nearby cameras (25 miles)
• Satellite image (15 miles)

Mean image 3 components from time lapse of
downtown st. louis over the course of 2 hours
69
2 Hour time Lapse in St Louis Depth from
co-varying regions
70
Surface Orientation False Color PCA images
71
Factored Time Lapse Video
Sunkavalli, Matusik, Pfister, Rusinkiewicz,
Sig07
Factor into shadow, illumination, and
reflectance. Relight, recover surface normals,
reflectance editing.
72
(No Transcript)
73
Suppressing Unwanted Light Glare Talvala et
al, Siggraph 2007
74
Ground Truth
All detail lost in dark regions
75
Modulating light entering lens

• Fraction a of the mask is clear
• Glare G reduced roughly to aG
• 1/a photos needed

1. Scene
2. Occlusion mask
3. Camera

ACM SIGGRAPH 07
75
Veiling Glare in HDR Imaging
76
Estimating Glare Contribution

• Inside hole imageglare (low-frequency
  assumption)

ACM SIGGRAPH 07
76
Veiling Glare in HDR Imaging
77

• Backlit still life, using Canon 20D, occluder mask

HDR capture, 5601
Our result, 224001
ACM SIGGRAPH 07
77
Veiling Glare in HDR Imaging
78
HDR capture, 5601
Our result, 224001
ACM SIGGRAPH 07
78
Veiling Glare in HDR Imaging
79
Smarter Lighting Equipment
Programmable Parameters
80
Computational Illumination
Light Sources
Modulators
Novel Cameras
Generalized Optics
GeneralizedSensor
Generalized Optics
Processing
Programmable 4D Illumination field Time
Wavelength
4D Ray Bender
Ray Reconstruction
Upto 4D Ray Sampler
4D Light Field
Display
Scene 8D Ray Modulator
Recreate 4D Lightfield
81
Computational IlluminationProgrammable 4D
Illumination Field Time Wavelength

• Presence or Absence, Duration, Brightness
• Flash/No-flash
• Light position
• Relighting Programmable dome
• Space-detail Enhancement Multi-flash for depth
  edges
• Spatial Modulation
• Flash as Projector
• Synthetic Aperture Illumination
• Temporal Modulation
• TV remote, Motion Tracking, Sony ID-cam, RFIG
• Exploiting (uncontrolled) natural lighting
  condition
• Day/Night Fusion, Time Lapse, Glare

82
830 Introduction (Raskar, 10
minutes) 840 Concepts in Computational
Photography (Tumblin, 30 minutes) 910
Image Processing Tools (Raskar, 30
mins) 940 Understanding Film-like Photography
(Tumblin, 30 minutes) Break 1030
Computational Camera (Nayar, 40
minutes) 1050 Advances in Optics
(Nayar, 20 minutes) 1130 Improving Film-like
Photography (Tumblin, 40 minutes)
Lunch Break 145 Multi-perspective
Photography (Davidhazy, 35 minutes) 220
Lightfield photography and microscopy
(Levoy, 30 minutes) 250 Fourier Analysis of
Light Fields (Georgiev, 35 minutes)
Break 345 Computational Illumination
(Raskar, 40 minutes) 425 Computational
Imaging in the Sciences (Levoy, 30
minutes) 455 Future Cameras (Raskar,
20 minutes) 515 Summary and Discussion
(15 minutes)
Course Page http//www.merl.com/people/raskar/ph
oto/