Fourier Slice Photography

1
Fourier Slice Photography
Ren Ng Stanford University
2
Conventional Photograph
3
Light Field Photography

• Capture the light field inside the camera body

4
Hand-Held Light Field Camera
Medium format digital camera
Camera in-use
16 megapixel sensor
Microlens array
5
Light Field in a Single Exposure
6
Light Field in a Single Exposure
7
Light Field Inside the Camera Body
8
Digital Refocusing
9
Digital Refocusing
10
Questions About Digital Refocusing

• What is the computational complexity?Are there
  efficient algorithms?
• What are the limits on refocusing?How far can we
  move the focal plane?

11
Overview

• Fourier Slice Photography Theorem
• Fourier Refocusing Algorithm
• Theoretical Limits of Refocusing

12
Previous Work

• Integral photography
• Lippmann 1908, Ives 1930
• Lots of variants, especially in 3D TVOkoshi
  1976, Javidi Okano 2002
• Closest variant is plenoptic cameraAdelson
  Wang 1992
• Fourier analysis of light fields
• Chai et al. 2000
• Refocusing from light fields
• Isaksen et al. 2000, Stewart et al. 2003

13
Fourier Slice Photography Theorem

• In the Fourier domain, a photograph is a 2D
  slice in the 4D light field. Photographs
  focused at different depths correspond to 2D
  slices at different trajectories.

14
Digital Refocusing by Ray-Tracing
u
x
Sensor
Lens
15
Digital Refocusing by Ray-Tracing
u
x
Imaginary film
Sensor
Lens
16
Digital Refocusing by Ray-Tracing
u
x
Imaginary film
Sensor
Lens
17
Digital Refocusing by Ray-Tracing
u
x
Imaginary film
Sensor
Lens
18
Digital Refocusing by Ray-Tracing
u
x
Imaginary film
Sensor
Lens
19
Refocusing as Integral Projection
u
x
Imaginary film
Sensor
Lens
20
Refocusing as Integral Projection
u
x
Imaginary film
Sensor
Lens
21
Refocusing as Integral Projection
u
x
Imaginary film
Sensor
Lens
22
Refocusing as Integral Projection
u
x
Imaginary film
Sensor
Lens
23
Classical Fourier Slice Theorem
2D FourierTransform
1D FourierTransform
24
Classical Fourier Slice Theorem
2D FourierTransform
1D FourierTransform
25
Classical Fourier Slice Theorem
2D FourierTransform
1D FourierTransform
26
Classical Fourier Slice Theorem
Spatial Domain
Fourier Domain
27
Classical Fourier Slice Theorem
Spatial Domain
Fourier Domain
28
Fourier Slice Photography Theorem
IntegralProjection
Spatial Domain
Fourier Domain
Slicing
29
Fourier Slice Photography Theorem
IntegralProjection
4D FourierTransform
Slicing
30
Fourier Slice Photography Theorem
IntegralProjection
4D FourierTransform
2D FourierTransform
Slicing
31
Fourier Slice Photography Theorem
IntegralProjection
4D FourierTransform
2D FourierTransform
Slicing
32
Fourier Slice Photography Theorem
IntegralProjection
4D FourierTransform
2D FourierTransform
Slicing
33
Photographic Imaging Equations

• Spatial-Domain Integral Projection

Fourier-Domain Slicing
34
Photographic Imaging Equations

• Spatial-Domain Integral Projection

Fourier-Domain Slicing
35
Photographic Imaging Equations

• Spatial-Domain Integral Projection

Fourier-Domain Slicing
36
Theorem Limitations

• Film parallel to lens
• Everyday camera, not view camera
• Aperture fully open
• Closing aperture requires spatial mask

37
Overview

• Fourier Slice Photography Theorem
• Fourier Refocusing Algorithm
• Theoretical Limits of Refocusing

38
Existing Refocusing Algorithms

• Existing refocusing algorithms are expensive
• O(N4) where light field has N samples in each
  dimension
• All are variants on integral projection
• Isaksen et al. 2000
• Vaish et al. 2004
• Levoy et al. 2004
• Ng et al. 2005

39
Refocusing in Spatial Domain
IntegralProjection
4D FourierTransform
2D FourierTransform
Slicing
40
Refocusing in Fourier Domain
IntegralProjection
Inverse2D FourierTransform
4D FourierTransform
Slicing
41
Refocusing in Fourier Domain
IntegralProjection
Inverse2D FourierTransform
4D FourierTransform
Slicing
42
Asymptotic Performance

• Fourier-domain slicing algorithm
• Pre-process O(N4 log N)
• Refocusing O(N2 log N)
• Spatial-domain integration algorithm
• Refocusing O(N4)

43
Resampling Filter Choice
Kaiser-Bessel filter(width 2.5)
Gold standard (spatial integration)

• Triangle filter (quadrilinear)

44
Overview

• Fourier Slice Photography Theorem
• Fourier Refocusing Algorithm
• Theoretical Limits of Refocusing

45
Problem Statement

• Assume a light field camera with
• An f /A lens
• N x N pixels under each microlens
• If we compute refocused photographs from these
  light fields, over what range can we move the
  focal plane?
• Analytical assumption
• Assume band-limited light fields

46
Band-Limited Analysis
47
Band-Limited Analysis
Band-width of measured light field
Light field shotwith camera
48
Band-Limited Analysis
49
Band-Limited Analysis
50
Band-Limited Analysis
51
Photographic Imaging Equations

• Spatial-Domain Integral Projection

Fourier-Domain Slicing
52
Results of Band-Limited Analysis

• Assume a light field camera with
• An f /A lens
• N x N pixels under each microlens
• From its light fields we can
• Refocus exactly within depth of field of an f
  /(A N) lens
• In our prototype camera
• Lens is f /4
• 12 x 12 pixels under each microlens
• Theoretically refocus within depth of field
  of an f/48 lens

53
Light Field Photo Gallery
54
Stanford Quad
55
Rodins Burghers of Calais
56
Palace of Fine Arts, San Francisco
57
Palace of Fine Arts, San Francisco
58
Waiting to Race
59
Start of the Race
60
Summary of Main Contributions

• Formal theorem about relationship between light
  fields and photographs
• Computational application gives asymptotically
  fast refocusing algorithm
• Theoretical application gives analytic solution
  for limits of refocusing

61
Future Work

• Apply general signal-processing techniques
• Cross-fertilization with medical imaging

62
Thanks and Acknowledgments

• Collaborators on camera tech report
• Marc Levoy, Mathieu Brédif, Gene Duval, Mark
  Horowitz and Pat Hanrahan
• Readers and listeners
• Ravi Ramamoorthi, Brian Curless, Kayvon
  Fatahalian, Dwight Nishimura, Brad Osgood,
  Mike Cammarano, Vaibhav Vaish, Billy Chen,
  Gaurav Garg, Jeff Klingner
• Anonymous SIGGRAPH reviewers
• Funding sources
• NSF, Microsoft Research Fellowship, Stanford
  Birdseed Grant

63
Questions?
Start of the race, Stanford University Avery
Pool, July 2005