Highperformance imaging using dense arrays of cameras

1
Using Plane Parallax to Calibrate Dense Camera
Arrays
Vaibhav Vaish, Bennett Wilburn, Neel Joshi, Marc
Levoy
Computer Science Department Stanford University
2
The Stanford Multi-Camera Array
3
Example Seeing through Foliage
4
Example Synthetic Aperture Photography
5
Outline

• Problem Statement
• Synthetic aperture photography using an array of
  cameras
• Calibration required
• Calibration Pipeline
• Results
• Future Work

6
Synthetic aperture photography
7
SAP Prior Work

• Synthetic Aperture Radar
• Light Field Rendering Levoy 96
• Dynamically Reparametrized Light Fields Isaksen
  00
• Single lens SAP Favaro 03

8
Outdoor SAP Array layout

• width of aperture 2m
• number of cameras 45
• spacing between cameras 13cm
• cameras field of view 4.5

9
Outdoor SAP The scene

• distance to occluder 33m
• distance to targets 45m
• field of view at target 3m

10
Outdoor SAP Calibration
Calibration Volume
Focal Depth

• Narrow field of view and long-range imaging makes
  accurate pose estimation difficult
• Cannot take calibration measurements at the
  desired focal depth (behind occluding bushes)

11
Calibration Goal
Focal Plane

• Given a focal plane, compute the projective
  transform (homography) to project each camera
  image onto the plane.

12
Approaches to Calibration

• Metric Calibration
• Computes camera intrinsics, position,
  orientation(10 parameters/camera)
• Nonlinear optimization, requires initial guess
• Not stable for narrow angle lenses and long range
  imaging
• Non-metric Calibration
• Plane Parallax methods Irani 96, Triggs 00
• Homography Spaces Zelnik-Manor 99

13
Calibration Pipeline

• Problem Statement
• Calibration Pipeline
• Focus cameras on one plane (using homographies)
• Compute relative camera positions from parallax
  measurements
• Use camera positions to vary focal plane over a
  range of depths
• Results
• Future Work

14
Focusing on one plane
Add camera images so that points on one plane are
in good focus

15
Focusing at different depths

16
Focusing at different depths
To focus at a different depth, we have to shift
the images by an amount equal to the parallax

17
Parallax and Camera Geometry
Reference Plane
Camera Plane

• ?p1 ?X1 . ?z/(Z0 ?z) ?X1 . dP
• Parallax Camera shift Relative Depth

18
Parallax and Camera Geometry
P
Reference Plane
?p1
?p2
Camera Plane
?X1
?X2

• Measure parallax of P in all cameras (wrt
  reference camera)
• ?p1 ?p2 . . .

19
Recovering Camera Positions

• Parallax of point P

For multiple points P1, Pn
Relative camera positions ?Xi can be recovered
robustly (up to scale) using SVD.
20
Computing SAP Images at different focal depths

• To change the focal depth, images have to be
  shifted by the amount of parallax.
• In camera C1 , the parallax for a parallel focal
  plane is f . ?X1 ,where f is a constant that
  depends only on the depth of the plane.
• f is analogous to the focus distance of the
  synthetic lens varying f changes the depth of
  the focal plane.

?X1
21
Algorithm for SAP

• Focus cameras onto a plane
• Compute parallax for one (or more) scene points
• Recover relative camera positions ?Xi (up to an
  unknown scale)
• For a range of values of f
• Shift image from camera Ci by f . ?Xi and
  average shifted images.
• Varying f corresponds to changing the focus

22
Results
Synthetic Aperture Sequence
23
Parallax v/s Metric Calibration
Parallax-based calibration
Metric calibration
24
Summary

• Calibration of camera arrays for synthetic
  aperture photography
• Decompose warps into reference homography and
  shifts
• Use parallax measurements to compute camera
  positions
• Avoids computing camera intrinsics and
  orientation explicitly
• Robust, linear solution
• Metric information not available
• Algorithm requires planar camera array and
  frontoparallel reference plane

25
Extension Tilted Focal Planes
Reference Plane

• Parallax is described by a projective warp (not
  a shift)
• Rank-1 factorization is still possible

26
Future Work

• Real-time applications
• Warp images in hardware
• Track moving objects by moving focal plane
• 3D Reconstruction from synthetic focus
• Is this more robust to occlusions ?
• Quantitative analysis of synthetic aperture
  photography
• Effect of occluder density, number of cameras,
  aperture shape

27
Acknowledgements

• Sponsors
• NSF IIS-0219856-001
• DARPA NBCH 1030009
• Assistance in acquisition
• Gaurav Garg, Augusto Roman, Billy Chen, Pradeep
  Sen, Doantam Phan, Guillaume Poncin, Jeff
  Klingner
• High Speed Videography Using a Dense Camera Array
• B Wilburn, N Joshi, V Vaish, M Levoy, M Horowitz
• Session 5A, 420pm

28
Seeing through Foliage
29
SAP with tilted focal planes
Frontoparallel focal planes
Rotating focal plane