High-performance imaging using dense arrays of cameras

1
Synthetic aperture confocal imaging
Marc Levoy Billy Chen Vaibhav Vaish
Mark Horowitz Ian McDowall Mark Bolas
2
Outline

• technologies
• large camera arrays
• large projector arrays
• cameraprojector arrays

• optical effects
• synthetic aperture photography
• synthetic aperture illumination
• synthetic confocal imaging

3
Stanford Multi-Camera ArrayWilburn 2002

• 640 480 pixels 30fps 128 cameras
• synchronized timing
• continuous video streaming
• flexible physical arrangement

4
Ways to use large camera arrays

• widely spaced light field capture
• tightly packed high-performance imaging
• intermediate spacing synthetic aperture
  photography

5
Synthetic aperture photography
6
Synthetic aperture photography
7
Synthetic aperture photography
8
Synthetic aperture photography
9
Synthetic aperture photography
10
Synthetic aperture photography
11
Related work

• not like synthetic aperture radar (SAR)
• more like X-ray tomosynthesis
• Levoy and Hanrahan, 1996
• Isaksen, McMillan, Gortler, 2000

12
Example using 45 cameras
13
Synthetic pull-focus
14
Crowd scene
15
Crowd scene
16
Synthetic aperture photographyusing an array of
mirrors
?

• 11-megapixel camera
• 22 planar mirrors

17

18

19
Synthetic aperture illumation
20
Synthetic aperture illumation

• technologies
• array of projectors
• array of microprojectors
• single projector array of mirrors
• applications
• bright display
• autostereoscopic display Matusik 2004
• confocal imaging this paper

21
Confocal scanning microscopy
22
Confocal scanning microscopy
23
Confocal scanning microscopy
light source
pinhole
pinhole
photocell
24
Confocal scanning microscopy
light source
pinhole
pinhole
photocell
25
UMIC SUNY/Stonybrook
26
Synthetic confocal scanning
light source
27
Synthetic confocal scanning
light source
28
Synthetic confocal scanning

• works with any number of projectors 2
• discrimination degrades if point to left of
• no discrimination for points to left of
• slow!
• poor light efficiency

29
Synthetic coded-apertureconfocal imaging

• different from coded aperture imaging in
  astronomy
• Wilson, Confocal Microscopy by Aperture
  Correlation, 1996

30
Synthetic coded-apertureconfocal imaging
31
Synthetic coded-apertureconfocal imaging
32
Synthetic coded-apertureconfocal imaging
33
Synthetic coded-apertureconfocal imaging
100 trials ? 2 beams 50/100 trials 1
? 1 beam 50/100 trials 0.5
34
Synthetic coded-apertureconfocal imaging
100 trials ? 2 beams 50/100 trials 1
? 1 beam 50/100 trials 0.5 floodlit
? 2 beams ? 2 beams trials ¼
floodlit ? 1 ¼ ( 2 ) 0.5 ? 0.5
¼ ( 2 ) 0
35
Synthetic coded-apertureconfocal imaging
100 trials ? 2 beams 50/100 trials 1
? 1 beam 50/100 trials 0.5 floodlit
? 2 beams ? 2 beams trials ¼
floodlit ? 1 ¼ ( 2 ) 0.5 ? 0.5
¼ ( 2 ) 0

• 50 light efficiency
• any number of projectors 2
• no discrimination to left of
• works with relatively few trials (16)

36
Synthetic coded-apertureconfocal imaging
100 trials ? 2 beams 50/100 trials 1
? 1 beam 50/100 trials 0.5 floodlit
? 2 beams ? 2 beams trials ¼
floodlit ? 1 ¼ ( 2 ) 0.5 ? 0.5
¼ ( 2 ) 0

• 50 light efficiency
• any number of projectors 2
• no discrimination to left of
• works with relatively few trials (16)
• needs patterns in which illumination of tiles are
  uncorrelated

37
Example pattern
38
Patterns with less aliasing
39
Implementation using an array of mirrors
40

41
Confocal imaging in scattering media

• small tank
• too short for attenuation
• lit by internal reflections

42
Experiments in a large water tank
50-foot flume at Woods Hole Oceanographic
Institution (WHOI)
43
Experiments in a large water tank

• 4-foot viewing distance to target
• surfaces blackened to kill reflections
• titanium dioxide in filtered water
• transmissometer to measure turbidity

44
Experiments in a large water tank

• stray light limits performance
• one projector suffices if no occluders

45
Seeing through turbid water
floodlit
scanned tile
46
Application tounderwater exploration
Ballard/IFE 2004
47
Research challenges in SAP and SAI

• theory
• aperture shapes and sampling patterns
• illumination patterns for confocal imaging
• optical design
• How to arrange cameras, projectors,lenses,
  mirrors, and other optical elements?
• How to compare the performance of different
  arrangements (in foliage, underwater,...)?

48
Challenges (continued)

• systems design
• multi-camera or multi-projector chips
• communication in camera-projector networks
• calibration in long-range or mobile settings
• algorithms
• tracking and stabilization of moving objects
• compression of dense multi-view imagery
• shape from light fields

49
Challenges (continued)

• applications of confocal imaging
• remote sensing and surveillance
• shape measurement
• scientific imaging
• applications of shaped illumination
• shaped searchlights for surveillance
• shaped headlamps for driving in bad weather
• selective lighting of characters for stage and
  screen

50
Computational imagingin other fields

• medical imaging
• rebinning
• tomography
• airborne sensing
• multi-perspective panoramas
• synthetic aperture radar
• astronomy
• coded-aperture imaging
• multi-telescope imaging

51
Computational imagingin other fields

• geophysics
• accoustic array imaging
• borehole tomography
• biology
• confocal microscopy
• deconvolution microscopy
• physics
• optical tomography
• inverse scattering

52
The team

• staff
• Mark Horowitz
• Marc Levoy
• Bennett Wilburn
• students
• Billy Chen
• Vaibhav Vaish
• Katherine Chou
• Monica Goyal
• Neel Joshi
• Hsiao-Heng Kelin Lee
• Georg Petschnigg
• Guillaume Poncin
• Michael Smulski
• Augusto Roman

• collaborators
• Mark Bolas
• Ian McDowall
• Guillermo Sapiro
• funding
• Intel
• Sony
• Interval Research
• NSF
• DARPA

53
Related papers

• The Light Field Video Camera
• Bennett Wilburn, Michael Smulski, Hsiao-Heng
  Kelin Lee, and Mark Horowitz
• Proc. Media Processors 2002, SPIE Electronic
  Imaging 2002
• Using Plane Parallax for Calibrating Dense
  Camera Arrays
• Vaibhav Vaish, Bennett Wilburn, Neel Joshi,, Marc
  Levoy
• Proc. CVPR 2004
• High Speed Video Using a Dense Camera Array
• Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Marc
  Levoy, Mark Horowitz
• Proc. CVPR 2004
• Spatiotemporal Sampling and Interpolation for
  Dense Camera Arrays
• Bennett Wilburn, Neel Joshi, Katherine Chou, Marc
  Levoy, Mark Horowitz
• ACM Transactions on Graphics (conditionally
  accepted)

54
http//graphics.stanford.edu/papers/confocal