CS 395: Adv' Computer Graphics

1
CS 395 Adv. Computer Graphics

• Light Fields and
• their Approximations
• Jack Tumblin
• jet_at_cs.northwestern.edu

2
GOAL First-Class Primitive

• Want images as first-class primitives
• Useful as BOTH input and output
• Convert to/from traditional scene descriptions
• Want to mix real synthetic scenes freely
• Want to extend photography
• Easily capture sceneshape, movement,
  surface/BRDF, lighting
• Modify Render the captured scene data
• --BUT--
• images hold only PARTIAL scene information
• You cant always get what you want (Mick Jagger
  1968)

3
Back To Basics Scene Image

• Light 3D Scene
• Illumination, shape, movement, surface BRDF,

2D Image Collection of rays through a point
Image Plane I(x,y)
Position(x,y)
Angle(?,?)
4
Trad. Computer Graphics
2D Image Collection of rays through a point

• Light 3D Scene
• Illumination, shape, movement, surface BRDF,

Reduced, Incomplete Information
Image Plane I(x,y)
Position(x,y)
Angle(?,?)
5
Trad. Computer Vision
2D Image Collection of rays through a point

• Light 3D Scene
• Illumination, shape, movement, surface BRDF,

!TOUGH! ILL-POSED Many Simplifications, External
knowledge
Image Plane I(x,y)
Position(x,y)
Angle(?,?)
6
Plenoptic Function (Adelson, Bergen 91)

• for a given scene, describe
• ALL rays through
• ALL pixels, of
• ALL cameras, at
• ALL wavelengths,
• ALL time
• F(x,y,z, ?,?, ?, t)
• Eyeballs Everywhere function (7-D!)

7
Image-Based 3D Photography

• Cleaner Formulation
• Orthographic camera,
• positioned on sphere around object/scene
• Orthographic projector,
• positioned on spherearound object/scene
• F(xc,yc,?c,?c,xl,yl ?l,?l, ?, t)

camera
8
Image-Based 3D Photography

• Cleaner Formulation
• Orthographic camera,
• positioned on sphere around object/scene
• Orthographic projector,
• positioned on spherearound object/scene
• F(xc,yc,?c,?c,xl,yl ?l,?l, ?, t)

camera
?c
?c
9
Image-Based 3D Photography

• Cleaner Formulation
• Orthographic camera,
• positioned on sphere around object/scene
• Orthographic projector,
• positioned on spherearound object/scene
• F(xc,yc,?c,?c,xl,yl ?l,?l, ?, t)

camera
projector
10
Image-Based 3D Photography

• Cleaner Formulation
• Orthographic camera,
• positioned on sphere around object/scene
• Orthographic projector,
• positioned on spherearound object/scene
• (and wavelength and time)
• F(xc,yc,?c,?c,xl,yl ?l,?l, ?, t)

camera
projector
11
Image-Based 3D Photography

• Cleaner Formulation
• Orthographic camera,
• positioned on sphere around object/scene
• Orthographic projector,
• positioned on spherearound object/scene
• (and wavelength and time)
• F(xc,yc,?c,?c,xl,yl ?l,?l, ?, t)

camera
projector
12
Plenoptic Function (Adelson, Bergen 91)

• ! Complete !
• Geometry, Lighting, BRDF, ! NOT REQUIRED !
• Preposterously Huge (?!?! 8-D function
  sampled at image resolution !?!?), but
• Hugely Redundant
• Wavelength?RGB triplets, ignore Time, and
  Restrict eyepoint movement maybe 3 or 4D ?
• Very Similar imagesuse Warping rules? (SIGG2002)
• Exploit Movie Storage/Compression Methods

13
The Big Idea Image-Based Modeling Rendering

• Computer Graphics given scene description,
  synthesize images
• Digital Photography given scene, capture
  images
• Computer Vision given images, assumptions, (
  hope) describe scene
• Image-Based Modeling Rendering Collect images
  to estimate Plenoptic Function QUICKLY recall /
  interpolate images
• Image-Based Modeling Collect images... Estimat
  e scene (maybe all that data makes it easier...)

14
OLDEST IBR Shadow Maps (1984)

• Fast Shadows from Z-buffer hardware
• 1) Make the Shadow Map
• Render image seen at light source, BUT
• Keep ONLY the Z-buffer values (depth)
• 2) Render Scene from Eyepoint
• Pixel Z depth gives 3D position of surface
• Project 3D position into Shadow map
• If Shadow Map depth lt 3D depth, SHADOW!
• MANY sampling hacks (See Eric Haines)
• nVidia / OpenGL implementations available
  http//developer.nvidia.com/view.asp?IOShadow_Map

15
Early IBR QuickTime VR (Chen, Williams 93)

• 1) Four Planar Images ? 1 Cylindrical Panorama
• Re-sampling Required!
• Planar Pixels equal distance on x,y plane
  (tan-1?)
• Cylinder Pixs horiz equal angle on cylinder
  (?) vert equal distance on y (tan-1?)

16
Early IBR QuickTime VR (Chen, Williams 93)

• 1) Four Planar Images ? 1 Cylindrical Panorama

IN OUT
17
Early IBR QuickTime VR (Chen, Williams 93)

• 2) Windowing, Horizontal-only Reprojection

IN OUT
18
Plenoptic Array The Matrix Effect

• Simple arc, line, or ring array of cameras
• Synchronized shutter
• Warp/blend between images to change viewpoint on
  time-frozen scene
• http//www.ruffy.com/firingline.html

19
View Interpolation How?

• Traditional Stereo Disparity Map
  pixel-by-pixel search for correspondence

(Lighter more Horiz. Separation)
20
View Interpolation How?

• Store Depth at each pixel reproject
• Coarse or Simple 3D model

21
View Interpolation

• Problems
• Visibility Changes
• Disocclusion, or holes in the picture
• Partial Answers
• More Pictures
• Interpolate Nearby Pixels
• Better Multiple Samples (LDI Trees SIGG99)

Lview
Rview
22
Plenoptic Modeling (McMillan95)

• Alerted CGI community to notion of Plenoptic
  Function
• Solved depth order problem read images in
  occlusion-compatible order
• Demonstrated reprojection from two spherical
  projections

23
Seitz View Morphing SIGG96

• http//www.cs.washington.edu/homes/seitz/vmorph/v
  morph.htm
• 1)Manually set some
• corresp.points
• (eye corners, etc.)
• 2) pre-warp and
• post-warp to match
• points in 3D,
• 3) Reproject for
• Virtual cameras

24
Seitz View Morphing SIGG96

• http//www.cs.washington.edu/homes/seitz/vmorph/vm
  orph.htm

25
Seitz View Morphing SIGG96

• http//www.cs.washington.edu/homes/seitz/vmorph/vm
  orph.htm

26
Seitz View Morphing SIGG96

• http//www.cs.washington.edu/homes/seitz/vmorph/vm
  orph.htm

27
Seitz View Morphing SIGG96

• http//www.cs.washington.edu/homes/seitz/vmorph/vm
  orph.htm

28
Light Fields / Lumigraphs

• Subset of Plenoptic Function
• Let Convex Surface surround scene
• Place in-facing camera at every surface point
• Captures ALL viewing rays for object!
• (But does NOT separate external light efx)

29
Light Fields / Lumigraphs

• Two-plane parameterization
• Generalized camera TWO image planes
• Sets angle AND position
• http//research.microsoft.com/MSRSIGGRAPH/96/Lumig
  raph.htm

L(u,v,s,t)
t
v
s
u
30
Light Fields / Lumigraphs

• .

u
31
Hierarchical Layered Depth Images

• A stack of unstructured 3-D sample points
• Store, retrieve data at multiple resolutions
• Eliminates redundant storage
• Can be updated dynamically with new data (from
  renderer or camera)
• Demonstrated for real-time walkthroughs

32
Hybrid Approaches

• View-Based Modeling (Debevec 96)
• Simple Initial Geometry (building box)
• decorated by image textures shape details
• Voxel-Carving (Dellaert? check this!)
• Remove all voxels you can see through
• Enough views?tight hull around geometry
• Image-based Visual Hull (Matusic2000,2002)
• Intersection of silhouettes defines enclosing
  surface
• put light field on that surface

33
IBR-Motivating Opinions

• Computer Graphics Hard (asymptotically false)
• Complex! geometry, texture, lighting, shadows,
  compositing, BRDF, interreflections, etc. etc.,
  etc.,
• Irregular! Visibility,Topology, Rendering Eqn.,
• Isolated! Tough to use real objects in CGI vice
  versa
• Slow! compute-bound, off-line only for good
  stuff,
• Digital Imaging Easy (for 1 picture, yeah,
  but 10,000 ?!)
• Simple! More quality? Just pump more pixels!
• Regular! Vectorized, compressible, pipelined
• Accessible! Use real OR synthetic (CGI) images!
• Fast! Scalable, Image reuse, a path to
  interactivity

34
IBMR Difficulties

• Sampling / Reconstruction problems
• Brutal Force Bandwidth Memory Hog
• Easy to Display??Difficult to Edit
• Increased Flexibility ? Reduced Regularity
• more complex storage / retrieval,
• more computing,
• less deterministic (cache miss, etc.)

35
Interesting Recent Work

• Surface Light Fields for 3D PhotographyWood et
  al., SIGG2000 http//www.cs.washington.edu/homes/d
  aniel/siggraph2000-slf.pdf
• Plenoptic Stitching Aliaga et al. SIGG2001
  http//www.bell-labs.com/user/aliaga/sig01.pdf
• Single-view Modeling Seitz,CVPR2001
  http//grail.cs.washington.edu/projects/svm/