Surface%20Light%20Fields%20for%203D%20Photography

1
Surface Light Fieldsfor 3D Photography

• Daniel Wood Daniel Azuma Wyvern Aldinger
• Brian Curless Tom Duchamp
• David Salesin Werner Stuetzle

2
3D Photography
Goals Rendering and editing Inputs Photographs
and geometry Requirements Estimation and
compression
3
View-dependent texture mapping
Debevec et al. 1996, 1998 Pulli et al. 1997
4
View-dependent texture mapping
Debevec et al. 1996, 1998 Pulli et al. 1997
5
Two-plane light field
Levoy and Hanrahan 1996 Gortler et al. 1996
6
Surface light fields
Walter et al. 1997 Miller et al. 1998 Nishino et
al. 1999
7
Lumisphere-valued texture maps
Lumisphere
8
Overview
Data acquisition
Estimation and compression
Rendering
Editing
9
Overview
Data acquisition
Estimation and compression
Rendering
Editing
10
Scan and reconstruct geometry
Reconstructed geometry
Range scans (only a few shown . . .)
11
Take photographs
Camera positions
Photographs
12
Register photographs to geometry
Photographs
Geometry
13
Register photographs to geometry
User selected correspondences (rays)
14
Parameterizing the geometry
Map
Base mesh
Scanned geometry
15
Sample base mesh faces
Base mesh
Detailed geometry
16
Assembling data lumispheres
Data lumisphere
17
Overview
Data acquisition
Estimation and compression
Rendering
Editing
18
Pointwise fairing
Faired lumisphere
Data lumisphere
19
Pointwise fairing results
Pointwise faired (177 MB)
Input photograph
20
Pointwise fairing
Many faired lumispheres
Many input data lumispheres
21
Compression
Small set of prototypes
22
Compression / Estimation
Small set of prototypes
Many input data lumispheres
23
Reflected reparameterization
24
Reflected reparameterization
25
Reflected reparameterization
26
Reflected reparameterization
Before
After
27
Median removal
Reflected
Median (diffuse)


Median-removed (specular)
28
Median removal
Median values
Specular
Result
29
Function quantization
Input data lumisphere
Codebook of lumispheres
30
Lloyd iteration
Input data lumispheres
31
Lloyd iteration
Codeword
32
Lloyd iteration
Perturb codewords to create larger codebook
33
Lloyd iteration
Form clusters around each codeword
34
Lloyd iteration
Optimize codewords based on clusters
35
Lloyd iteration
Create new clusters
36
Function quantization results
Function quantized (1010 codewords, 2.6 MB)
Input photograph
37
Principal function analysis
Input data lumisphere
Prototype lumisphere
Subspace of lumispheres
38
Principal function analysis
Prototype lumisphere
Approximating subspace
39
Principal function analysis
40
Principal function analysis
41
Principal function analysis results
PFA compressed (Order 5 - 2.5 MB)
Input photograph
42
Compression comparison
Pointwise fairing (177 MB)
Function quantization (2.6 MB)
Principal function analysis (2.5 MB)
43
Comparison with 2-plane light field(uncompressed)
Pointwise-faired surface light field (177 MB)
Uncompressed lumigraph / light field (177 MB)
44
Comparison with 2-plane light field(compressed)
Compressed (PFA) surface light field (2.5 MB)
Vector-quantized lumigraph / light field (8.1 MB)
45
Overview
Data acquisition
Estimation and compression
Rendering
Editing
46
View-dependent level-of-detail
47
Render texture domain and coordinates in false
color
48
Evaluate surface light field
49
Interactive rendererscreen capture
50
Overview
Data acquisition
Estimation and compression
Rendering
Editing
51
Lumisphere filtering
Original surface light field
Glossier coat
52
Lumisphere filtering
53
Rotating the environment
Original surface light field
Rotated environment
54
Deformation
Original
Deformed
55
Deformation
56
Summary

• Estimation and compression
• Function quantization
• Principal function analysis
• Rendering
• From compressed representation
• With view-dependent level-of-detail
• Editing
• Lumisphere filtering
• Geometric deformations and transformations

57
Future work

• Better geometry-to-image registration
• More complex surfaces (mirrored, refractive,
  fuzzy) under more complex illumination
• Derive geometry from images
• Combining FQ and PFA

58
Acknowledgements

• Marc Levoy and Pat Hanrahan
• (Thanks for the use of the Stanford Spherical
  Gantry)
• Michael Cohen and Richard Szeliski
• National Science Foundation

59
The end
60
Geometry (fish)

• Reconstruction 129,000 faces
• Memory for reconstruction 2.5 MB
• Base mesh 199 faces
• Re-mesh (4x subdivided) 51,000 faces
• Memory for re-mesh 1 MB
• Memory with view-dependence 7.5 MB

61
Light field data and rep (fish)

• Time to acquire 1 hour
• Input images 661
• Raw data size 500 MB
• Lumisphere representation
• 3 times subdivided octehedron
• Lumisphere size 258 directions

62
Lumigraph (fish)

• Lumigraph images 400x400
• Lumigraph viewpoints per slab 8x8
• Number of slabs 6
• Lumigraph size (w/o geom) 184 MB
• Lumigraph VQ dimension 16384
• Lumigraph VQ codewords 2x2x2x2x3
• Compressed size (w/o geom) 8.1 MB

63
Compression (fish)

• Pointwise faired
• Memory 177 MB RMS error 9
• FQ (2000 codewords)
• Memory 3.4 MB RMS error 23
• PFA (dimension 3)
• Memory 2.5 MB RMS error 24
• PFA (dimension 5)
• Memory 2.9 MB RMS error ?

64
Pre-processing times (fish)

• Compute times on 450 MHz P-III
• Range scanning time 3 hours
• Geometry registration 2 hours
• Image to geometry alignment 6 hours
• MAPS (sub-optimal) 5 hours
• Assembling data lumispheres 24 hours
• Pointwise fairing 30 minutes
• FQ codebook construction (10) 30 hours
• FQ encoding 4 hours
• PFA codebook construction (0.1) 20 hours
• PFA encoding 2 hours

65
Breakdown and rendering (fish)

• For PFA dimension 3
• Direction mesh 11 KB
• Normal maps 680 KB
• Median maps 680 KB
• Index maps 455 KB
• Weight maps 680 KB
• Codebook 3 KB
• Geometry w/o view dependence lt1 MB
• Geometry w/ view dependence 7.5 MB
• Rendering platform 550 MHz PIII, linux, Mesa
• Rendering performance 6-7 fps (typical)

66
Construct codebook using Lloyd iteration

• Iterate until convergence
• Assign all data lumispheres to closest codeword,
  forming clusters.
• Compute new codeword for each cluster by
  cluster-wise fairing.
• Then split all codewords and start over.

67
Data extrapolation
Photograph
Surface light field
68
Comparison with 2-plane light field(uncompressed)
Pointwise-faired surface light field (177 MB)
Uncompressed 2-plane light field (177 MB)
69
Comparison with 2-plane light field(compressed)
Principal function analysis surface light field
(2.5 MB)
Vector-quantized 2-plane light field (8.1 MB)
70
Details
Pointwise fairing (177 MB)
Principal function analysis (2.5 MB)
Input photograph
Function quantization (3.4 MB)