Unstructured Lumigraph Rendering

1
Unstructured Lumigraph Rendering

• Chris Buehler
• Michael Bosse
• Leonard McMillan MIT-LCS
• Steven J. Gortler Harvard University
• Michael F. Cohen Microsoft Research

2
The Image-Based Rendering Problem

• Synthesize novel views from reference images
• Static scenes, fixed lighting
• Flexible geometry and camera configurations

3
The ULR Algorithm

• Designed to work over a range of image and
  geometry configurations
• Designed to satisfy desirable properties

of Images
Geometric Fidelity
4
Light Field Rendering, SIGGRAPH 96
Desired color interpolated from nearest cameras
5
Light Field Rendering, SIGGRAPH 96
6
The Lumigraph, SIGGRAPH 96
7
The Lumigraph, SIGGRAPH 96
8
The Lumigraph, SIGGRAPH 96
9
The Lumigraph, SIGGRAPH 96
Rebinning
Note all images are resampled.
10
The Lumigraph, SIGGRAPH 96
11
View-Dependent Texture Mapping, SIGGRAPH 96,
EGRW 98
12
View-Dependent Texture Mapping, SIGGRAPH 96,
EGRW 98
13
View-Dependent Texture Mapping, SIGGRAPH 96,
EGRW 98
?1
?3
?2
14
View-Dependent Texture Mapping, SIGGRAPH 96,
EGRW 98
?1
?3
?2
15
(No Transcript)
16
(No Transcript)
17
Previous Work

• Light fields and Lumigraphs
• Levoy and Hanrahan, Gortler et al., Isaksen et
  al.
• View-dependent Texture Mapping
• Debevec et al., Wood et al.
• Plenoptic Modeling w/Hand-held Cameras
• Heigl et al.
• Many others

18
Unstructured Lumigraph Rendering

1. Epipole consistency
2. Use of geometric proxy
3. Unstructured input
4. Real-time implementation
5. Continuous reconstruction
6. Angles measured w.r.t. proxy
7. Resolution sensitivity

19
Blending Fields
20
Blending Fields
21
Unstructured Lumigraph Rendering

• Explicitly construct blending field
• Computed using penalties
• Sample and interpolate over desired image
• Render with hardware
• Projective texture mapping and alpha blending

22
Angle Penalty
Geometric Proxy
C6
C1
C5
C2
C4
C3
Cdesired
penaltyang(Ci) ?i
23
Resolution Penalty
Geometric Proxy
penaltyres
Ci
distdesired
penaltyres(Ci) max(0,dist(Ci) dist(Cdesired ))
24
Field-Of-View Penalty
penaltyFOV
angle
25
Total Penalty
26
K-Nearest Continuous Blending

• Only use cameras with K smallest penalties
• C0 Continuity contribution drops to zero as
  camera leaves K-nearest set
• w(Ci) 1- penalty(Ci)/penalty(Ck1st closest )
• Partition of Unity normalize
• w(Ci) w(Ci)/Sw(Cj)

j
27
Blending Field Visualization
28
Sampling Blending Fields
Just epipole sampling
Epipole and grid sampling
29
Hardware Assisted Algorithm
Sample Blending Field
Select blending field sample locations for each
sample location j do for each camera i
do Compute penalty(i) for sample location
j end for Find K smallest penalties Compute
blending weights for sample location j end
for Triangulate sample locations
Render with Graphics Hardware

• Clear frame buffer
• for each camera i do
• Set current texture and projection matrix
• Copy blending weights to vertices alpha channel
• Draw triangles with non-zero alphas
• end for

30
Blending over one triangle
Just epipole sampling
Epipole and grid sampling
31
Hardware Assisted Algorithm
Sample Blending Field
Select blending field sample locations for each
sample location j do for each camera i
do Compute penalty(i) for sample location
j end for Find K smallest penalties Compute
blending weights for sample location j end
for Triangulate sample locations
Render with Graphics Hardware

• Clear frame buffer
• for each camera i do
• Set current texture and projection matrix
• Copy blending weights to vertices alpha channel
• Draw triangles with non-zero alphas
• end for

32
Demo
33
Future Work

• Optimal sampling of the camera blending field
• More complete treatment of resolution effects in
  IBR
• View-dependent geometry proxies
• Investigation of geometry vs. images tradeoff

34
Conclusions

• Unstructured Lumigraph Rendering
• unifies view-dependent texture mapping and
  lumigraph rendering methods
• allows rendering from unorganized images
• sampled camera blending field

35
Acknowledgements

• Thanks to the members of the
• MIT Computer Graphics GroupandMicrosoft
  ResearchGraphics and Computer Vision Groups
• DARPA ITO Grant F30602-971-0283
• NSF CAREER Awards 9875859 9703399
• Microsoft Research Graduate Fellowship Program
• Donations from Intel Corporation, Nvidia,
  andMicrosoft Corporation