Video Mosaics for Virtual Environments

1
Video Mosaics for Virtual Environments

• by Richard Szeliski
• Presented by Joseph Cooper

2
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

3
Possible Applications

• Virtual Environments
• Big Pictures
• Model Construction

4
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

5
Basic Transformation Matrix

• Scale
• Rotation
• Translation
• Projection

6
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

7
Planar Mosaics

• Two degrees of freedom
• Three degrees of freedom
• Six degrees of freedom
• Eight degrees of freedom

8
Alignment

• Manually associate four points
• Eight equations, eight unknowns
• Ugh
• Minimize sum-squared intensity error
• Calculate eight partial derivatives
• Levenberg-Marquardt minimization
• Essentially just gradient descent
• Variable learning rate for speed-up

9
Transform and Match

• Select a base image
• Blend edges
• May leave splotches
• Gradient descent only does local maximization

10
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

11
Global Registration

• Hierarchical Matching
• Multiple levels of zoom
• Match progressively more detail
• 2D Fourier Correlation
• Maximize overlap
• Does not work well with excessive zoom or
  rotation

12
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

13
Panoramic Mosaic

• Rotate around the optical center
• Lack of Parallax means this is essentially still
  2D
• Single Plane
• Multiple Planes
• Cylinder

14
Panoramic Mosaic

• Textured Cylinder
• (x,y,z) -gt u(T,v) T e (-p,p
• T tan-1(x/z), v y/v(x2 z2)

15
Stitching Imagery

• Possible applications
• Matrix math
• Planar mosaics
• Global registration
• Panoramic images
• Depth recovery

16
Depth Recovery

• Piecewise-planar
• Similar to the 2D case
• Someone elses work

17
Structure-from-motion

• Transformation of scene and camera
• -gt too many degrees of freedom
• Alternating Levenberg-Marquardt (planar)
  maximization with depth parameters of spline-mesh
  control points

18
Results

• Sampled vertically
• Unknown number of frames
• Cannot recover depth from homogenous area
• Texture mesh to render from a novel viewpoint

19
Acknowledgments