Stereo matching Class 7

1
Stereo matchingClass 7

• Read Chapter 7 of tutorial
• http//cat.middlebury.edu/stereo/

Tsukuba dataset
2
Stereo

• Standard stereo geometry
• Stereo matching
• Correlation
• Optimization (DP, GC)
• General camera configuration
• Rectification
• Plane-sweep
• Multi-view stereo

3
Standard stereo geometry
pure translation along X-axis
4
Standard stereo geometry
5
Stereo matching

• Search is limited to epipolar line (1D)
• Look for most similar pixel

?
6
Aggregation

• Use more than one pixel
• Assume neighbors have similar disparities
• Use correlation window containing pixel
• Allows to use SSD, ZNCC, Census, etc.

7
Comparing image regions

• Compare intensities pixel-by-pixel

I(x,y)
I(x,y)
Dissimilarity measures
Sum of Square Differences
8
Comparing image regions

• Compare intensities pixel-by-pixel

I(x,y)
I(x,y)
Similarity measures
Zero-mean Normalized Cross Correlation
9
Comparing image regions

• Compare intensities pixel-by-pixel

I(x,y)
I(x,y)
Similarity measures
Census
125 126 125
127 128 130
129 132 135
0 0 0
0 1
1 1 1
only compare bit signature
(Real-time chip from TYZX based on Census)
10
Aggregation window sizes

• Small windows
• disparities similar
• more ambiguities
• accurate when correct

• Large windows
• larger disp. variation
• more discriminant
• often more robust
• use shiftable windows to deal with
  discontinuities

(Illustration from Pascal Fua)
11
Occlusions
(Slide from Pascal Fua)
12
(No Transcript)
13
Real-time stereo on GPU
(Yang and Pollefeys, CVPR2003)

• Computes Sum-of-Square-Differences (use
  pixelshader)
• Hardware mip-map generation for aggregation over
  window
• Trade-off between small and large support window

290M disparity hypothesis/sec (Radeon9800pro) e.g.
512x512x36disparities at 30Hz
GPU is great for vision too!
14
Exploiting scene constraints
15
Ordering constraint
surface slice
surface as a path
6
5
occlusion left
4
3
1
2
4,5
6
1
2,3
2,3
4
5
6
occlusion right
1
3
6
2
1
4,5
16
Uniqueness constraint

• In an image pair each pixel has at most one
  corresponding pixel
• In general one corresponding pixel
• In case of occlusion there is none

17
Disparity constraint
surface slice
surface as a path
bounding box
disparity band
constant disparity surfaces
18
Stereo matching

• Constraints
• epipolar
• ordering
• uniqueness
• disparity limit
• Trade-off
• Matching cost (data)
• Discontinuities (prior)

Consider all paths that satisfy the
constraints pick best using dynamic programming
19
Hierarchical stereo matching
Allows faster computation Deals with large
disparity ranges
Downsampling (Gaussian pyramid)
Disparity propagation
20
Disparity map
image I(x,y)
image I(x,y)
Disparity map D(x,y)
(x,y)(xD(x,y),y)
21
Example reconstruct image from neighboring images
22
(No Transcript)
23
Energy minimization
(Slide from Pascal Fua)
24
Graph Cut
(general formulation requires multi-way cut!)
(Slide from Pascal Fua)
25
Simplified graph cut
(Roy and Cox ICCV98)
(Boykov et al ICCV99)
26
(No Transcript)
27
Stereo matching with general camera configuration
28
Image pair rectification
29
Planar rectification
Distortion minimization (uncalibrated)
Bring two views to standard stereo setup (moves
epipole to ?) (not possible when in/close to
image)
30
(No Transcript)
31
Polar rectification
(Pollefeys et al. ICCV99)
Polar re-parameterization around
epipoles Requires only (oriented) epipolar
geometry Preserve length of epipolar lines Choose
?? so that no pixels are compressed
original image
rectified image
Works for all relative motions Guarantees minimal
image size
32
original image pair
planar rectification
polar rectification
33
Example Béguinage of Leuven
Does not work with standard Homography-based
approaches
34
Example Béguinage of Leuven
35
General iso-disparity surfaces
(Pollefeys and Sinha, ECCV04)
Example polar rectification preserves disp.
Application Active vision
Also interesting relation to human horopter
36
Stereo camera configurations
(Slide from Pascal Fua)
37
Multi-camera configurations
(illustration from Pascal Fua)
Okutami and Kanade
38
Multi-view depth fusion
(Koch, Pollefeys and Van Gool. ECCV98)

• Compute depth for every pixel of reference image
• Triangulation
• Use multiple views
• Up- and down sequence
• Use Kalman filter

Allows to compute robust texture
39
Plane-sweep multi-view matching

• Simple algorithm for multiple cameras
• no rectification necessary
• doesnt deal with occlusions

Collins96 Roy and Cox98 (GC) Yang et
al.02/03 (GPU)
40
Space Carving
41
3D Reconstruction from Calibrated Images
Scene Volume V
Input Images (Calibrated)
Goal Determine transparency, radiance of points
in V
42
Discrete Formulation Voxel Coloring
Discretized Scene Volume
Input Images (Calibrated)
Goal Assign RGBA values to voxels in
V photo-consistent with images
43
Complexity and Computability
Discretized Scene Volume
3
N voxels C colors
44
Issues

• Theoretical Questions
• Identify class of all photo-consistent scenes
• Practical Questions
• How do we compute photo-consistent models?

45
Voxel Coloring Solutions

• 1. C2 (silhouettes)
• Volume intersection Martin 81, Szeliski 93
• 2. C unconstrained, viewpoint constraints
• Voxel coloring algorithm Seitz Dyer 97
• 3. General Case
• Space carving Kutulakos Seitz 98

46
Reconstruction from Silhouettes (C 2)
Binary Images
Approach Backproject each silhouette Intersect
backprojected volumes
47
Volume Intersection

• Reconstruction Contains the True Scene
• But is generally not the same
• In the limit get visual hull
• Complement of all lines that dont intersect S

48
Voxel Algorithm for Volume Intersection

• Color voxel black if on silhouette in every image
• O(MN3), for M images, N3 voxels
• Dont have to search 2N3 possible scenes!

49
Properties of Volume Intersection

• Pros
• Easy to implement, fast
• Accelerated via octrees Szeliski 1993
• Cons
• No concavities
• Reconstruction is not photo-consistent
• Requires identification of silhouettes

50
Voxel Coloring Solutions

• 1. C2 (silhouettes)
• Volume intersection Martin 81, Szeliski 93
• 2. C unconstrained, viewpoint constraints
• Voxel coloring algorithm Seitz Dyer 97
• 3. General Case
• Space carving Kutulakos Seitz 98

51
Voxel Coloring Approach
Visibility Problem in which images is each
voxel visible?
52
The Global Visibility Problem
Which points are visible in which images?
53
Depth Ordering visit occluders first!
Scene Traversal
Condition depth order is view-independent
54
Panoramic Depth Ordering

• Cameras oriented in many different directions
• Planar depth ordering does not apply

55
Panoramic Depth Ordering
Layers radiate outwards from cameras
56
Panoramic Layering
Layers radiate outwards from cameras
57
Panoramic Layering
Layers radiate outwards from cameras
58
Compatible Camera Configurations
Depth-Order Constraint Scene outside convex hull
of camera centers
59
Calibrated Image Acquisition
Selected Dinosaur Images

• Calibrated Turntable
• 360 rotation (21 images)

Selected Flower Images
60
Voxel Coloring Results (Video)
Dinosaur Reconstruction 72 K voxels colored 7.6
M voxels tested 7 min. to compute on a 250MHz
SGI
Flower Reconstruction 70 K voxels colored 7.6 M
voxels tested 7 min. to compute on a 250MHz SGI
61
Limitations of Depth Ordering

• A view-independent depth order may not exist

Need more powerful general-case
algorithms Unconstrained camera
positions Unconstrained scene geometry/topology
62
Voxel Coloring Solutions

• 1. C2 (silhouettes)
• Volume intersection Martin 81, Szeliski 93
• 2. C unconstrained, viewpoint constraints
• Voxel coloring algorithm Seitz Dyer 97
• 3. General Case
• Space carving Kutulakos Seitz 98

63
Space Carving Algorithm
Image 1
Image N
...

• Space Carving Algorithm

64
Convergence

• Consistency Property
• The resulting shape is photo-consistent
• all inconsistent points are removed
• Convergence Property
• Carving converges to a non-empty shape
• a point on the true scene is never removed

65
What is Computable?
V
True Scene

• The Photo Hull is the UNION of all
  photo-consistent scenes in V
• It is a photo-consistent scene reconstruction
• Tightest possible bound on the true scene
• Computable via provable Space Carving Algorithm

66
Space Carving Algorithm

• The Basic Algorithm is Unwieldy
• Complex update procedure
• Alternative Multi-Pass Plane Sweep
• Efficient, can use texture-mapping hardware
• Converges quickly in practice
• Easy to implement

67
Space Carving Algorithm

• Step 1 Initialize V to volume containing true
  scene
• Step 2 For every voxel on surface of V
• test photo-consistency of voxel
• if voxel is inconsistent, carve it
• Step 3 Repeat Step 2 until all voxels
  consistent
• Convergence
• Always converges to a photo-consistent model
• (when all assumptions are met)
• Good results on difficult real-world scenes

68
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

True Scene
Reconstruction
69
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

70
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

71
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

72
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

73
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

74
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

75
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

76
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

77
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

78
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

79
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

80
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

81
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

82
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

83
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

84
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

85
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

86
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

87
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

88
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

89
Multi-Pass Plane Sweep

• Sweep plane in each of 6 principle directions
• Consider cameras on only one side of plane
• Repeat until convergence

90
Space Carving Results African Violet
Input Image (1 of 45)
Reconstruction
Reconstruction
Reconstruction
91
Space Carving Results Hand
Input Image (1 of 100)
Views of Reconstruction
92
Other Features

• Coarse-to-fine Reconstruction
• Represent scene as octree
• Reconstruct low-res model first, then refine
• Hardware-Acceleration
• Use texture-mapping to compute voxel projections
• Process voxels an entire plane at a time
• Limitations
• Need to acquire calibrated images
• Restriction to simple radiance models
• Bias toward maximal (fat) reconstructions
• Transparency not supported

93
(No Transcript)
94
Probalistic Space Carving
Broadhurst et al. ICCV01
95
The Master's Lodge Image Sequence
Bayesian
96
Space-carving for specular surfaces (Yang,
Pollefeys Welch 2003)
Extended photoconsistency
I Light Intensity Object Color
N Normal vector L Lighting vector
V View Vector R Reflection vector
97
Experiment
98
Animated Views
Our result
99
Other Approaches
Level-Set Methods Faugeras Keriven
1998 Evolve implicit function by solving
PDEs
100

• Graph-cut-based approach
• Refine using surface evolution

e.g. Sinha et al. ICCV07
101
Interesting comparison of multi-view stereo
approaches
http//vision.middlebury.edu/mview/
102
Next class structured light and active scanning