More on stereo and correspondence

1
More on stereo and correspondence
2
Comparing Windows
For each window, match to closest window on
epipolar line in other image.
(slides O. Camps)
3
Window size

• Effect of window size

• Better results with adaptive window
• T. Kanade and M. Okutomi, A Stereo Matching
  Algorithm with an Adaptive Window Theory and
  Experiment,, Proc. International Conference on
  Robotics and Automation, 1991.
• D. Scharstein and R. Szeliski. Stereo matching
  with nonlinear diffusion. International Journal
  of Computer Vision, 28(2)155-174, July 1998

(S. Seitz)
4
Dynamic Programming (Baker and Binford, 1981)

• Find the minimum-cost path going monotonically
• down and right from the top-left corner of the
• graph to its bottom-right corner.
• Nodes matched feature points (e.g., edge
  points).
• Arcs matched intervals along the epipolar
  lines.
• Arc cost discrepancy between intervals.

5
Dynamic Programming (Baker and Binford, 1981)

• Find the minimum-cost path going monotonically
• down and right from the top-left corner of the
• graph to its bottom-right corner.
• Nodes matched feature points (e.g., edge
  points).
• Arcs matched intervals along the epipolar
  lines.
• Arc cost discrepancy between intervals.

6
Dynamic Programming (Ohta and Kanade, 1985)
Reprinted from Stereo by Intra- and
Intet-Scanline Search, by Y. Ohta and T. Kanade,
IEEE Trans. on Pattern Analysis and
Machine Intelligence, 7(2)139-154 (1985). ? 1985
IEEE.
7
Other constraints

• Smoothness disparity usually doesnt change too
  quickly.
• Unfortunately, this makes the problem 2D again.
• Solved with a host of graph algorithms, Markov
  Random Fields, Belief Propagation, .
• Uniqueness constraint (each feature can at most
  have one match)
• Occlusion and disparity are connected.

8
Feature-based Methods

• Conceptually very similar to Correlation-based
  methods, but
• They only search for correspondences of a sparse
  set of image features.
• Correspondences are given by the most similar
  feature pairs.
• Similarity measure must be adapted to the type of
  feature used.

9
Feature-based Methods

• Features most commonly used
• Corners
• Similarity measured in terms of
• surrounding gray values (SSD, Cross-correlation)
• location
• Edges, Lines
• Similarity measured in terms of
• orientation
• contrast
• coordinates of edge or lines midpoint
• length of line

10
Example Comparing lines

• ll and lr line lengths
• ql and qr line orientations
• (xl,yl) and (xr,yr) midpoints
• cl and cr average contrast along lines
• wl wq wm wc weights controlling influence

The more similar the lines, the larger S is!
11
Correspondence By Features
RIGHT IMAGE

• Search in the right image the disparity (dx, dy)
  is the displacement when the similarity measure
  is maximum

12
Dense Stereo Matching Examples

• View extrapolation results input depth
  image novel view Matthies,Szeliski,Kanade88
  

13
Dense Stereo Matching

• Some other view extrapolation resultsinput
  depth image novel view

14
Dense Stereo Matching

• Compute certainty map from correlations
• input depth map certainty map

15
Ordering constraint

• Usually, order of points in two images is same.
• Is this always true?
• If we match pixel i in image 1 to pixel j in
  image 2, no matches that follow will affect which
  are the best preceding matches.
• Example with pixels (a la Cox et al.).

16
The Ordering Constraint
Points on the epipolar lines appear in the same
order
But it is not always the case ... This enables
dynamic programming
17
Correspondence Problem 2

• Correspondence fail for smooth surfaces
• There is currently no good solution to the
  correspondence problem

18
Correspondence Problem 3

• Regions without texture
• Highly Specular surfaces
• Translucent objects

19
Stereo Vision Outline

• Basic Equations
• Epipolar Geometry
• Image Rectification
• Reconstruction
• Correspondence
• Active Range Imaging Technology
• Dense and Layered Stereo
• Smoothing With Markov Random Fields

20
How can We Improve Stereo?
Space-time stereo scanneruses unstructured light
to aidin correspondence
Result Dense 3D mesh (noisy)
21
Prof Marc Levoy _at_ Stanford

• By James Davis, Honda Research,
• Now UCSC

22
Active Stereo (Structured Light)
23
DP for Correspondence

• Does this always work?
• When would it fail?
• Failure Example 1
• Failure Example 2
• Failure Example 3

24
Stereo Correspondences
Left scanline
Right scanline
25
Stereo Correspondences
Left scanline
Right scanline
26
Search Over Correspondences
Left scanline
Right scanline
Disoccluded Pixels

• Three cases
• Sequential cost of match
• Occluded cost of no match
• Disoccluded cost of no match

27
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
28
Stereo Matching with Dynamic Programming
Left scanline
Start

• Dynamic programming yields the optimal path
  through grid. This is the best set of matches
  that satisfy the ordering constraint

Dis-occluded Pixels
Right scanline
End
29
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal
30
Stereo Matching with Dynamic Programming
Left scanline

• Scan across grid computing optimal cost for
  each node given its upper-left neighbors.Backtrac
  k from the terminal to get the optimal path.

Dis-occluded Pixels
Right scanline
Terminal