ROBOT VISION Lesson 6: Shape from Stereo Matthias R

1
ROBOT VISION Lesson 6 Shape from
StereoMatthias RütherSlides partial courtesy
of Marc Pollefeys Department of Computer
ScienceUniversity of North Carolina, Chapel Hill
2
Content

• Two View Geometry
• Epipolar Geometry
• 3D reconstruction
• Computing F
• Point Correspondences
• Interest Points
• Matching

3
Epipolar Geometry
C, C, x, x and X are coplanar
4
Three Questions

1. Correspondence geometry Given an image point x
   in the first view, how does this constrain the
   position of the corresponding point x in the
   second image?

• (ii) Camera geometry (motion) Given a set of
  corresponding image points xi ?xi, i1,,n,
  what are the cameras P and P for the two views?

• (iii) Scene geometry (structure) Given
  corresponding image points xi ?xi and cameras
  P, P, what is the position of (their pre-image)
  X in space?

5
Epipolar Geometry
What if only C, C, x are known?
6
Epipolar Geometry
All points on p project on l and l
7
Epipolar Geometry
Family of planes p and lines l and l
Intersection in e and e
8
Epipolar Geometry
epipoles e,e intersection of baseline with
image plane projection of projection center in
other image vanishing point of camera motion
direction
an epipolar plane plane containing baseline
(1-D family)
an epipolar line intersection of epipolar plane
with image (always come in corresponding pairs)
9
Example Converging Cameras
10
Example motion parallel with image plane
11
Example forward motion
e
e
12
The Fundamental Matrix (F)
algebraic representation of epipolar geometry
we will see that mapping is (singular)
correlation (i.e. projective mapping from points
to lines) represented by the fundamental matrix F
13
The Fundamental Matrix (F)
geometric derivation
mapping from 2-D to 1-D family (rank 2)
14
The Fundamental Matrix (F)
algebraic derivation
(note doesnt work for CC ? F0)
15
The Fundamental Matrix (F)
correspondence condition
The fundamental matrix satisfies the condition
that for any pair of corresponding points x?x in
the two images
16
The Fundamental Matrix (F)
F is the unique 3x3 rank 2 matrix that satisfies
xTFx0 for all x?x

1. Transpose if F is fundamental matrix for (P,P),
   then FT is fundamental matrix for (P,P)
2. Epipolar lines lFx lFTx
3. Epipoles on all epipolar lines, thus eTFx0, ?x
   ?eTF0, similarly Fe0
4. F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)
5. F is a correlation, projective mapping from a
   point x to a line lFx (not a proper
   correlation, i.e. not invertible)

17
Epipolar Line Geometry
l,l epipolar lines, k line not through e ?
lFkxl and symmetrically lFTkxl
(pick ke, since eTe?0)
18
Invariance under projective transformation
Derivation based purely on projective concepts
F invariant to transformations of projective
3-space
unique
not unique
canonical form
19
Canonical cameras given F
F matrix corresponds to P,P iff PTFP is
skew-symmetric
20
The Essential Matrix
fundamental matrix for calibrated cameras
(remove K)
5 d.o.f. (3 for R 2 for t up to scale)
E is essential matrix if and only if two singular
values are equal (and third0)
21
Reconstruction from E four possible solutions
(only one solution where points is in front of
both cameras)
22
Epipolar Geometry
Underlying structure in set of matches for rigid
scenes

1. Computable from corresponding points
2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration

23
3D reconstruction of cameras and structure
reconstruction problem
given xi?xi , compute P,P and Xi
for all i
without additional information possible up to
projective ambiguity
24
Outline of reconstruction

1. Compute F from correspondences
2. Compute camera matrices from F
3. Compute 3D point for each pair of corresponding
   points

computation of F use xiFxi0 equations, linear
in coeff. F 8 points (linear), 7 points
(non-linear), 8 (least-squares)
computation of camera matrices use
triangulation compute intersection of two
backprojected rays
25
Reconstruction ambiguity similarity
26
Reconstruction ambiguity projective
27
Terminology
xi?xi Original scene Xi Projective, affine,
similarity reconstruction reconstruction that
is identical to original up to projective,
affine, similarity transformation Literature
Metric and Euclidean reconstruction
similarity reconstruction
28
The projective reconstruction theorem
If a set of point correspondences in two views
determine the fundamental matrix uniquely, then
the scene and cameras may be reconstructed from
these correspondences alone, and any two such
reconstructions from these correspondences are
projectively equivalent

• along same ray of P2, idem for P2

two possibilities X2iHX1i, or points along
baseline
key result allows reconstruction from pair of
uncalibrated images
29
(No Transcript)
30
Stratified reconstruction

1. Projective reconstruction
2. Affine reconstruction
3. Metric reconstruction

31
Direct metric reconstruction using ground truth
use control points XEi with known coordinates to
go from projective to metric
32
Epipolar Geometry computation of F
Underlying structure in set of matches for rigid
scenes

1. Computable from corresponding points
2. Simplifies matching
3. Allows to detect wrong matches
4. Related to calibration

33
Computation of F basic equation
separate known from unknown
(data)
(unknowns)
(linear)
34
Imposing the singularity constraint
SVD from linearly computed F matrix (rank 3)
Compute closest rank-2 approximation
35
(No Transcript)
36
The NOT normalized 8-point algorithm
37
The normalized 8-point algorithm

• Transform image to -1,1x-1,1

Least squares yields good results (Hartley,
PAMI97)
38
The Gold Standard Algorithm
Maximum Likelihood Estimation
( least-squares for Gaussian noise)
Initialize normalized 8-point, (P,P) from F,
reconstruct Xi
Parameterize
(overparametrized)
Minimize cost using Levenberg-Marquardt (preferabl
y sparse LM, see book)
39
Examples
40
Examples
41
Examples
42
Recommendations

1. Do not use unnormalized algorithms

• Quick and easy to implement 8-point normalized

• Better enforce rank-2 constraint during
  minimization

• Best Maximum Likelihood Estimation (minimal
  parameterization, sparse implementation)

43
The correspondence problem feature points

• Extract feature points to relate images
• Required properties
• Well-defined
• (i.e. neigboring points should all be
  different)
• Stable across views

(i.e. same 3D point should be extracted as
feature for neighboring viewpoints)
44
Feature points
(e.g.HarrisStephens88 ShiTomasi94)
Find points that differ as much as possible from
all neighboring points
homogeneous
edge
corner
M should have large eigenvalues
Feature local maxima (subpixel) of F(?1, ? 2)
45
Feature points

• Select strongest features (e.g. 1000/image)

46
Feature matching

• Evaluate NCC for all features with
• similar coordinates

Keep mutual best matches Still many wrong matches!
47
Similarity Example
0.96 -0.40 -0.16 -0.39 0.19
-0.05 0.75 -0.47 0.51 0.72
-0.18 -0.39 0.73 0.15 -0.75
-0.27 0.49 0.16 0.79 0.21
0.08 0.50 -0.45 0.28 0.99
Gives satisfying results for small image motions
48
RANSAC

• Step 1. Extract features
• Step 2. Compute a set of potential matches
• Step 3. do
• Step 3.1 select minimal sample (i.e. 8 matches)
• Step 3.2 compute F
• Step 3.3 determine inliers
• until ?(inliers,samples)lt95

Step 4. Compute F based on all inliers Step 5.
Look for additional matches Step 6. Refine F
based on all correct matches
inliers 90 80 70 60 50
samples 5 13 35 106 382
49
Finding more matches
restrict search range to neighborhood of
epipolar line (?1.5 pixels) relax disparity
restriction (along epipolar line)
50
Degenerate Cases

• Degenerate cases
• Planar scene
• Pure rotation
• No unique solution
• Remaining DOF filled by noise
• Use simpler model (e.g. homography)
• Model selection (Torr et al., ICCV98, Kanatani,
  Akaike)
• Compare H and F according to expected residual
  error (compensate for model complexity)

51
More problems

• Absence of sufficient features (no texture)
• Repeated structure ambiguity

• Robust matcher also finds
• support for wrong hypothesis
• solution detect repetition

(Schaffalitzky and Zisserman, BMVC98)