Multiple View Geometry

1
Multiple View Geometry

• Marc Pollefeys
• COMP 256

2
Last class
Gaussian pyramid
Laplacian pyramid
Gabor filters
Fourier transform
3
Not last class
Histograms co-occurrence matrix
4
Texture synthesis Zalesny Van Gool 2000
2 analysis iterations
6 analysis iterations
9 analysis iterations
5
View-dependent texture synthesisZalesny Van
Gool 2000
6
Efros Leung 99
non-parametric sampling
Input image
Synthesizing a pixel

• Assuming Markov property, compute P(pN(p))
• Building explicit probability tables infeasible
• Instead, lets search the input image for all
  similar neighborhoods thats our histogram for
  p
• To synthesize p, just pick one match at random

7
Efros Leung 99 extended
non-parametric sampling
Input image

• Observation neighbor pixels are highly correlated

8
block
Input texture
B1
B2
Random placement of blocks
9
Minimal error boundary
overlapping blocks
vertical boundary
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
Why do we see more flowers in the distance?
Leung Malik CVPR97
Perpendicular textures
14
Shape-from-texture
15
Tentative class schedule
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources Shadows Color
Feb 6/8 Linear filters edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
16
THE GEOMETRY OF MULTIPLE VIEWS

• Epipolar Geometry
• The Essential Matrix
• The Fundamental Matrix
• The Trifocal Tensor
• The Quadrifocal Tensor

Reading Chapter 10.
17
Epipolar Geometry

• Epipolar Plane

• Baseline

• Epipoles

• Epipolar Lines

18
Epipolar Constraint

• Potential matches for p have to lie on the
  corresponding
• epipolar line l.

• Potential matches for p have to lie on the
  corresponding
• epipolar line l.

19
Epipolar Constraint Calibrated Case
Essential Matrix (Longuet-Higgins, 1981)
20
Properties of the Essential Matrix
T

• E p is the epipolar line associated with p.
• ETp is the epipolar line associated with p.
• E e0 and ETe0.
• E is singular.
• E has two equal non-zero singular values
• (Huang and Faugeras, 1989).

T
21
Epipolar Constraint Small Motions
To First-Order
Pure translation Focus of Expansion
22
Epipolar Constraint Uncalibrated Case
Fundamental Matrix (Faugeras and Luong, 1992)
23
Properties of the Fundamental Matrix

• F p is the epipolar line associated with p.
• FT p is the epipolar line associated with p.
• F e0 and FT e0.
• F is singular.

T
T
24
The Eight-Point Algorithm (Longuet-Higgins, 1981)
25
Non-Linear Least-Squares Approach (Luong et al.,
1993)
Minimize
with respect to the coefficients of F , using an
appropriate rank-2 parameterization.
26
Problem with eight-point algorithm
linear least-squares unit norm vector F
yielding smallest residual What happens when
there is noise?
27
The Normalized Eight-Point Algorithm (Hartley,
1995)

• Center the image data at the origin, and scale
  it so the
• mean squared distance between the origin and the
  data
• points is 2 pixels q T p , q T p.
• Use the eight-point algorithm to compute F from
  the
• points q and q .
• Enforce the rank-2 constraint.
• Output T F T.

i
i
i
i
i
i
T
28
Epipolar geometry example
29
Example converging cameras
courtesy of Andrew Zisserman
30
Example motion parallel with image plane
(simple for stereo ? rectification)
courtesy of Andrew Zisserman
31
Example forward motion
e
e
courtesy of Andrew Zisserman
32
Fundamental matrix for pure translation
auto-epipolar
courtesy of Andrew Zisserman
33
Fundamental matrix for pure translation
courtesy of Andrew Zisserman
34
Trinocular Epipolar Constraints
These constraints are not independent!
35
Trinocular Epipolar Constraints Transfer
Given p and p , p can be computed as the
solution of linear equations.
1
2
3
36
Trinocular Epipolar Constraints Transfer

• problem for epipolar transfer in trifocal plane!

There must be more to trifocal geometry
image from Hartley and Zisserman
37
Trifocal Constraints
38
Trifocal Constraints
Calibrated Case
All 3x3 minors must be zero!
Trifocal Tensor
39
Trifocal Constraints
Uncalibrated Case
Trifocal Tensor
40
Trifocal Constraints 3 Points
Pick any two lines l and l through p and p .
2
3
2
3
Do it again.
41
Properties of the Trifocal Tensor
T
i

• For any matching epipolar lines, l G l
  0.
• The matrices G are singular.
• They satisfy 8 independent constraints in the
• uncalibrated case (Faugeras and Mourrain, 1995).

2
1
3
i
1
Estimating the Trifocal Tensor

• Ignore the non-linear constraints and use linear
  least-squares
• Impose the constraints a posteriori.

42
T
i
For any matching epipolar lines, l G l
0.
2
1
3
The backprojections of the two lines do not
define a line!
43
Trifocal Tensor Example
108 putative matches
18 outliers
(26 samples)
95 final inliers
88 inliers
(0.19)
(0.43) (0.23)
courtesy of Andrew Zisserman
44
Trifocal Tensor Example
additional line matches
images courtesy of Andrew Zisserman
45
Transfer trifocal transfer
(using tensor notation)
doesnt work if lepipolar line
image courtesy of Hartley and Zisserman
46
Image warping using T(1,2,N)
(Avidan and Shashua 97)
47
Multiple Views (Faugeras and Mourrain, 1995)
48
Two Views
Epipolar Constraint
49
Three Views
Trifocal Constraint
50
Four Views
Quadrifocal Constraint (Triggs, 1995)
51
Geometrically, the four rays must intersect in P..
52
Quadrifocal Tensor and Lines
53
Quadrifocal tensor

• determinant is multilinear
• thus linear in coefficients of lines
  !
• There must exist a tensor with 81 coefficients
  containing all possible combination of x,y,w
  coefficients for all 4 images the quadrifocal
  tensor

54
Scale-Restraint Condition from Photogrammetry
55
from perspective to omnidirectional cameras
3 constraints allow to reconstruct 3D point
perspective camera (2 constraints / feature)
more constraints also tell something about cameras
l(y,-x)
(x,y)
(0,0)
multilinear constraints known as epipolar,
trifocal and quadrifocal constraints
radial camera (uncalibrated) (1 constraints /
feature)
56
Radial quadrifocal tensor
(x,y)

• Linearly compute radial quadrifocal tensor Qijkl
  from 15 pts in 4 views
• Reconstruct 3D scene and use it for calibration

(2x2x2x2 tensor)
Not easy for real data, hard to avoid degenerate
cases (e.g. 3 optical axes intersect in single
point). However, degenerate case leads to
simpler 3 view algorithm for pure rotation

• Radial trifocal tensor Tijk from 7 points in 3
  views
• Reconstruct 2D panorama and use it for
  calibration

(2x2x2 tensor)
57
Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV05)

• Models fish-eye lenses, cata-dioptric systems,
  etc.

angle
normalized radius
58
Non-parametric distortion calibration
(Thirthala and Pollefeys, ICCV05)

• Models fish-eye lenses, cata-dioptric systems,
  etc.

90o
angle
normalized radius
59
Next classStereo
image I(x,y)
image I(x,y)
Disparity map D(x,y)
(x,y)(xD(x,y),y)
FP Chapter 11