Phase Correlation

1
Phase Correlation
2
Phase Correlation
Take cross correlation
Take inverse Fourier transform
? Location of the impulse function gives the
translation amount between the images
3
Phase Correlation
4
Computer Vision

• Stereo Vision

5
Coordinate Systems

• Let O be the origin of a 3D coordinate system
  spanned by the unit vectors i, j, and k
  orthogonal to each other.

i
P
O
k
j
Coordinate vector
6
Homogeneous Coordinates
n
H
P
O
Homogeneous coordinates
7
Coordinate System Changes

• Translation

8
Coordinate System Changes

• Rotation

where
Exercise Write the rotation matrix for a 2D
coordinate system.
9
Coordinate System Changes

• Rotation Translation

10
Perspective Projection

• Perspective projection equations

11
Review Pinhole Camera
12
Review Perspective Projection
13
Multi-View Geometry
Relates

• Camera Orientations

• Camera Parameters

14
Stereo
scene point
p
p
image plane
optical center
15
Finding Correspondences
p
p
16
Three Questions

• Correspondence geometry Given an image point p
  in the first view, how does this constrain the
  position of the corresponding point p in the
  second?
• Camera geometry (motion) Given a set of
  corresponding image points pi ? pi, i1,,n,
  what are the cameras C and C for the two views?
  Or what is the geometric transformation between
  the views?
• Scene geometry (structure) Given corresponding
  image points pi ? pi and cameras C, C, what is
  the position of the point X in space?

17
Stereo Constraints
M
Image plane
Y1
p
O1
Z1
X1
Focal plane
18
Epipolar Constraint
19
From Geometry to Algebra
All vectors shown lie on the same plane.
20
From Geometry to Algebra
21
Matrix form of cross product
aaxiayjazk
ababsin(?)u
bbxibyjbzk
22
The Essential Matrix
Essential matrix
23
Stereo Vision

• Two cameras.
• Known camera positions.
• Recover depth.

24
Recovering Depth Information
P
Q
P1
P2Q2
Q1
O2
O1
Depth can be recovered with two images and
triangulation.
25
A Simple Stereo System
LEFT CAMERA
RIGHT CAMERA
baseline
Right image target
Left image reference
Zw0
26
Stereo View
Right View
Left View
Disparity
27
Stereo Disparity

• The separation between two matching objects is
  called the stereo disparity.

28
Parallel Cameras
P
Z
xl
xr
pl
f
pr
Ol
Or
Disparity
T
T is the stereo baseline
29
Disparity Equation
P(X,Y,Z)
Stereo system with parallel optical axes
Depth
T Baseline
30
Disparity vs. Baseline
P(X,Y,Z)
Stereo system with parallel optical axes
Depth
Disparity
T Baseline
31
Finding Correspondences
32
Correlation Approach
LEFT IMAGE

• For Each point (xl, yl) in the left image, define
  a window centered at the point

33
Correlation Approach
RIGHT IMAGE
(xl, yl)

• search its corresponding point within a search
  region in the right image

34
Correlation Approach
RIGHT IMAGE
(xl, yl)
dx
(xr, yr)

• the disparity (dx, dy) is the displacement when
  the correlation is maximum

35
Stereo correspondence

• Epipolar Constraint
• Reduces correspondence problem to 1D search along
  epipolar lines

36
Stereo correspondence

• Compare with every pixel on same epipolar line in
  right image
• Pick pixel with the minimum matching error

37
Comparing Windows
For each window, match to closest window on
epipolar line in other image.
38
Comparing Windows
Minimize
Sum of Squared Differences
Maximize
Cross correlation
39
Feature-based correspondence

• Features most commonly used
• Corners
• Similarity measured in terms of
• surrounding gray values (SSD, Cross-correlation)
• location
• Edges, Lines
• Similarity measured in terms of
• orientation
• contrast
• coordinates of edge or lines midpoint
• length of line

40
Feature-based Approach
LEFT IMAGE

• For each feature in the left image

41
Feature-based Approach
RIGHT IMAGE

• Search in the right image the disparity (dx, dy)
  is the displacement when the similarity measure
  is maximum

42
Correspondence Difficulties

• Why is the correspondence problem difficult?
• Some points in each image will have no
  corresponding points in the other image.
• (1) the cameras might have different fields of
  view.
• (2) due to occlusion.
• A stereo system must be able to determine the
  image parts that should not be matched.

43
Structured Light

• Structured lighting
• Feature-based methods are not applicable when the
  objects have smooth surfaces (i.e., sparse
  disparity maps make surface reconstruction
  difficult).
• Patterns of light are projected onto the surface
  of objects, creating interesting points even in
  regions which would be otherwise smooth.

• Finding and matching such points is simplified by
  knowing the geometry of the projected patterns.

44
Stereo results

• Data from University of Tsukuba

Ground truth
Scene
(Seitz)
45
Results with window correlation
Estimated depth of field (a fixed-size window)
Ground truth
(Seitz)
46
Results with better method
A state of the art method Boykov et al., Fast
Approximate Energy Minimization via Graph Cuts,
International Conference on Computer Vision,
September 1999.
Ground truth
(Seitz)
47
Window size

• Effect of window size

• Better results with adaptive window
• T. Kanade and M. Okutomi, A Stereo Matching
  Algorithm with an Adaptive Window Theory and
  Experiment,, Proc. International Conference on
  Robotics and Automation, 1991.
• D. Scharstein and R. Szeliski. Stereo matching
  with nonlinear diffusion. International Journal
  of Computer Vision, 28(2)155-174, July 1998

(Seitz)
48
Other constraints

• It is possible to put some constraints.
• For example smoothness. (Disparity usually
  doesnt change too quickly.)

49
Parameters of a Stereo System

• Intrinsic Parameters
• Characterize the transformation from camera to
  pixel coordinate systems of each camera
• Focal length, image center, aspect ratio
• Extrinsic parameters
• Describe the relative position and orientation of
  the two cameras
• Rotation matrix R and translation vector T

50
Applications
First-down line
courtesy of Sportvision
51
Applications
Virtual advertising
courtesy of Princeton Video Image