Optical Flow

1
Optical Flow
Jana Kosecka George Mason University
2
Image Primitives and Correspondence
Given an image point in left image, what is the
(corresponding) point in the right image, which
is the projection of the same 3-D point
3
Image Primitives and Correspondence
Difficulties ambiguities, large changes of
appearance, due to change Of viewpoint,
non-uniqueness
4
Matching - Correspondence
radiance
Lambertian assumption
Rigid body motion
Correspondence
5
Local Deformation Models

• Translational model
• Affine model
• Transformation of the intensity values taking
  into account occlusions
• and noise

6
Feature Tracking and Optical Flow

• Translational model

• Small baseline

• RHS approximation by the first two terms of
  Taylor series

• Brightness constancy constraint

7
Aperture Problem

• Normal flow

Given brightness constancy constraint at single
point all we can recover is normal flow
8
Optical Flow

• Integrate around over image patch

• Solve

9
Optical Flow, Feature Tracking
Conceptually
rank(G) 0 blank wall problem rank(G) 1
aperture problem rank(G) 2 enough texture
good feature candidates
In reality choice of threshold is involved
10
Optical Flow

• Previous method - assumption locally constant
  flow

• Alternative regularization techniques (locally
  smooth flow fields,
• integration along contours)

• Qualitative properties of the motion fields

11
Recall Again - Point Feature Extraction

• Compute eigenvalues of G
• If smalest eigenvalue ? of G is bigger than ? -
  mark pixel as candidate
• feature point

• Alternatively feature quality function (Harris
  Corner Detector)

12
Feature Selection

• Compute Image Gradient
• Compute Feature Quality measure for each
  pixel
• Search for local maxima

Feature Quality Function
Local maxima of feature quality function
13
Feature Tracking

• Translational motion model
• Closed form solution
• Build an image pyramid
• Start from coarsest level
• Estimate the displacement at the coarsest level
• Iterate until finest level

14
Coarse to fine feature tracking
0
1
2

1. compute
2. warp the window in the second image by
3. update the displacement
4. go to finer level
5. At the finest level repeat for several
   iterations

15
Affine Feature Tracking

• Translational model
• Affine model (Shi Tomasi Good Features to
  track)

• Small baseline

• RHS approximation by the first two terms of
  Taylor series

16
Affine feature tracking

Intensity offset
Contrast change
17
Tracked Features
18
Structure and Motion Recovery from Video
1. Use multiple image stream to compute the
information about camera motion and 3D structure
of the scene 2. Tracking image features over time
Tracked Features
Original sequence
19
Structure and Motion Recovery from Video
Computed model 3D coordinates of the feature
points
Original picture
20
Multiple Motion Segmentation

• A dynamic scene multiple 3D motion models

• A static scene multiple 2D motion models

• Given an image sequence, determine
• Number of motion models (affine, Euclidean, etc.)
• Motion model affine (2D) or Euclidean (3D)
• Segmentation model to which each pixel belongs

21
f test image g reference image f aligned image
S. Periaswamy and H. Farid Differential Affine
Motion Estimation for Medical Registration, SPIE
2000
22
Stereo - Dynamic Programming (Ohta and Kanade,
1985)
Reprinted from Stereo by Intra- and
Intet-Scanline Search, by Y. Ohta and T. Kanade,
IEEE Trans. on Pattern Analysis and
Machine Intelligence, 7(2)139-154 (1985). ? 1985
IEEE.
23
General Matching - Example - NCC score for two
widely separated views
NCC score
24
Wide baseline matching

• Images are represented by their salient image
  regions - features (SIFT, Harris-Affine, MSER)

• Each region is characterized by the SIFT
  descriptor, a 128 dimension vector
• SIFT descriptor robust with respect to the
  variations in viewpoint, illumination and scale

25
Image Matching - Location Recognition

• Given all detected features in the query view
• Find the nearest and second nearest match in
  the database
• Keep discriminative and repetitive matches

• Number of matched features indicates similarity
  between images
• From matches to correspondences

26
Localization by means of Location/Building
recognition
27
3D scan matching
28

• Detect Salient Features in 3D range maps
• Compute associated descriptors for them
• Spin Images
• Match / Align overlapping 3D scans
• 3D texture mapped model

29
Image rectification

• Given general displacement how to warp the views
• Such that epipolar lines are parallel to each
  other
• How to warp it back to canonical configuration
• (more details later)

(Seitz)
30
Epipolar rectification

• Rectified Image Pair
• Corresponding epipolar lines are aligned with
  the scan-lines
• Search for dense correspondence is a 1D search

31
Epipolar rectification
Rectified Image Pair
32
Texture mapping