Optical Flow

1
Optical Flow

• Marc Pollefeys
• COMP 256

Some slides and illustrations from L. Van Gool,
T. Darell, B. Horn, Y. Weiss, P. Anandan, M.
Black, K. Toyama
2
last week polar rectification
3
Last week polar rectification
4
Stereo matching

• Constraints
• epipolar
• ordering
• uniqueness
• disparity limit
• disparity gradient limit
• Trade-off
• Matching cost (data)
• Discontinuities (prior)

(Cox et al. CVGIP96 Koch96 Falkenhagen97
Van Meerbergen,Vergauwen,Pollefeys,VanGool
IJCV02)
5
Hierarchical stereo matching
Allows faster computation Deals with large
disparity ranges
Downsampling (Gaussian pyramid)
Disparity propagation
(Falkenhagen97Van Meerbergen,Vergauwen,Pollefeys
,VanGool IJCV02)
6
Disparity map
image I(x,y)
image I(x,y)
Disparity map D(x,y)
(x,y)(xD(x,y),y)
7
Example reconstruct image from neighboring images
8
Multi-view depth fusion
(Koch, Pollefeys and Van Gool. ECCV98)

• Compute depth for every pixel of reference image
• Triangulation
• Use multiple views
• Up- and down sequence
• Use Kalman filter

Allows to compute robust texture
9
Real-time stereo on graphics hardware
Yang and Pollefeys CVPR03

• Computes Sum-of-Square-Differences
• Hardware mip-map generation used to aggregate
  results over support region
• Trade-off between small and large support window

140M disparity hypothesis/sec on Radeon
9700pro e.g. 512x512x20disparities at 30Hz
10
Sample Re-Projections
near
far
11
Combine multiple aggregation windows using
hardware mipmap and multiple texture units in
single pass
(1x12x2 4x48x8)
(1x1)
(1x12x2 4x48x8 16x16)
(1x12x2)
video
12
Cool ideas

• Space-time stereo
• (varying illumination, not shape)

13
More on stereo
The Middleburry Stereo Vision Research
Page http//cat.middlebury.edu/stereo/
Recommended reading D. Scharstein and R. Szeliski. A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms.IJCV 47(1/2/3)7-42, April-June 2002. PDF file (1.15 MB) - includes current evaluation.Microsoft Research Technical Report MSR-TR-2001-81, November 2001. PDF file (1.27 MB).
14
Tentative class schedule
Jan 16/18 - Introduction
Jan 23/25 Cameras Radiometry
Jan 30/Feb1 Sources Shadows Color
Feb 6/8 Linear filters edges Texture
Feb 13/15 Multi-View Geometry Stereo
Feb 20/22 Optical flow Project proposals
Feb27/Mar1 Affine SfM Projective SfM
Mar 6/8 Camera Calibration Silhouettes and Photoconsistency
Mar 13/15 Springbreak Springbreak
Mar 20/22 Segmentation Fitting
Mar 27/29 Prob. Segmentation Project Update
Apr 3/5 Tracking Tracking
Apr 10/12 Object Recognition Object Recognition
Apr 17/19 Range data Range data
Apr 24/26 Final project Final project
15
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

16
Optical FlowWhere do pixels move to?
17
(No Transcript)
18
(No Transcript)
19
Motion is a basic cue
Even impoverished motion data can elicit a strong
percept
20
Applications

• tracking
• structure from motion
• motion segmentation
• stabilization
• compression
• Mosaicing

21
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

22
Definition of optical flow
OPTICAL FLOW apparent motion of
brightness patterns
Ideally, the optical flow is the projection of
the three-dimensional velocity vectors on the
image
?
23
Caution required !
Two examples
1. Uniform, rotating sphere ? O.F. 0
2. No motion, but changing lighting
? O.F. ? 0
?
24
Caution required !
25
Mathematical formulation
I (x,y,t) brightness at (x,y) at time t
Brightness constancy assumption
Optical flow constraint equation
?
26
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

27
The aperture problem
1 equation in 2 unknowns
?
28
The aperture problem
0
29
(No Transcript)
30
The aperture problem
31
Remarks
32
Apparently an aperture problem
?
33
What is Optic Flow, anyway?

• Estimate of observed projected motion field
• Not always well defined!
• Compare
• Motion Field (or Scene Flow)
• projection of 3-D motion field
• Normal Flow
• observed tangent motion
• Optic Flow
• apparent motion of the brightness pattern
• (hopefully equal to motion field)
• Consider Barber pole illusion

34
(No Transcript)
35
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

36
Horn Schunck algorithm
Additional smoothness constraint
besides OF constraint equation term
minimize es?ec
?
37
Horn Schunck
The Euler-Lagrange equations
In our case ,
?
38
Horn Schunck
Remarks
1. Coupled PDEs solved using iterative
methods and finite differences
2. More than two frames allow a better
estimation of It
3. Information spreads from corner-type
patterns
?
39
?
40
Horn Schunck, remarks
1. Errors at boundaries
2. Example of regularisation (selection
principle for the solution of illposed
problems)
?
41
Results of an enhanced system
42
Structure from motion with OF
43
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

44

45
(No Transcript)
46
(No Transcript)
47
(No Transcript)
48
(No Transcript)
49
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

50
(No Transcript)
51
(No Transcript)
52
(No Transcript)
53
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

54
(No Transcript)
55
(No Transcript)
56
(No Transcript)
57
(No Transcript)
58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
(No Transcript)
66
(No Transcript)
67
(No Transcript)
68
(No Transcript)
69
(No Transcript)
70
(No Transcript)
71
(No Transcript)
72
(No Transcript)
73
(No Transcript)
74
(No Transcript)
75
(No Transcript)
76
(No Transcript)
77
(No Transcript)
78
(No Transcript)
79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
(No Transcript)
83
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

84
(No Transcript)
85
(No Transcript)
86
(No Transcript)
87
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

88
(No Transcript)
89
(No Transcript)
90
(No Transcript)
91
(No Transcript)
92
(No Transcript)
93
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

94
(No Transcript)
95
(No Transcript)
96
(No Transcript)
97
(No Transcript)
98
(No Transcript)
99
(No Transcript)
100
Optical Flow

• Brightness Constancy
• The Aperture problem
• Regularization
• Lucas-Kanade
• Coarse-to-fine
• Parametric motion models
• Direct depth
• SSD tracking
• Robust flow
• Bayesian flow

101
(No Transcript)
102
Rhombus Displays
http//www.cs.huji.ac.il/yweiss/Rhombus/
103
(No Transcript)
104
(No Transcript)
105
(No Transcript)
106
(No Transcript)
107
(No Transcript)
108
(No Transcript)