Recent progress in optical flow

1
Recent progress in optical flow
progress
Presented by Darya Frolova and Denis Simakov
2
Optical Flow is not in favor
Very popular slide
Often not using Optical Flow is stated as one of
the main advantages of a method
Optical Flow methods have a reputation of either
unreliable or slow
Recent works claim
Optical Flow can be computed fast and accurately
3
Optical Flow Research Timeline
HornSchunck
LucasKanade
1981
1992
1998
now
BenchmarkGalvin et.al.
BenchmarkBarron et.al.
Seminal papers
A slow and not very consistent improvement in
results, but a lot of useful ingredients were
developed
4
In This Lecture
We will describe

• Ingredients for an accurate and robust optical
  flow
  
• How people combine these ingredients
• Fast algorithms

Papers

• Combining the advantages of local and global
  optic flow methods (Lucas/Kanade meets
  Horn/Schunck)
  
• A. Bruhn, J. Weickert, C. Schnörr,
  2002 - 2005
  
• High accuracy optical flow estimation based on a
  theory for warping
  
• T. Brox, A. Bruhn, N. Papenberg, J.
  Weickert, 2004 - 2005
  
• Real-Time Optic Flow Computation with Variational
  Methods
  
• A. Bruhn, J. Weickert, C. Feddern, T.
  Kohlberger, C. Schnörr, 2003 - 2005
  
• Towards ultimate motion estimation Combining
  highest accuracy with real-time performance A.
  Bruhn, J. Weickert, 2005
  
• Bilateral filtering-based optical flow estimation
  with occlusion detection J.Xiao, H.Cheng,
  H.Sawhney, C.Rao, M.Isnardi, 2006

5
What is Optical Flow?

6
Definitions
The optical flow is a velocity field in the image
which transforms one image into the next image in
a sequence HornSchunck


frame 2
frame 1
flow field
The motion field is the projection into the
image of three-dimensional motion vectors
HornSchunck
7
Ambiguity of optical flow
Frame 1
8
Applications
optical flow

• video compression
• 3D reconstruction
• segmentation
• object detection
• activity detection
• key frame extraction
• interpolation in time

motion field
We are usually interested in actual motion
9
Outline

• Ingredients for an accurate and robust optical
  flow
  
• Local image constraints on motion
• Robust statistics
• Spatial coherence
• How people combine these ingredients
• Fast algorithms

10
Local image constraints
11
Brightness Constancy
u
frame t1
v
frame t
12
Linearized brightness constancy
Deviation from brightness constancy (we want it
to be zero)
Linearize
13
Linearized brightness constancy
Let us square the difference
J motion tensor, or structure tensor
14
Averaged linearized constraint
J is a function of x, y (a matrix for every
point)
Combine over small neighborhoods (more robust to
noise)

J

15
Method of LucasKanade

• Solve independently for each point
  LucasKanade 1981

linear system
Can be solved for every point where matrix is not
degenerate
16
LukasKanade - Results
Rubik cube
Hamburg taxi
flow field
flow field
17
Brightness is not always constant
Rotating cylinder
Brightness constancy does not always hold
Gradient constancy holds
intensity
intensity derivative
position
position
18
Local constraints - Summary
We have seen
linearized

• brightness constancy

averaged linearized
averaged linearized

• gradient constancy

19
Local constraints are not enough!
20
Local constraints work poorly
Optical flow direction using only local
constraints
input video
color encodes direction as marked on the boundary
21
Where local constraints fail
Uniform regions
Motion is not observable in the image (locally)
22
Where local constraints fail
Aperture problem We can estimate only one flow
component (normal)
23
Where local constraints fail
Occlusions
We have not seen where some points moved
Occluded regions are marked in red
24
Obtaining support from neighbors

• Two main problems with local constraints
• information about motion is missing in some
  points need spatial coherency
  
• constraints do not hold everywhere need
  methods to combine them robustly

good
missing
wrong
25
Robust combination of partially reliable data

• or How to hold elections

26
Toy example
Find best representative for the set of numbers
xi
27
Elections and robust statistics
many ordinary people
a very rich man
wealth
Votes proportional to the wealth
One vote per person
like in L1 norm minimization
like in L2 norm minimization
28
Combination of two flow constraints
A. Bruhn, J. Weickert, 2005 Towards ultimate
motion estimation Combining highest accuracy
with real-time performance
29
Spatial Propagation
30
Obtaining support from neighbors

• Two main problems with local constraints
• information about motion is missing in some
  points need spatial coherency
  
• constraints do not hold everywhere need
  methods to combine them robustly

good
missing
wrong
31
Homogeneous propagation
This constraint is not correct on motion
boundaries over-smoothing of the resulting
flow
HornSchunck 1981
32
Robustness to flow discontinuities
e
(also known as isotropic flow-driven
regularization)
T. Brox, A. Bruhn, N. Papenberg, J. Weickert,
2004 High accuracy optical flow estimation based
on a theory for warping
33
Selective flow filtering

• We want to propagate information
• without crossing image and flow discontinuities
• from good points only (not occluded)

Solution use bilateral
filter in space, intensity, flow
taking into account
occlusions
J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi,
2006 Bilateral filtering-based optical flow esti
mation with occlusion detection
34
Bilateral filter
Unilateral (usual)
Bilateral
x
Preserves discontinuities!
C. Tomasi, R. Manduchi, 1998 Bilateral
filtering for gray and color images.
35
Using of bilateral filter - Example
cyan rectangle moves to the right and occludes
background region marked by red
J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi,
2006 Bilateral filtering-based optical flow esti
mation with occlusion detection
36
Learning of spatial coherence

• Come to the next lecture

37
Spatial coherence Summary
Homogeneous propagation - oversmoothing
Robust statistics with homogeneous propagation -
preserves flow discontinuities
Bilateral filtering - combines information from
regions with similar flow and similar
intensities
Handles occlusions
38
Two more useful ingredients
in brief one slide each
39
2D vs. 3D
Several frames allow more accurate optical flow
estimation
2 frames
Several frames
40
Multiscale Optical Flow
Linearization valid only for small flow
pyramid for frame 1
pyramid for frame 2
frame 1warped
?

upsample

(other names warping, coarse-to-fine,
multiresolution)
41
Methods

• How to make tasty soup with these ingredients
  several recipes

42
Outline

• Ingredients for an accurate and robust optical
  flow
  
• How people combine these ingredients
• Lukas Kanade meet Horn Schunck
• The more ingredients the better
• Bilateral filtering and occlusions
• Fast algorithms

43
Combining ingredients

• Spatial coherency
• Homogeneous
• Flow-driven
• Bilateral filtering occlusions

• Local constraints
• Brightness constancy
• Image gradient constancy

Energy ?? (Data) ?? (Smoothness)
Combined using robust statistics
Computed coarse-to-fine
Use several frames
44
Combining Local and Global
Remember
LucasKanade
HornSchunk
Basic Combining local and global
A. Bruhn, J. Weickert, C. Schnörr, 2002
45
Sensitivity to noise quantitative results
frame t1
Error measure angle between true and computed
flow in (x,y,t) space
frame t
ground truth flow
46
The more ingredients - the better
brightness constancy
spatial coherence
gradient constancy
Bruhn, Weickert, 2005Towards ultimate motion
estimation Combining highest accuracy with
real-time performance
47
Quantitative results
Angular error
Method
Yosemite sequence with clouds
Average error decreases, but standard deviation
is still high.
48
Influence of each ingredient
For Yosemite sequence with clouds
49
Handling occlusions
bilateral filtering of flowpreserve intensity
and flow discontinuities
model occlusions
J.Xiao, H.Cheng, H.Sawhney, C.Rao, M.Isnardi,
ECCV 2006 Bilateral filtering-based optical flo
w estimation with occlusion detection
50
Qualitative results
51
Quantitative results
Angular error
Method
Yosemite sequence with clouds
52
Outline

• Ingredients for an accurate and robust optical
  flow
  
• How people combine these ingredients
• Fast algorithms
• Energy functional discrete equation
• Multigrid solver nearly real-time

53
How to minimize energy
Analogy
Necessary condition
Necessary condition
Euler-Lagrange equation
54
An example
Let us see how to derive discretized equation for
1D Horn Schuhck
HornSchunk
1D version (simplified)
55
Iterative minimization (simple example)
Euler-Lagrange
Linear system of equation for u
Discretized
Local iterations
56
Life is not a picnic
Linear discretized system
Non-linear in u, non-linear discretized system
Even more complicated
57
Optimization algorithms

• Simple iterative minimization
• Multigrid much faster convergence

58
Solving the system
How to solve?
Start with some initial guess

and apply some iterative method

• fast convergence
• good initial guess

2 components of success
59
Relaxation smoothes the error
Relaxation schemes have smoothing property
It may take thousands of iterations to propagate
information to large distance
Only neighboring pixels are coupled in relaxation
scheme
60
Relaxation smoothes the error Examples
1D case
2D case
Error of initial guess
Error after 5 relaxation
Error after 15 relaxations
61
Idea coarser grid
initial grid fine grid
On a coarser grid low frequencies become higher
Hence, relaxations can be more effective
coarse grid we take every second point
62
Multigrid 2-Level V-Cycle
5. Correct the previous solution
6. Iterate ? remove interpolation artifacts
1. Iterate ? error becomes smooth
2. Transfer error equation to the coarse level ?
low frequencies become high
4. Transfer error to the fine level
3. Solve for the error on the coarse level ? good
error estimation
63
Coarse grid - advantages
Coarsening allows

• make iteration process faster (on the coarse
  grid we can effectively minimize the error)
  
• obtain better initial guess (solve
  directly on the coarsest grid)

go to the coarsest grid
interpolate to the finer grid
solve here the equation to find
64
Multigrid approach Full scheme
65
Non-linear Full Approximation Scheme
A() non-linear
Difference from the linear case
Equation for error involves current solution u0
? Need to transfer current solution to the
coarser level
66
Multigrid Summary

• Used to solve linear or non-linear
  equations
  
• Method combine two techniques
• Basic iterative solver quickly removes high
  frequencies of the error
  
• Coarsening makes low frequencies high
• Contribution fast minimization of loosely
  coupled equations

67
Fast Optimization Results
Time sec
frames/sec
HornSchunck
CLG
Towards ultimate
image size 160 x 120
68
Summary of the Talk

• 25 years of Optical Flow a lot of useful
  ingredients were developed
  
• local constraints
• brightness constancy
• gradient constancy
• smoothing techniques
• homogeneous
• flow-driven (preserving discontinuities)
• bilateral filters
• handling of occlusions
• robust functions
• multiscale
• All ingredients are combined an a global Energy
  Minimization approach
  
• This difficult global optimization can be done
  very fast using Multigrid

69
Thank you!