Optical Flow

1
Optical Flow
2
Optical Flow

• Distribution of apparent velocities of movement
  of brightness pattern in an image

3
Optical Flow

• Perspective projection (pinhole camera)
• Instantaneous 2D velocity

4
Definitions
The optical flow is a velocity field in the image
which transforms one image into the next image in
a sequence HornSchunck


frame 2
frame 1
flow field
The motion field is the projection into the
image of three-dimensional motion vectors
HornSchunck
5
Ambiguity of optical flow
Frame 1
6
Determining Optical Flow

• Berthold K.P. Horn and Brian G. Rhunck
• Artificial Intelligence Laboratory, MIT.

7
Optical Flow

• Optical flow field Estimate the 2D motion field,
  which are the 2D velocities for all visible
  points
• Two key problems
• Determine what image property to track
• Determine how to track it

8
Brightness Constancy
u
frame t1
v
frame t
9
Brightness Constancy

• Track points of constant brightness, assuming
  that surface radiance is constant over time
• Brightness constancy is often violated by Mother
  Nature, so the resulting optical flow field is
  sometimes a very poor approximation to the 2D
  motion field

10
Optical Flow

• The optical flow cannot be computed at a point
  independently of neighboring points without
  introducing additional constraints

11
Brightness Constancy Constraint

• The brightness of a particular point in the
  pattern is constant
• Cannot determine the flow velocity
• locally without additional
  constraints

Constraint line
12
Brightness Constancy Constraint

• Normal Velocity
• No constraint when the gradient magnitude is zero
  

13
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
14
Optical Flow Assumptions
Slide from Michael Black, CS143 2003
15
Smoothness Constraint

• Constraint Assumption
• Neighboring points on the objects have similar
  velocities and
• The velocity field of the brightness patterns in
  the image varies smoothly almost everywhere
• Discontinuities in flow can be expected where one
  object occludes another
• Algorithms based on smoothness constraint are
  likely to have difficulties with occluding edges

16
Smoothness Constraint

• Minimize the square of the gradient magnitude of
  the optical flow velocity
• Sum of the squares of Laplacians of x- and y-
  components of the flow

17
Estimate the Partial Derivatives
18
Estimate the Laplacian of Flow Velocities
19
Minimization

• Minimize the sum of errors in rate of change of
  image brightness
• And measure of departure from smoothness
• Total error to be minimized

20
Minimization

• Good (u,v) occurs at

21
Difference of Flow from Local Average

• (u,v) lies in the direction towards the
  constraint line along a line that perpendicular
  to it
• a2 plays significant role when Ex, Ey is small
  roughly equal to the expected noise in the
  estimate of

22
Constrained Minimization

• Minimizing
• While maintaining
  0
• The (u,v) here is just the intersection on the
  constraint line

23
Iterative Solution
24
Optical Flow 1D Case
Brightness Constancy Assumption
25
Tracking in the 1D case
26
Tracking in the 1D case
27
Tracking in the 1D case
Iterating helps refining the velocity vector
Converges in about 5 iterations
28
Algorithm for 1D tracking
29
Iterative Solution

• Very costly to solve these equations
  simultaneously
• Iterative algorithms, such as Gauss-Seidel method

It is interesting to note that the new estimates
at a particular point do not depend directly on
the previous estimate at the same point
30
Fill in Uniform Region

• Uniform region brightness gradient is zero
• No local information to constrain the vecocity
• Fill in uniform region
• Given the values on region boudnary
• Solve the Laplace equation under Dirichlet
  boundary condition

31
Tightness of Constraint

• In general, the solution is most accurately
  determined in the region where the brightness is
  not too small and varies in direction from point
  to point
• Constraint information is available in a relative
  small neighborhood

32
Choice of Iterative Scheme

• How to interlace the iterations with the time
  step?
• Get stable values before going to next frame
• Only one iteration per time-step gives good
  initial guess
• Ability to deal with more images per unit time
• Better estimation in uniform regions because of
  the noise in measurements of images will be
  independent and tend to cancel out

33
Experiment

• Very low resolution video, say, 3232
• A rotating ball with smoothly varying patterns,
  but no shading

34
Results I

• Simple linear translation of the entire pattern
• Estimate between two images
• Showing the velocity
• Few changes occur after 32 iterations when errors
  10

35
Results I
1 time step
4 time steps
36
Results I
16 time steps
64 time steps
37
Results II

• Only one iteration per time-step
• More rapid
• Few changes after 16 iteration when errors 7

38
Results II
1 time steps
4 time steps
39
Results II
16 time steps
64 time steps
40
Results III

• Simple rotation and contraction of the brightness
  pattern

41
Result III
initial
32 time steps
42
Result IV

• Rigid body motion
• Laplacian for one of the velocity components
  becomes infinite on the occluding bound
• Reasonable results are still obtained with finite
  velocities

43
Results IV
Rotating cylinder
Rotating sphere
44
Results IV
Exact flow pattern for a rotating cylinder and
rotating sphere
45
Summary

• A method for computing optical flow from a
  sequence of images
• Two components of flow velocity
• Additional constraint smoothness
• Iterative method

46
Optical Flow Research Timeline
HornSchunck
LucasKanade
1981
1992
1998
now
BenchmarkGalvin et.al.
BenchmarkBarron et.al.
Seminal papers
A slow and not very consistent improvement in
results, but a lot of useful ingredients were
developed
47
http//people.csail.mit.edu/celiu/ECCV2008/zz