Tal Nir

1
Over-Parameterized Variational Optical Flow

• Tal Nir
• Alfred M. Bruckstein
• Ron Kimmel

Technion, Israel institute of technology Haifa
32000 ISRAEL
2
What is optic flow?

• Optic flow relates to the perception of motion.
• Optic flow the apparent motion of objects in
  the scene as seen on the 2D image plane.

3
An image
4
Warped image
5
The corresponding optical flow
6
Applications of optic flow

• An important pre-processing for many visual tasks
• Tracking.
• Segmentation.
• Compression.
• Super-resolution requires high accuracy.
• 3D reconstruction (structure from motion).

7
Basic equations
Brightness constancy equation
u,v are the optic flow components between frame t
and t1
Linearized brightness constancy equation
8
The aperture problem
Only the flow component in the gradient direction
can be determined (normal flow).
From an algebraic point of view this is an
ill-posed problem An image with N pixels N
equations with 2N unknowns.
9
Going around the aperture problem

• Looking for locations where the image has
• Multiple gradient directions,
• Discontinuous first image derivatives,
• Corners.

10
The Lucas-Kanade method
B. D. Lucas and T. Kanade, An iterative image
registration technique with an application to
stereo vision, Proc. DARPA Image Understanding
Workshop, April, 1981.
11
Lucas-Kanade continued
Solve the linear 2x2 system of equations

• The aperture problem can occur in certain
  regions (zero eigenvalue).
• Typically, the aperture problem does not appear
  in an exact sense.
• Method may yield a sparse flow field estimate.

12
Neighborhood based methods

• The flow in the patch can be described by a
  constant, affine, or other model.
• M. Irani, B. Rousso, S. Peleg, Recovery of
  Ego-Motion Using Region Alignment . IEEE Trans.
  on Pattern Analysis and Machine Intelligence
  (PAMI), Vol. 19, No. 3, pp. 268--272, March 1997
• The smoothness within the patch is inherently
  enforced.
• Discontinuities of the model within the patch may
  cause inaccuracies.
• The resulting problem is over-constrained.

13
Motion in a patch Over constrained solution
(Lucas-Kanade)
Optical flow estimation an ill posed problem
Our work
Over-parameterized Variational
14
The variational approachB. K. P. Horn and B. G.
Schunck, "Determining optical flow," Artificial
Intelligence, vol. 17, pp. 185--203, 1981.
Find the flow which minimizes the functional
Composed of a data and smoothness
(regularization) term
The resulting Euler-Lagrange equations
15
Variational approach. Cont.

• Dense optical flow field (i.e. a vector at each
  pixel).
• The smoothness (regularization) term enables the
  completion of the flow in locations with
  insufficient information.
• Global solution incorporates all the available
  information.
• The best results are achieved by modern
  variational approaches.

16
T. Brox, A. Bruhn, N. Papenberg, J.
WeickertHigh Accuracy Optical Flow Estimation
Based on a Theory for Warping, ECCV 2004.

• L1 non-linear data term with a gradient
  constancy term

• L1 smoothness term in x,y,t space (3D)

Euler-Lagrange equation for u (G0)
17
Brox et. al. High Accuracy Optical Flow
Estimation. Cont.

• Three loops of iteration
• Outer loop k.
• Inner loop fixed point iteration in order to deal
  with the nonlinearity in ?.
• Gauss-Seidel iterations are used in order to
  solve the resulting sparse linear system of
  equations.

18
Brox et. al. High Accuracy Optical Flow
Estimation. Advantages

• Solution in Multi-scale helps to avoid being
  trapped in local minima large motion (reduction
  factor of 0.95).
• The 3D smoothness term solves the problem in the
  volume in contrast to the 2D (two frames)
  solution.
• The gradient constancy term reduces the
  sensitivity to brightness changes.
• Choosing ? as an approximately L1 function
• In the smoothness term it allows discontinuities
  in the flow field.
• In the data term it reduces the sensitivity to
  outliers.
• The addition of e is for numerical reasons.

19
Results Brox et al.
20
Our motivation
Our motivation stems from the smoothness term
Weighted spatio-temporal gradient
Penalty for changes in the optical flow
Penalty for changes from an optical flow model
21
The proposed over-parameterization model

• Basis functions of the flow model
• Space and time varying coefficients

The optical flow is now estimated via the
coefficients

• The different roles of the coefficients and basis
  functions
• The basis functions are selected a-priori, the
  coefficients are estimated.
• The regularization is applied only to the
  coefficients.

22
Over-parameterization - one frame
Conventional representation
u
u
v

Basis functions


Coefficients
Basis functions

Over-parameterized representation
v
23
Over-parameterized functional
The new regularization term penalizes for changes
in the model parameters.
24
Euler-Lagrange equations
The Euler-Lagange equation for the coefficient Aq
25
Over-Parameterization models Constant motion
model

• This case reduces to the regular variational
  approach of solving directly for u and v.

The number of coefficients is n2
26
Affine over-parameterization model

• Six basis functions

27
Rigid motion over-parameterization model

• The optic flow of a rigid body

is the translation vector divided by the depth (z)
is the rotation vector
28
Rigid motion, cont

• In a seminal paper
• The optical flow calculation is a pre-processing
  followed by motion and structure estimation.
• In our formulation, the rigid motion model is
  used directly in the optical flow estimation
  process.

29
Pure translation over-parameterization model

• Rigid motion with pure translation

Use only the first three coefficients and basis
functions of the general rigid motion model.
30
Numerical scheme

• Multi-resolution necessary to deal with large
  displacements.
• At each resolution, three loops of iterations are
  applied.

We adopt parts of the numerical scheme from T.
Brox, A. Bruhn, N. Papenberg, and J.
Weickert,High Accuracy Optical Flow Estimation
Based on a Theory for Warping, ECCV 2004.to our
over-parameterization model
31
Outer loop k
Euler-Lagrange equations, q1...n
Insert first order Taylor approximation to the
brightness constancy equation
32
Inner loop fixed point iteration l
Solves the nonlinearity of the convex function ?
At each pixel we have n linear equations with n
unknowns the increments of the coefficients - dAi
33
Experimental results
The parameters were set experimentally to the
following values
34
Synthetic piecewise affine flow example
35
Synthetic piecewise affine flow ground truth
36
Results
Our method is better in the AAE by 68
37
Piecewise affine test case
The estimated affine parameters are approximately
piecewise constant
38
Ground truth
Our method - affine model
39
Yosemite without clouds sequence
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
(No Transcript)
46
The End
47
39
35
16
15
48
Yosemite without clouds ground truth
49
Images of the angular error
50
Histogram of the angular error
Our method pure translation model
Brox et. al.
51
Yosemite - Solution of the affine parameters
52
Noise sensitivity results
53
Variational Joint optic-flow Computation and
Video RestorationT. Nir, A.M. Bruckstein, R.
Kimmel

• Errors in the data term appear for two reasons
• Errors in the computed flow.
• Errors in the image data noise, blur,
  interlacing, lossy compression,
• The proposed functional

54
Variational Joint optic-flow Computation and
Video Restoration. Cont.

• Minimization is performed with respect to the
  optical flow u,v and the image sequence I.
• The fidelity term requires that the minimization
  would not deviate too far from the measured
  sequence, thus avoiding trivial solutions.
• If the expected noise is large, a lower choice of
  ? is appropriate, allowing larger deviations from
  the measured sequence.
• For , the sequence is constrained
  to be equal to the measurement, resulting in a
  regular optic flow scheme.

55
Solution strategy

• Iterations between optic flow and denoising.
• Initialization zero optic flow and initial
  sequence.
• Solve for the optic flow.
• Perform denoising.
• Iterate steps 2,3 until convergence.

56
The Denoising step

• For the denoising step we use the discrete
  approximation with bilinear interpolation
• Minimize with respect to I1,I2,I3,I4 and I is
  performed by gradient descent (A,B,C,D are
  constant frozen flow).
• The denoising step performs smoothing along the
  optical flow trajectories.
• Remark Smoothing by total variation is not good
  for optic flow calculation.

57
Office sequence Frame 7
58
Office sequence Frame 8
59
Office sequence Frame 9
60
Office sequence Frame 10
61
Office sequence Optic flow at frame 9
62
Experimental results - Office sequence
63
Office sequence results - Cont.
64
A. Borzi, K. Ito, K. Kunisch Optimal control
formulation for determining optical flow, SIAM
J. Sci. Comp. 24(3), 818-847, (2002)

• Minimize with respect to u,v,I
• Subject to the constraints

65
Comparison with Borzi
66
What is the actual gap between L1 and L2?
67
Summary

• Over-parameterized representation of the optic
  flow introduces better regularization.
• The per pixel model allows the functional
  minimization to decide on the locations of
  discontinuities in the higher dimensional space.
• Significant improvement for both the 2D and 3D
  cases.
• Coupling with our joint optic flow and denoising
  scheme gives excellent results under heavy noise.
• Future The improved accuracy of the method has
  the potential to improve motion segmentation,
  video compression, super-resolution