Tracking and Modeling Non-Rigid Objects with Rank Constraints

1
Tracking and Modeling Non-Rigid Objects with Rank
Constraints

• L. Torresani, D. Yang, E. Alexander, and C.
  Bregler

Presentation by Jeremy Weinsier
2
Motivation

• Can a non-rigid objects shape and pose be
  determined from a single video sequence without
  any information about the particular scene?
• In other words, can we determine a 3D model for
  the shape of the object without any prior
  knowledge?

3
Background

• In a previous paper, the authors proved that 2D
  point tracks from a single view (no stereo) are
  enough to recover the non-rigid motion as well as
  the structure of an object by exploiting low rank
  constraints.
• This procedure takes that idea one step further
  non-rigid motion and structure can now be
  recovered from a single view without any point
  tracks.

4
Rank Constraints

• Optical flow data is read into two F x P
  matrices, U and V.
• Let
• If W describes 3D rigid motion, then there is an
  upper bound on the rank of W.

5
Rank Constraints

• The particular bound is based on the assumed
  motion model. For example,
• Orthographic r4
• Projective r8
• These rank constraints are derived from the fact
  that W can be factored into Q and S, which
  represent pose and shape.

6
Rank Constraints

• Non-rigid motion can also be factored into two
  matrices, but the rank of these matrices will be
  higher than that those in the rigid case.
• Shape will be represented by K basis shapes, each
  described by a 3xP matrix.

7
Rank Constraints

• Assuming weak perspective projection

8
Rank Constraints

• Eliminate T by subtracting the mean of all 2D
  points and rewrite as matrix multiplication

9
Rank Constraints

• Combine all individual equations into one large
  matrix

10
Rank Constraints

• Since Q is size 2Fx3K and B is size 3KxP, the
  rank of W, r3K.
• This assumes that the sequence is free of noise
  and can be used as the bound on the rank of W in
  the case of non-rigid motion.

11
Basis Flow

• Assuming rank r, each column of W can be modeled
  as a linear combination of r basis-tracks,
  denoted Q.
• Q is estimated by removing all but the most
  reliable tracks in W and then computing its SVD.
  The first r eigenvectors of the SVD are taken as
  the first estimate of the basis-tracks.
• This eigenbase is then applied to the entire W
  matrix to estimate all P tracks.

12
Basis Flow

• The Lucas Kanade optical flow equation

13
Basis Flow

• Since the entire sequence is assumed to have a
  single image template, this equation could be
  rewritten as
• Where C,D,E are PxP diagonal matrices containing
  c,d,e values for all patches and F,G are FxP
  matrices containing f and g values for each patch
  and frame.

14
Basis Flow

• Now split Q into two matrices, one containing the
  even rows of Q and the other containing the odd
  rows of Q. Since Q is a basis for W

15
Basis Flow

• The determined values are used as initial
  estimates of the actual values.
• The image is warped according to these values and
  the process iterates to refine these values until
  they are close to the actual values.

16
Occlusion

• By vectorizing the B matrix into a Pr dimensional
  vector b, occlusion can be handled
• Missing entries due to occlusion or mistracking
  are removed from the matrices on both sides.

17
Occlusion

• If enough points are still visible, the system is
  still overconstrained and can be solved.
• The displacement of the missing points can later
  be estimated using QB.

18
3D Reconstruction

• The factorization of W into Q and B is not
  unique. Other factorizations can be found by
• Q does not initially comply with its necessary
  structure

19
3D Reconstruction

• Tomasi-Kanade suggest using a linear
  approximation scheme in the rigid case. The
  sub-blocks of Q are treated as rotation matrices.
• A similar approach can be used in the non-rigid
  case, but a second factorization step is
  necessary.

20
Sub-Block Factorization
21
Sub-Block Factorization

• Since Qt now has a rank of one, a non-linear
  optimization method is used to find an invertible
  A that orthonormalizes all of the sub-blocks.
• This produces a matrix with scaled rotation
  matrices as its sub-blocks.

22
Limitations

• The limitation of sub-block factorization is in
  the noisy and ambiguous cases. The second and
  higher eigenvalues will not disappear in some of
  the sub-blocks. This leads to bad rank-1
  approximations of Rt.
• The alternative is an iterative technique that
  solves the system directly.

23
Iterative Optimization

• The first step in this method is the same as in
  sub-block factorization. R is factored into Qrig
  and Brig. Qrig is then reorganized into a matrix
  with sub-blocks that are weak perspective
  rotation matrices.

24
Iterative Optimization

• Qrig is then used as an initial guess of the pose
  of the non-rigid object.
• Linear least squares is used to find B.
• Linear least squares is used to find L.
• Solve for R, such that Rt fit the equation

25
Iterative Optimization

• To solve for R such that Rt remain rotation
  matrices, Rt can be parameterized with
  exponential coordinates.

26
Iterative Optimization

• Linearizing the previous equation around the
  previous estimate leads to
• These steps are iterated until convergence.
  Missing entries are handled as before.

27
Multi-View Input

• If M cameras are used, the input matrix W is
  enlarged to size 2FW x P.

28
Multi-View Input

• The process is the same as before, but now there
  is an additional constraint that the deformation
  must be the same for every camera in every frame.

29
Basis Shapes

• The number of basis shapes used has an effect on
  the error in the result. Here is some sample K
  vs. error data

30
Basis Shapes

• The number of necessary basis shapes is not
  currently an automatic calculation.
• A simple solution to this problem is to
  continuously increase K until the error is below
  a certain threshold.

31
Results