Title: Structure%20from%20Motion
1Structure from Motion
- (With a digression on SVD)
2Structure from Motion
- For now, static scene and moving camera
- Equivalently, rigidly moving scene andstatic
camera - Limiting case of stereo with many cameras
- Limiting case of multiview camera calibration
with unknown target - Given n points and N camera positions, have 2nN
equations and 3n6N unknowns
3Approaches
- Obtaining point correspondences
- Optical flow
- Stereo methods correlation, feature matching
- Solving for points and camera motion
- Nonlinear minimization (bundle adjustment)
- Various approximations
4Orthographic Approximation
- Simplest SFM case camera approximated by
orthographic projection
Perspective
Orthographic
5Weak Perspective
- An orthographic assumption is sometimes well
approximated by a telephoto lens
Weak Perspective
6Consequences ofOrthographic Projection
- Scene can be recovered up to scale
- Translation perpendicular to image planecan
never be recovered
7Orthographic Structure from Motion
- Method due to Tomasi Kanade, 1992
- Assume n points in space p1 pn
- Observed at N points in time at image coordinates
(xij, yij) - Feature tracking, optical flow, etc.
8Orthographic Structure from Motion
- Write down matrix of data
Points ?
Frames ?
9Orthographic Structure from Motion
- Step 1 find translation
- Translation parallel to viewingdirection can not
be obtained - Translation perpendicular to viewing direction
equals motion of average position of all points
10Orthographic Structure from Motion
- Subtract average of each row
11Orthographic Structure from Motion
- Step 2 try to find rotation
- Rotation at each frame defines local coordinate
axes , , and - Then
12Orthographic Structure from Motion
- So, can write where R is a
rotation matrix and S is a shape matrix
13Orthographic Structure from Motion
- Goal is to factor
- Before we do, observe that rank( ) 3(in
ideal case with no noise) - Proof
- Rank of R is 3 unless no rotation
- Rank of S is 3 iff have noncoplanar points
- Product of 2 matrices of rank 3 has rank 3
- With noise, rank( ) might be gt 3
14DigressionSingular Value Decomposition (SVD)
- Handy mathematical technique that has application
to many problems - Given any m?n matrix A, algorithm to find
matrices U, V, and W such that - A U W VT
- U is m?n and orthonormal
- V is n?n and orthonormal
- W is n?n and diagonal
15SVD
- Treat as black box code widely available
(svd(A,0) in Matlab)
16SVD
- The wi are called the singular values of A
- If A is singular, some of the wi will be 0
- In general rank(A) number of nonzero wi
- SVD is mostly unique (up to permutation of
singular values, or if some wi are equal)
17SVD and Inverses
- Why is SVD so useful?
- Application 1 inverses
- A-1(VT)-1 W-1 U-1 V W-1 UT
- This fails when some wi are 0
- Its supposed to fail singular matrix
- Pseudoinverse if wi0, set 1/wi to 0 (!)
- Closest matrix to inverse
- Defined for all (even non-square) matrices
18SVD and Least Squares
- Solving Axb by least squares
- xpseudoinverse(A) times b
- Compute pseudoinverse using SVD
- Lets you see if data is singular
- Even if not singular, ratio of max to min
singular values (condition number) tells you how
stable the solution will be - Set 1/wi to 0 if wi is small (even if not exactly
0)
19SVD and Eigenvectors
- Let AUWVT, and let xi be ith column of V
- Consider ATA xi
- So elements of W are squared eigenvalues and
columns of V are eigenvectors of ATA
20SVD and Eigenvectors
- Eigenvectors of ATA turned up in finding
least-squares solution to Ax0 - Solution is eigenvector corresponding to smallest
eigenvalue
21SVD and Matrix Similarity
- One common definition for the norm of a matrix is
the Frobenius norm - Frobenius norm can be computed from SVD
- So changes to a matrix can be evaluated by
looking at changes to singular values
22SVD and Matrix Similarity
- Suppose you want to find best rank-k
approximation to A - Answer set all but the largest k singular values
to zero - Can form compact representation by eliminating
columns of U and V corresponding to zeroed wi
23SVD and PCA
- Principal Components Analysis (PCA)
approximating a high-dimensional data set with a
lower-dimensional subspace
Original axes
24SVD and PCA
- Data matrix with points as rows, take SVD
- Columns of Vk are principal components
- Value of wi gives importance of each component
25SVD and Orthogonalization
- The matrix U is the closest orthonormal matrix
to A - Yet another useful application of the
matrix-approximation properties of SVD - Much more stable numerically thanGraham-Schmidt
orthogonalization - Find rotation given general affine matrix
26End of Digression
- Goal is to factor into R and S
- Lets apply SVD!
- But should have rank 3 ?all but 3 of the wi
should be 0 - Extract the top 3 wi, together with the
corresponding columns of U and V
27Factoring for Orthographic Structure from Motion
- After extracting columns, U3 has dimensions 2N?3
(just what we wanted for R) - W3V3T has dimensions 3?n (just what we wanted for
S) - So, let RU3, SW3V3T
28Affine Structure from Motion
- The i and j entries of R are not, in general,
unit length and perpendicular - We have found motion (and therefore shape)up to
an affine transformation - This is the best we could do if we didntassume
orthographic camera
29Ensuring Orthogonality
- Since can be factored as R S, it can also
be factored as (RQ)(Q-1S), for any Q - So, search for Q such that R R Q has the
properties we want
30Ensuring Orthogonality
- Want or
- Let T QQT
- Equations for elements of T solve byleast
squares - Ambiguity add constraints
31Ensuring Orthogonality
- Have found T QQT
- Find Q by taking square root of T
- Cholesky decomposition if T is positive definite
- General algorithms (e.g. sqrtm in Matlab)
32Orthogonal Structure from Motion
- Lets recap
- Write down matrix of observations
- Find translation from avg. position
- Subtract translation
- Factor matrix using SVD
- Write down equations for orthogonalization
- Solve using least squares, square root
- At end, get matrix R R Q of camera
positionsand matrix S Q-1S of 3D points
33Results
Tomasi Kanade
34Results
Tomasi Kanade
35Results
Front view
Top view
Tomasi Kanade