Accurate Non-Iterative O(n) Solution to the PnP Problem

1
Accurate Non-Iterative O(n) Solution to the PnP
Problem
Francesc Moreno-Noguer Vincent Lepetit Pascal Fua

• CVLab - Ecole Polytechnique Fédérale de Lausanne

2
The Perspective-n-Point (PnP) Problem

• Given 2D/3D correspondences Camera internal
  parameters
• We want Compute camera pose w.r.t. 3D points
• How to take advantage of the internal parameters?

2D/3D correspondences known
World Coordinate System
Internal parameters known
3
Old Problem with many References
1940 1970 1980
1990 2000 2007
4
Iterative vs. Non-Iterative Methods
Scherighofer Pinz 2006
Iterative
Ansar Daniilidis 2003
Lowe 1991
Horaud et al 1989
Fiore 2001
Fischler Bolles 1981
Lu et al 2000
Abbel Karara 1971
Triggs 1999
Non-Iterative
Church 1945
Dhome et al 1989
Gao et al 2003
Haralick et al 1991
Kumar Hanson 1994
DeMenthon Davis 1995
Oberkampf et al 1996
Horaud et al 1997
Quan Lan 1999
5
Iterative vs. Non-Iterative Methods
Iterative

• Accurate, but slow
• Risk of local minima

Non-Iterative

• No risk of local minima
• High Complexity O(n ) and larger
• Fast for small n
• Slow for large n
• n unknowns to solve 6 DoF !!!!

5
6
Current PnP Applications small n

• Many applications Comp. Vision, Robotics,
  Augmented Reality
• Example Estimating the 3D position of markers

7
Dealing with Hundreds of Points

• Potential Applications
• Structure from Motion
• SLAM

Williams et al 2007

• Tracking by Detection
• Process hundreds of noisy features
• Real-time

New Image 2D positions
Model Image 3D positions
8
EPnP Efficient PnP

• Fast
• Non-iterative algorithm O(n)
• Handle hundreds of points in real-time
• Accurate
• As accurate as iterative methods
• Robust to noise

9
EPnP Fast and Accurate
Accuracy
non-iterative Clamped DLT (DLT imposing known
intrinsic parameters) iterative Lu, Hager and
Mjolsness, PAMI'00

• As accurate as the iterative solution !!!
• Much faster

10
Introducing Control Points

• 3D points expressed as a weighted sum of 4
  control points

Known

• Unknowns
  Camera coord. of the 4 control points

11
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. The control points coordinates can be expressed
   as a linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Adding rigidity constraints gives quadratic
   equations in the ?i .
6. Solving for the ?i depends on their number
   (linearization or relinearization).

12
Point Reprojections give a Linear System

• For each correspondence

• For all n correspondences

Mx0
where
13
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. The control points coordinates can be expressed
   as a linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Adding rigidity constraints gives quadratic
   equations in the ?i .
6. Solving for the ?i depends on their number
   (linearization or relinearization).

14
Solution is in the Null-Space

• Transform to a system with small and constant
  size

?
Mx 0
15
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. Express the control points coordinates as a
   linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Adding rigidity constraints gives quadratic
   equations in the ?i .
6. Solving for the ?i depends on their number
   (linearization or relinearization).

16
Dimension N of the Null-Space

• We need to solve for N and
• What is the dimension N of the null space of
  ?
• Noise-Free Case N1
• In practice (noise)
• No zero eigenvalues,
• but several are small

17
Dimension N of the Null-Space

• We need to solve for N and

• We fix N to 1, 2, 3 and 4
• Solve for s independently for each case

• Take the best solution ? minimizes reprojection
  error

18
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. Express the control points coordinates as a
   linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Adding rigidity constraints gives quadratic
   equations in the ?i .
6. Solving for the ?i depends on their number
   (linearization or relinearization).

19
Solving for

• Impose Rigidity Constraints
  Distances between control points
  must be preserved

20
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. Express the control points coordinates as a
   linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Add rigidity constraints to get quadratic
   equations in the ?i .
6. Solving for the ?i depends on their number
   (linearization or relinearization).

21
Case N1

• We have

22
Case N1

• We have

And 6 quadratic equations
2

• can be easily computed
• Absolute Value Solution of a linear system

• Sign Chosen to preserve orientation in control
  points

23
Case N2

• We have

And 6 quadratic equations
2

• Linearization technique
• Substitute quadratic by linear terms

• 6 linear equations in , ,

24
Case N3

• We have

And 6 quadratic equations
2

• Same Linearization technique
• 6 Linear Equations for 6 unknowns

25
Case N4

• Six Quadratic Equations in , , and
• The linearization produces 10 products
• Do not have enough equations anymore !!!
• Relinearization
• The are expressed as a linear combination
  of eigenvectors

? Check paper for details
26
EPnP Algorithm Summary

1. The control points coordinates are the (12)
   unknowns.
2. Project the 3D points onto the image ? Build a
   linear system in the control points coordinates.
3. Express the control points coordinates as a
   linear combination of the null eigenvectors.
4. The weights ( the ?i ) are the new unknowns (not
   more than 4).
5. Add rigidity constraints to get quadratic
   equations in the ?i .
6. Solve for the ?i depending on their number
   (linearization or relinearization).

27
From s to Camera Pose
28
Results
Model Image 3D coordinates in world reference
New Image 2D image coordinates

• Detection in every single frame
• No temporal consistency

29
Final Optimization

• Refine Control Points Coordinates

paper

• Gauss Newton
• Independent on the number of input
  correspondences
• Converges in lt 5 iterations

30
Improves Accuracy with No-Cost
Computation Time Matlab
Accuracy

• Gauss Newton Optimization
• Improves the accuracy
• With almost No-Cost

31
Time vs. Accuracy
EPnP
Iterative Lu, Hager and Mjolsness, PAMI'00
Rotation Error ()
Computation Time (sec)
32
Conclusions Future Work

• O(n) non-iterative solution to the PnP problem
• As accurate as iterative methods
• Much faster
• Simple central idea
• Represent 3D coordinates as a weighted sum of 4
  control points
• Potentially applicable to other problems
• Structure from Motion
• Deformable surfaces

33
Thanks !!
Code Available http//cvlab.epfl.ch/fmoreno/Pub
lications/ICCV07