Camera calibration

1
Camera calibration

• Digital Visual Effects, Spring 2006
• Yung-Yu Chuang
• 2005/4/19

with slides by Richard Szeliski, Steve Seitz, and
Marc Pollefyes
2
Announcements

• Artifacts for assignment 1 voting
• http//www.csie.ntu.edu.tw/cyy/vfx/assignments/p
  roj1/artifacts/index.html

3
Outline

• Camera projection models
• Camera calibration
• Nonlinear least square methods
• Bundle adjustment

4
Camera projection models
5
Pinhole camera
6
Pinhole camera model
origin
principal point
(optical center)
7
Pinhole camera model
8
Pinhole camera model
9
Principal point offset
intrinsic matrix
10
Intrinsic matrix
Is this form of K good enough?

• non-square pixels (digital video)
• skew
• radial distortion

11
Radial distortion
12
Camera rotation and translation
extrinsic matrix
13
Two kinds of parameters

• internal or intrinsic parameters such as focal
  length, optical center, aspect ratiowhat kind
  of camera?
• external or extrinsic (pose) parameters including
  rotation and translationwhere is the camera?

14
Other projection models
15
Orthographic projection

• Special case of perspective projection
• Distance from the COP to the PP is infinite
• Also called parallel projection (x, y, z) ?
  (x, y)

Image
World
16
Other types of projections

• Scaled orthographic
• Also called weak perspective
• Affine projection
• Also called paraperspective

17
Fun with perspective
18
Perspective cues
19
Perspective cues
20
Fun with perspective
Ames room
21
Forced perspective in LOTR
22
Camera calibration
23
Camera calibration

• Estimate both intrinsic and extrinsic parameters
• Mainly, two categories
• Photometric calibration uses reference objects
  with known geometry
• Self calibration only assumes static scene, e.g.
  structure from motion

24
Camera calibration approaches

• linear regression (least squares)
• nonlinear optinization
• multiple planar patterns

25
Chromaglyphs (HP research)
26
Linear regression
27
Linear regression

• Directly estimate 11 unknowns in the M matrix
  using known 3D points (Xi,Yi,Zi) and measured
  feature positions (ui,vi)

28
Linear regression
29
Linear regression
30
Linear regression
Solve for Projection Matrix M using least-square
techniques
31
Normal equation

• Given an overdetermined system

the normal equation is that which minimizes the
sum of the square differences between left and
right sides
Why?
32
Normal equation
nxm, n equations, m variables
33
Normal equation
34
Normal equation
35
Linear regression

• Advantages
• All specifics of the camera summarized in one
  matrix
• Can predict where any world point will map to in
  the image
• Disadvantages
• Doesnt tell us about particular parameters
• Mixes up internal and external parameters
• pose specific move the camera and everything
  breaks

36
Nonlinear optimization

• Feature measurement equations
• Likelihood of M given (ui,vi)

37
Optimal estimation

• Log likelihood of M given (ui,vi)
• How do we minimize C?
• Non-linear regression (least squares), because ûi
  and vi are non-linear functions of M
• We can use Levenberg-Marquardt method to minimize
  it

38
Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

• Advantage
• Only requires a plane
• Dont have to know positions/orientations
• Good code available online!
• Intels OpenCV library http//www.intel.com/rese
  arch/mrl/research/opencv/
• Matlab version by Jean-Yves Bouget
  http//www.vision.caltech.edu/bouguetj/calib_doc/i
  ndex.html
• Zhengyou Zhangs web site http//research.micros
  oft.com/zhang/Calib/

39
Step 1 data acquisition
40
Step 2 specify corner order
41
Step 3 corner extraction
42
Step 3 corner extraction
43
Step 4 minimize projection error
44
Step 4 camera calibration
45
Step 4 camera calibration
46
Step 5 refinement
47
Nonlinear least square methods
48
Least square fitting
number of data points
number of parameters
49
Linear least square fitting
y
t
prediction
residual
50
Nonlinear least square fitting
51
Function minimization
Least square is related to function minimization.

• It is very hard to solve in general. Here, we
  only consider a simpler problem of finding local
  minimum.

52
Function minimization
53
Quadratic functions
Approximate the function with a quadratic
function within a small neighborhood
54
Quadratic functions
A is positive definite. All eigenvalues are
positive. Fall all x, xTAxgt0.
negative definite
A is singular
A is indefinite
55
Function minimization
56
Descent methods
57
Descent direction
58
Steepest descent method
the decrease of F(x) per unit along h direction
?
hsd is a descent direction because hTsd F(x)-
F(x)2lt0

• It has good performance in the initial stage of
  the iterative process. Converge very slow with a
  linear rate.

59
Steepest descent method
isocontour
gradient
60
Line search
61
Line search
62
Steepest descent method
63
Newtons method
?
?
?
?

• It has good performance in the final stage of the
  iterative process, where x is close to x.

64
Hybrid method

• This needs to calculate second-order derivative
  which might not be available.

65
Levenberg-Marquardt method

• LM can be thought of as a combination of steepest
  descent and the Newton method. When the current
  solution is far from the correct one, the
  algorithm behaves like a steepest descent method
  slow, but guaranteed to converge. When the
  current solution is close to the correct
  solution, it becomes a Newtons method.

66
Nonlinear least square
67
Levenberg-Marquardt method
68
Levenberg-Marquardt method

• µ0 ? Newtons method
• µ?8 ? steepest descent method
• Strategy for choosing µ
• Start with some small µ
• If F is not reduced, keep trying larger µ until
  it does
• If F is reduced, accept it and reduce µ for the
  next iteration

69
Bundle adjustment
70
Bundle adjustment

• Bundle adjustment (BA) is a technique for
  simultaneously refining the 3D structure and
  camera parameters
• It is capable of obtaining an optimal
  reconstruction under certain assumptions on image
  error models. For zero-mean Gaussian image
  errors, BA is the maximum likelihood estimator.

71
Bundle adjustment

• n 3D points are seen in m views
• xij is the projection of the i-th point on image
  j
• aj is the parameters for the j-th camera
• bi is the parameters for the i-th point
• BA attempts to minimize the projection error

predicted projection
Euclidean distance
72
Bundle adjustment
73
Bundle adjustment

• 3 views and 4 points

74
Typical Jacobian
75
Block structure of normal equation
76
Bundle adjustment
77
Bundle adjustment

• Multiplied by

78
Reference

• Manolis Lourakis and Antonis Argyros, The Design
  and Implementation of a Generic Sparse Bundle
  Adjustment Software Package Based on the
  Levenberg-Marquardt Algorithm, FORTH-ICS/TR-320
  2004.
• K. Madsen, H.B. Nielsen, O. Timgleff, Methods for
  Non-Linear Least Squares Problems, 2004.
• Zhengyou Zhang, A Flexible New Techniques for
  Camera Calibration, MSR-TR-98-71, 1998.
• Bill Triggs, Philip McLauchlan, Richard Hartley
  and Andrew Fitzgibbon, Bundle Adjustment - A
  Modern Symthesis, Proceedings of the
  International Workshop on Vision Algorithms
  Theory and Practice, pp298-372, 1999.