Camera calibration

1
Camera calibration

• Digital Visual Effects
• Yung-Yu Chuang

with slides by Richard Szeliski, Steve Seitz,,
Fred Pighin and Marc Pollefyes
2
Outline

• Camera projection models
• Camera calibration
• Nonlinear least square methods
• A camera calibration tool
• Applications

3
Camera projection models
4
Pinhole camera
5
Pinhole camera model
(X,Y,Z)
P
origin
p
(x,y)
principal point
(optical center)
6
Pinhole camera model
principal point
7
Pinhole camera model
principal point
8
Principal point offset
principal point
intrinsic matrix
only related to camera projection
9
Intrinsic matrix
Is this form of K good enough?

• non-square pixels (digital video)
• skew
• radial distortion

10
Distortion
No distortion
Pin cushion
Barrel

• Radial distortion of the image
• Caused by imperfect lenses
• Deviations are most noticeable for rays that pass
  through the edge of the lens

11
Camera rotation and translation
extrinsic matrix
12
Two kinds of parameters

• internal or intrinsic parameters such as focal
  length, optical center, aspect ratiowhat kind
  of camera?
• external or extrinsic (pose) parameters including
  rotation and translationwhere is the camera?

13
Other projection models
14
Orthographic projection

• Special case of perspective projection
• Distance from the COP to the PP is infinite
• Also called parallel projection (x, y, z) ?
  (x, y)

Image
World
15
Other types of projections

• Scaled orthographic
• Also called weak perspective
• Affine projection
• Also called paraperspective

16
Illusion
17
Illusion
18
Fun with perspective
19
Perspective cues
20
Perspective cues
21
Fun with perspective
Ames room
Ames video
BBC story
22
Forced perspective in LOTR
23
Camera calibration
24
Camera calibration

• Estimate both intrinsic and extrinsic parameters.
  Two main categories
• Photometric calibration uses reference objects
  with known geometry
• Self calibration only assumes static scene, e.g.
  structure from motion

25
Camera calibration approaches

• linear regression (least squares)
• nonlinear optimization

26
Chromaglyphs (HP research)
27
Camera calibration
28
Linear regression
29
Linear regression

• Directly estimate 11 unknowns in the M matrix
  using known 3D points (Xi,Yi,Zi) and measured
  feature positions (ui,vi)

30
Linear regression
31
Linear regression
32
Linear regression
Solve for Projection Matrix M using least-square
techniques
33
Normal equation

• Given an overdetermined system

the normal equation is that which minimizes the
sum of the square differences between left and
right sides
34
Linear regression

• Advantages
• All specifics of the camera summarized in one
  matrix
• Can predict where any world point will map to in
  the image
• Disadvantages
• Doesnt tell us about particular parameters
• Mixes up internal and external parameters
• pose specific move the camera and everything
  breaks
• More unknowns than true degrees of freedom

35
Nonlinear optimization

• A probabilistic view of least square
• Feature measurement equations
• Probability of M given (ui,vi)

P
36
Optimal estimation

• Likelihood of M given (ui,vi)
• It is a least square problem (but not necessarily
  linear least square)
• How do we minimize L?

P
L
37
Optimal estimation

• Non-linear regression (least squares), because
  the relations between ûi and ui are non-linear
  functions of M
• We can use Levenberg-Marquardt method to minimize
  it

unknown parameters
known constant
38
Nonlinear least square methods
39
Least square fitting
number of data points
number of parameters
40
Linear least square fitting
y
t
41
Linear least square fitting
y
t
42
Linear least square fitting
y
model
parameters
t
43
Linear least square fitting
y
model
parameters
t
44
Linear least square fitting
y
model
parameters
t
45
Nonlinear least square fitting
model
parameters
residuals
46
Function minimization
Least square is related to function minimization.

• It is very hard to solve in general. Here, we
  only consider a simpler problem of finding local
  minimum.

47
Function minimization
48
Quadratic functions
Approximate the function with a quadratic
function within a small neighborhood
49
Quadratic functions
A is positive definite. All eigenvalues are
positive. For all x, xTAxgt0.
negative definite
A is singular
A is indefinite
50
Function minimization

• Why?
• By definition, if is a local minimizer,

is small enough
51
Function minimization
52
Function minimization
53
Descent methods
54
Descent direction
55
Steepest descent method
the decrease of F(x) per unit along h direction
?
hsd is a descent direction because hTsd F(x)
-F(x)2 lt0
56
Line search
57
Line search
58
Steepest descent method
isocontour
gradient
59
Steepest descent method

• It has good performance in the initial stage of
  the iterative process. Converge very slow with a
  linear rate.

60
Newtons method
?
?
?
?
61
Newtons method

• Another view
• Minimizer satisfies

62
Newtons method

• It requires solving a linear system and H is not
  always positive definite.
• It has good performance in the final stage of the
  iterative process, where x is close to x.

63
Gauss-Newton method

• Use the approximate Hessian
• No need for second derivative
• H is positive semi-definite

64
Hybrid method

• This needs to calculate second-order derivative
  which might not be available.

65
Levenberg-Marquardt method

• LM can be thought of as a combination of steepest
  descent and the Newton method. When the current
  solution is far from the correct one, the
  algorithm behaves like a steepest descent method
  slow, but guaranteed to converge. When the
  current solution is close to the correct
  solution, it becomes a Newtons method.

66
Nonlinear least square
67
Levenberg-Marquardt method
68
Levenberg-Marquardt method

• µ0 ? Newtons method
• µ?8 ? steepest descent method
• Strategy for choosing µ
• Start with some small µ
• If F is not reduced, keep trying larger µ until
  it does
• If F is reduced, accept it and reduce µ for the
  next iteration

69
Recap (the Rosenbrock function)
Global minimum at (1,1)
70
Steepest descent
71
(No Transcript)
72
(No Transcript)
73
In the plane of the steepest descent direction
74
Steepest descent (1000 iterations)
75
Gauss-Newton method

• With the approximate Hessian
• No need for second derivative
• H is positive semi-definite

76
(No Transcript)
77
Newtons method (48 evaluations)
78
Levenberg-Marquardt

• Blends steepest descent and Gauss-Newton
• At each step, solve for the descent direction h
• If µ large, , steepest descent
• If µ small, ,
  Gauss-Newton

79
Levenberg-Marquardt (90 evaluations)
80
A popular calibration tool
81
Multi-plane calibration

Images courtesy Jean-Yves Bouguet, Intel Corp.

• Advantage
• Only requires a plane
• Dont have to know positions/orientations
• Good code available online!
• Intels OpenCV library http//www.intel.com/rese
  arch/mrl/research/opencv/
• Matlab version by Jean-Yves Bouget
  http//www.vision.caltech.edu/bouguetj/calib_doc/i
  ndex.html
• Zhengyou Zhangs web site http//research.micros
  oft.com/zhang/Calib/

82
Step 1 data acquisition
83
Step 2 specify corner order
84
Step 3 corner extraction
85
Step 3 corner extraction
86
Step 4 minimize projection error
87
Step 4 camera calibration
88
Step 4 camera calibration
89
Step 5 refinement
90
Optimized parameters
91
Applications
92
How is calibration used?

• Good for recovering intrinsic parameters It is
  thus useful for many vision applications
• Since it requires a calibration pattern, it is
  often necessary to remove or replace the pattern
  from the footage or utilize it in some ways

93
Example of calibration
94
Example of calibration
95
Example of calibration

• Videos from GaTech
• DasTatoo, MakeOf
• P!NG, MakeOf
• Work, MakeOf
• LifeInPaints, MakeOf

96
PhotoBook
PhotoBook
MakeOf