Title: Generic Imaging: Calibration and 3D Reconstruction
1Generic Imaging Calibration and 3D Reconstruction
Advancement to Candidacy by Srikumar
Ramalingam Department of Computer
Science University of California, Santa
Cruz srikumar_at_cse.ucsc.edu
2Overview
- Problem Statement
- Introduction and Motivation
- Previous Work
- Current Work
- - Generic Calibration
- - Generic Structure-from-Motion Framework
- Proposed Work
- - Solving the degeneracy cases Axial cameras
- - Towards complete generic calibration
- - Generic self-calibration
- - Generic epipolar curves
- Conclusion
3Problem Statement
1. Black box Camera Calibration
Some exotic camera
Can we calibrate any camera using a black box
calibration algorithm? - transformation from the
3D scene to its corresponding 2D image without
knowing anything about the internals of the
cameras
4Problem Statement
pinhole
Reconstruction
omni
Can we do 3D Reconstruction from any two cameras?
Input Images from any two cameras (pinhole,
stereo, omni, non-central, etc.)
Goals 3D reconstruction of the scene
5Problem Statement
x2
x1
- We propose the concept of generic imaging model
- Calibration algorithm for the generic imaging
model - 3D reconstruction algorithms for generic imaging
model - ..
X2
X1
general imaging model
6Motivation Pinhole model
Physical setup for perspective projection by
Durer 1575
Pinhole camera model
- central model
- closer to human perception
7Motivation Camera is a set of unconstrained rays
Thus we can obtain novel cameras by giving
more degrees of freedom to the rays
8Motivation
Calibration and recovery of the shape of the
cornea
Modifying the resolution characteristics of a
scene
9Motivation Summary
Pinhole spatial resolution, texture Stereo
absolute scale, 3D reconstruction of nonrigid
scenes Omnidirectional - motion estimation, large
field of view, better navigation Non-central -
absolute scale, motion estimation Catadioptric
Modify the resolution characteristics of a scene,
shape of the eye Hybrid scenarios have
advantages Example Scenarios 1) Video
conferencing / Surveillance systems 2) Large
scale modeling
pinhole
Omndirectional (with fisheye lens)
10Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
11Introduction
mirror (hyperboloid)
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Central catadioptric
12Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Central catadioptric
Non-central catadioptric
13Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels, with the associated
rays
Examples
Pinhole
Non-central mosaic
Central catadioptric
Non-central catadioptric
14Introduction
General Camera Model
A pixel watches along one viewing ray
A camera is a set of pixels the associated rays
Examples
Pinhole
Non-central mosaic
Multi- stereo
Central catadioptric
Non-central catadioptric
Etc. e.g. pinhole with radial or any
other distortion
15Introduction Practical systems
16Previous Work
Generic calibration M.D. Grossberg and S.K.
Nayar, A general imaging model and a method for
finding its parameters, ICCV-2001. P. Sturm and
S.Ramalingam, A generic method for camera
calibration, ECCV-2004. S. Ramalingam, S.K. Lodha
and P. Sturm, A generic cross-camera
structure-from-motion analysis,
Omnivis-2004. Cross-camera scenarios C. Geyer and
K. Daniilidis, A unifying theory of central
panoramic systems and practical implications,
ECCV-2000. P.Sturm, Mixing catadioptric and
perspective cameras, OMNIVIS-2002. Jingyi Yu and L
eonard McMillan, General Linear Cameras,
ECCV-2004. Structure-from-Motion R.I.Hartley and
A.Zisserman, Multiple view geometry in computer
vision, 2000. B.Micusik and T.Pajdla, 3D metric
reconstruction from uncalibrated omnidirectional
images, ACCV-2004. H. Bakstein and T. Pajdla, An
overview of non-central cameras, Research Report
CTU-CMP-2000-14. Robert Pless. Using many
cameras as one, CVPR-2003. Bundle adjustment B.
Triggs, P. McLauchlan, R.I. Hartley, and A.
Fitzgibbon, Bundle Adjustment- A modern
synthesis, Workshop on vision algorithms Theory
and Practice, 2000.
17Current Work Calibration in the case of known
motion
18Current Work Calibration in the case of unknown
motion
Estimate motions that make points collinear
camera
19Current Work Calibration in the case of unknown
motion
Q
Q
Q
Estimate motions that make points collinear
R, t
R, t
camera
rank lt 3
20Current Work Trifocal tensor
det 0
a trifocal tensor
21Current Work Trifocal tensor coefficients and
coupled variables
3D general camera with nonplanar calibration
grids
22Current Work Summary for generic calibration
(1) Take images of object in different poses
(2) 2D-3D matching (pixels to points on object)
(3) Estimation of tensors
(4) Extraction of motion parameters from tensors
(5) Computation of projection rays
(6) Bundle adjustment
Variants of the calibration method
3D vs. planar calibration object
general (non-central) vs. central camera model
23Current Work Experiments for generic calibration
Central model applied on pinhole camera with
slight radial distortion
24Current Work Experiments for generic calibration
Homemade non-central cameras
25Current Work Experiments for generic calibration
Central model applied on fisheye
Distortion correction
26Current Work Generic Structure-from-Motion
pipeline
27Current Work Motion estimation for generic
cameras
28Current Work Structure recovery
29Current Work Bundle adjustment
- Ray-point (relatively straightforward)
- We minimize the distance between the 3D point and
the intersecting rays. - Easily extends to non-parametrically calibrated
cameras and non-central cameras. - Reprojection based bundle adjustment
- No parametric association between the pixel and
the ray. - Optimization functions like Levenberg Marquardt
iteration uses the derivatives of reprojection
errors. - Difficult to compute in a non-parametric scenario.
30Current Work Any camera is a cluster of central
rays
Divide all the rays into k central clusters of
rays Eg k1 for pinhole camera, k2 for stereo
camera, kn for an oblique camera, etc.
We intersect a plane to each of the central
cluster
On each central cluster, reproj based bundle
adjustment is applied.
Issues Choice of the plane, clustering issues.
31Current Work Experiments for generic SfM
3D reconstruction after generic calibration for
pinhole images
32Current Work Experiments for generic SfM
algorithm
Cross camera SfM for stereo camera and
omnidirectional camera with fisheye lens
33Current Work Experiments for generic SfM
algorithm
Cross camera SfM for pinhole camera and
omnidirectional camera with fisheye lens
34Current Work Experiments for generic SfM
algorithm
Cross camera SfM for pinhole camera and
omnidirectional camera with fisheye lens
35Current Work Experiments for generic SfM
algorithm
Percentage errors in
the pinhole and omni scenario (house)
Percentage errors in the
stereo and omni scenario (objects)
Difference measures in the
pinhole and omni scenario (stevensons appts, UCSC)
36Proposed Work Solving the degeneracy cases-Axial
cameras
Axial cameras All projection rays intersect a
single ray in space
Examples stereo camera, multicamera setups
with centers along a single line, pushbroom
cameras, cross-slit cameras, catadioptric
cameras with perpective lens and spherical
mirrors, etc
37Proposed Work Solving the degeneracy cases-Axial
cameras
Status of variants Nature of solutions
(unique/degenerate, no of grids)
Axial camera parallel planes
38Proposed Work Towards complete generic
calibration
- Testing of our algorithm on more exotic systems
- Catadioptric cameras with hyperbolic mirror and
perspective lens - Catadioptric cameras with spherical mirror and
perspective lens - Catadioptric camera based on eye
- Multicamera setups
- Pushbroom and other cross-slit cameras
39Proposed Work Towards complete generic
calibration
- Calibration using multiple boards
- Pose estimation of additional grids
- Bundle adjustment to refine the pose of all the
boards - Model selection problem for approximately central
cameras
40Proposed Work Generic self-calibration
Can we calibrate our generic imaging model
without using calibration grids (without any
knowledge of the scene)
Coplanarity constraints in the case of pure
translation
41Proposed Work Generic self-calibration
Pure rotation The ray corresponding to p1
becomes the ray corresponding to p2 . We also
have coplanarity constraints.
- Further goals
- For rotations about axes that do not go through
the optical center - Finally for completely general motions
42Proposed Work Generic epipolar curves
- Pinhole
- Epipolar curve is a line
- One epipole per image
- Hyperbolic mirror with perspective lens
- Epipolar curve can be line, circle, parabola,
conic, etc - Two epipole per image
- Parametric algorithms become very complex for
novel cameras
43Proposed Work Generic epipolar curves
- Epipolar curve computation problem simplifies
itself to a simple ray intersection problem after
the non-parametric generic calibration - Goal Compute f(x1,y1,x2,y2) where
- (x1,y1) and (x2,y2) are corresponding image
points - f can be linear, quadratic or cubic depending on
the nature of the cameras
44Impact
- Simplifies the existing theories for calibration
and 3D reconstruction - Allows us to simultaneously avail the advantages
of several novel cameras - Solves challenging problems like cross-camera
structure-from-motion analysis
Thanks!