Registration for Augmented Reality

1
Registration for Augmented Reality

• Neil Birkbeck
• 3/27/2006

2
Outline

• What is AR?
• Applications
• Research Areas
• Single Plane-Based Calibration
• Multiple Plane-Based Calibration
• Other Techniques
• Summary

3
Augmented Reality (AR)

• Definition (according to Azuma)
• A system which combines virtual objects with the
  real world in real-time. The virtual objects
  should be correctly registered with the real
  world.
• Display Methods
• Head Mounted Display (HMD)
• Video HMD
• Optical HMD
• Monitor/TV

4
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

5
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

http//www.informationinplace.com/Solutions/CaseSt
udies/case_RDECOM/Demos/Blast/lBlast.html
6
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

http//jama.ama-assn.org/cgi/content/full/292/18/2
214-b/JLD40609F1
7
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

http//hci.rsc.rockwell.com/AutomationFair_2003/
8
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

http//wearables.unisa.edu.au/projects/ARQuake/www
/
9
AR Applications

• Advertising
• Military
• Surgical/Medical
• Maintenance
• Entertainment
• Commerce

http//virtual.vtt.fi/multimedia/clipvf.html
10
Research Areas

• Display Technologies
• improving the wearable HMD devices
• Mobile Computing
• Calibration/Registration
• hardware
• GPS, magnetic tracking
• vision-based/range-based
• hybrid methods
• Rendering Issues
• Calibration of illumination for correct/realistic
  shading

11
Vision-Based Registration

• Motivation
• inexpensive/ubiquitous
• video HMD already uses images
• Accurate
• pixel/sub-pixel precision possible
• Images useful for recovering shading/occlusions
• Potential Downsides
• computational requirements
• feasibility over wide-ranges

12
Vision-Based Registration

• Image features are used to register position of
  viewer (camera)
• Existing methods can be categorized using the
  following characteristics
• Feature type
• natural - corners, lines, image patches.
• synthetic patterns, LEDs
• 3D Position of features
• known the 3D positions of the features are
  known
• unknown the 3D positions of features are
  unknown (e.g., similar to SFM)

13
Vision-Based Registration Overview

• When 3D positions of features are known,
  Xi(xi,yi,zi), find camera parameters that align
  the projection of the 3D feature points with the
  observed 2D feature points, xi(ui,vi). This is
  an optimization problem over the space of camera
  parameters p
• argminp ?if(p,Xi)-xi2

14
The Planar CaseA registration method for
features on a plane

• A specific implementation for features on a
  single plane using the typical perspective
  projection model.
• Why planes?
• Easy to make planar calibration patterns
• Measurement of relative 2D positions of feature
  points on planes is straightforward.
• Occur frequently on man-made structures
• Rooftops, building walls, etc.

15
Zhangs Planar Calibration Method

• Calibrate the camera using correspondences
  between (at least) four planar points to their 3D
  reference positions
• The calibration method is based on determining
  the 2D projective Homography between the observed
  points and their reference position.
• A 2D projective Homography is a 3x3 matrix that
  operates on 2D homogeneous points

16
Zhangs Planar Calibration Method
The 3x3 Homography defines the motion from the
image coordinates of the pattern to the reference
coordinates
The camera calibration parameters are extracted
from the recovered Homography
17
Zhangs Planar Calibration Method

• The method is based on the following observation
• Where R is a 3x3 rotation matrix, t is a 3x1
  translation vector, and K is the internal
  parameters of the camera.

18
Zhangs Planar Calibration Method

• Assuming intrinsics are known

19
Planar Calibration Example

• Augmented Reality with simple pattern
• Planar pattern detected each frame by
  thresholding and finding connected components
• Quadrilateral shapes are warped to squares and
  correct orientation is found
• Homography is recovered and used to calibrate
  camera

20
Region-based Alternative
Input Image at time t

• On initialization, a user selects a plane of
  interest
• The rectifying Homography and rectified template
  image are retained

H
Template
21
Region-based Alternative
Image at time t
Image at time t1

• When new image arrives, use image intensities to
  refine the Homography

H
H
Template
22
Region-Based SSD Tracking(Lucas-Kanade Tracking)

• Mathematically, given
• a template, T, which is indexed by a set of 2D
  points,
• Define the warp, which in the most general case
  is a homography
• The parameters of the warp are

23
Region-Based SSD Tracking

• Find the parameter update that minimizes the sum
  of squared differences (SSD)
• To minimize, first perform Taylor series
  Expansion

24
Region-Based SSD Tracking

• The minimization problem is equivalent to a
  least-squares problem, with m equations, one for
  each xi
• Giving, the following

25
Region-Based SSD Example
Image at time t

• Image at time t1

Diff. Between template1
Template
Warped Image at t1
26
Region-Based SSD Example
Image derivatives w.r.t the homography parameters
Image Diff
h11
h13
h12
h21
h22
h23
h31
h32

Update is Essentially a linear combination of
Partial Derivative Images
27
Region-Based SSD Example

• Successive improvement after several iterations

Rectified
Diff. From Template
SSD score
6867
2809
1799
583
28
Region-Based AR Example

• A single planar region was identified, tracked,
  and used to register the world coordinate frame
  with the camera

29
Planar Calibration for AR

• Problems with plane-based approaches
• Poor registration for objects far from the plane
• Registration degrades at grazing views
• Limited viewing range with single planar
  pattern/region
• Potential Solutions
• use several planes (Buenaposada et al.)
• use other feature types (Marchand et al.)

30
Multiple Region-Based Registration

• Use multiple planar regions that are registered
  with respect to one another (3D model)
• Mathematical formulation is similar to the single
  plane-based SSD tracking
• Update in camera parameters is influenced by all
  planar regions

Approximate 3D model
31
Other Approaches

• Compute 3D model and register camera (Davison et
  al.)
• Camera/User state is modeled with a position,
  orientation, 3D velocity, and angular velocity
• Salient image features are detected and matched
  in subsequent frames to initialize uncertain 3D
  features
• Extended Kalman Filter (EKF) is used to update
  camera state, 3D feature position, and their
  covariance matrices

http//www.doc.ic.ac.uk/ajd/
32
Summary

• Vision-based registration useful for AR
• High accuracy is possible
• Plane-based techniques simple and efficient
• Use of non-planar features more stable through
  wider ranges

33
References

• Azuma, Ronald T. "A Survey of Augmented Reality."
  Presence Teleoperators and Virtual Environments
  6, 4 (August 1997), 355 - 385
• Z. Zhang. A flexible new technique for camera
  calibration. IEEE Transactions on Pattern
  Analysis and Machine Intelligence,
  22(11)1330-1334, 2000.
• José Miguel Buenaposada, Enrique Muñoz, Luis
  Baumela. Tracking heads using piecewise planar
  models.. Proc. of Iberian Conference on Pattern
  Recognition and Image Analysis, IbPRIA 2003. LNCS
  2652, pp. 126-133 (ISBN 3-540-40217-9), (c)
  Springer-Verlag. Palma de Mallorca, Spain, June
  2003.
• Dana Cobzas and Peter Sturm, 3D SSD Tracking
  with Estimated 3D Planes,In proceedings of
  Computer Robot Vision (CRV05), Pages 129-134,
  2005.
• José Miguel Buenaposada Biencinto, Luis Baumela
  Molina. Real-time tracking and estimation of
  plane pose., In Proc. of International Conference
  on Pattern Recognition, ICPR 2002. Vol. II, pp.
  697-700. (c) IEEE. Quebec, Canada, August 2002.
• E. Marchand, F. Chaumette. Virtual Visual
  Servoing a framework for real-time augmented
  reality. In EUROGRAPHICS 2002 Conference
  Proceeding, G. Drettakis, H.-P. Seidel (eds.),
  Computer Graphics Forum, Volume 21(3), Pages
  289-298, Sarrebruck, Germany, September 2002.
• Simon Baker and Iain Matthews. Lucas-Kanade 20
  Years On A Unifying Framework. International
  Journal of Computer Vision, Vol. 56, No. 3,
  March, 2004, pp. 221 - 25