Automatic Image Stitching using Invariant Features

1
Automatic Image Stitching using Invariant Features

• Matthew Brown and David Lowe,
• University of British Columbia

2
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35

3
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135

4
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135
• Panoramic Mosaic 360 x 180

5
Recognising Panoramas
6
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

7
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

8
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

9
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

10
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

11
Recognising Panoramas

• 1D Rotations (q)
• Ordering ? matching images

12
Recognising Panoramas
13
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

14
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

15
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

16
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

17
Invariant Features

• Schmid Mohr 1997, Lowe 1999, Baumberg 2000,
  Tuytelaars Van Gool 2000, Mikolajczyk Schmid
  2001, Brown Lowe 2002, Matas et. al. 2002,
  Schaffalitzky Zisserman 2002

18
SIFT Features

• Invariant Features
• Establish invariant frame
• Maxima/minima of scale-space DOG ? x, y, s
• Maximum of distribution of local gradients ? q
• Form descriptor vector
• Histogram of smoothed local gradients
• 128 dimensions
• SIFT features are
• Geometrically invariant to similarity transforms,
• some robustness to affine change
• Photometrically invariant to affine changes in
  intensity

19
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

20
Nearest Neighbour Matching

• Nearest neighbour matching
• Use k-d tree
• k-d tree recursively bi-partitions data at mean
  in the dimension of maximum variance
• Approximate nearest neighbours found in O(n log
  n)
• Find k-NN for each feature
• k ? number of overlapping images (we use k 4)

Beis Lowe 1997, Nene Nayar 1997, Gray Moore
2000, Shakhnarovich 2003
21
K-d tree
22
K-d tree
23
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

24
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

25
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

26
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

27
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

28
RANSAC for Homography
29
RANSAC for Homography
30
RANSAC for Homography
31
RANSAC 1D Line Fitting
32
RANSAC 1D Line Fitting
33
RANSAC 1D Line Fitting
34
The RANSAC Algorithm
function H RANSAC(points, nIterations) bestIn
liers 0 bestH zeros(3, 3) for (i 0 i
lt nIterations i) samplePoints
RandomSample(points) H ComputeTransform(sampl
ePoints) nInliers Consistent(H) if
(nInliers gt bestInliers) bestInliers
nInliers bestH H // end if // end
for // end RANSAC
35
2D Transforms

• Linear (affine)
• Homography

36
Finding the panoramas
37
Finding the panoramas
38
Finding the panoramas
39
Finding the panoramas
40
Connected Components
41
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

42
Bundle Adjustment

• Adjust rotation, focal length of each image to
  minimise error in matched features

43
Bundle Adjustment

• Adjust rotation, focal length of each image to
  minimise error in matched features

44
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

45
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

46
Multi-band Blending

• Burt Adelson 1983
• Blend frequency bands over range ? l

47
2-band Blending
Low frequency (l gt 2 pixels)
High frequency (l lt 2 pixels)
48
Linear Blending
49
2-band Blending
50
(No Transcript)
51
(No Transcript)
52
Multi-band Blending

• No blending

53
Multi-band Blending

• Linear blending
• Each pixel is a weighted sum

54
Multi-band Blending

• Multi-band blending
• Each pixel is a weighted sum (for each band)

55
Multi-band Blending

• Linear blending

• Multi-band blending

56
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

57
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

58
Results
59
Results
60
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

61
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

62
Conclusions

• Fully automatic panoramas
• A recognition problem
• Invariant feature based method
• SIFT features, RANSAC, Bundle Adjustment,
  Multi-band Blending
• O(nlogn)
• Future Work
• Advanced camera modelling
• radial distortion, camera motion, scene motion,
  vignetting, exposure, high dynamic range, flash
  
• Full 3D case recognising 3D objects/scenes in
  unordered datasets. PhotoTourism.

http//www.autostitch.net