Recognising Panoramas

1
Recognising Panoramas

• M. Brown and D. Lowe,
• University of British Columbia

2
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35

3
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135

4
Introduction

• Are you getting the whole picture?
• Compact Camera FOV 50 x 35
• Human FOV 200 x 135
• Panoramic Mosaic 360 x 180

5
Why Recognising Panoramas?
6
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

7
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

8
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

9
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

10
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

11
Why Recognising Panoramas?

• 1D Rotations (q)
• Ordering ? matching images

12
Why Recognising Panoramas?
13
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

14
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

15
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

16
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

17
Invariant Features

• Schmid Mohr 1997, Lowe 1999, Baumberg 2000,
  Tuytelaars Van Gool 2000, Mikolajczyk Schmid
  2001, Brown Lowe 2002, Matas et. al. 2002,
  Schaffalitzky Zisserman 2002

18
SIFT Features

• Invariant Features
• Establish invariant frame
• Maxima/minima of scale-space DOG ? x, y, s
• Maximum of distribution of local gradients ? q
• Form descriptor vector
• Histogram of smoothed local gradients
• 128 dimensions
• SIFT features are
• Geometrically invariant to similarity transforms,
• some robustness to affine change
• Photometrically invariant to affine changes in
  intensity

19
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

20
Nearest Neighbour Matching

• Find k-NN for each feature
• k ? number of overlapping images (we use k 4)
• Use k-d tree
• k-d tree recursively bi-partitions data at mean
  in the dimension of maximum variance
• Approximate nearest neighbours found in O(nlogn)

21
Overview

• Feature Matching
• SIFT Features
• Nearest Neighbour Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

22
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

23
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

24
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Probabilistic model for verification
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

25
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Probabilistic model for verification
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

26
RANSAC for Homography
27
RANSAC for Homography
28
RANSAC for Homography
29
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Probabilistic model for verification
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

30
Probabilistic model for verification
31
Probabilistic model for verification

• Compare probability that this set of RANSAC
  inliers/outliers was generated by a correct/false
  image match
• ni inliers, nf features
• p1 p(inlier match), p0 p(inlier match)
• pmin acceptance probability
• Choosing values for p1, p0 and pmin

32
Finding the panoramas
33
Finding the panoramas
34
Finding the panoramas
35
Finding the panoramas
36
Overview

• Feature Matching
• Image Matching
• RANSAC for Homography
• Probabilistic model for verification
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

37
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

38
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

39
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Error function
• Multi-band Blending
• Results
• Conclusions

40
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Error function
• Multi-band Blending
• Results
• Conclusions

41
Error function

• Sum of squared projection errors
• n images
• I(i) set of image matches to image i
• F(i, j) set of feature matches between images
  i,j
• rijk residual of kth feature match between
  images i,j
• Robust error function

42
Homography for Rotation

• Parameterise each camera by rotation and focal
  length
• This gives pairwise homographies

43
Bundle Adjustment

• New images initialised with rotation, focal
  length of best matching image

44
Bundle Adjustment

• New images initialised with rotation, focal
  length of best matching image

45
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Error function
• Multi-band Blending
• Results
• Conclusions

46
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

47
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

48
Multi-band Blending

• Burt Adelson 1983
• Blend frequency bands over range ? l

49
2-band Blending
Low frequency (l gt 2 pixels)
High frequency (l lt 2 pixels)
50
Linear Blending
51
2-band Blending
52
(No Transcript)
53
(No Transcript)
54
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

55
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

56
Results
57
Results
58
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

59
Overview

• Feature Matching
• Image Matching
• Bundle Adjustment
• Multi-band Blending
• Results
• Conclusions

60
Conclusions

• Fully automatic panoramas
• A recognition problem
• Invariant feature based method
• SIFT features, RANSAC, Bundle Adjustment,
  Multi-band Blending
• O(nlogn)
• Future Work
• Advanced camera modelling
• radial distortion, camera motion, scene motion,
  vignetting, exposure, high dynamic range, flash
  
• Full 3D case recognising 3D objects/scenes in
  unordered datasets

http//www.cs.ubc.ca/mbrown/panorama/panorama.htm
l
61
Questions?
http//www.cs.ubc.ca/mbrown/panorama/panorama.htm
l
62
Analytical computation of derivatives
63
Levenberg-Marquardt

• Iteration step of form