Scale Invariant Feature Transform SIFT

1
Scale Invariant Feature Transform (SIFT)

• Jan Prokaj

2
Outline

• What is SIFT
• Algorithm overview
• Example images
• Applications
• Summary

3
Overview

• By David Lowe, 1999
• Generates image features, keypoints
• invariant to image scaling and rotation
• partially invariant to change in illumination and
  3D camera viewpoint
• many can be extracted from typical images
• highly distinctive

4
Algorithm overview

• Build scale-space
• Find extrema -- keypoints
• Calculate keypoint sub-pixel localization
• Assign orientation(s) to each keypoint
• Calculate keypoint descriptor
• Vector used in matching

5
Scale-space

• Series of images differing in their blurriness, s
• Formal definition
• Convolution of Gaussian kernel, G, with image I
• Gaussian

6
Scale-space

• Keypoints are detected as scale-space extrema in
  the difference-of-Gaussian function D
• Subtract adjacent G blurred images to get D image
• Efficient to compute (only a subtraction)

7
Scale-space construction
8
Frequency of sampling

• There is no minimum
• Best frequency determined experimentally

9
Smoothing of each octave

• Increasing s increases robustness, but costs
• s 1.6 a good tradeoff
• Doubling the image initially increases number of
  keypoints

10
Scale-space images
original
1st image in 1st first octave
11
Scale-space images
4th
3rd octave
1st image in 2nd octave
12
Difference-of-Gaussian images
2nd level 2nd octave
4th level 2nd octave
3rd level 2nd octave
1st level 2nd octave
13
Finding extrema

• Sample point is selected only if it is a minimum
  or a maximum of these points

DoG scale space
14
Sub-pixel Localization

• 3D quadratic function is fit to the local sample
  points
• Start with Taylor expansion with sample point as
  the origin
• where
• Take the derivative with respect to X, and set it
  to 0, giving
• as the location of the
  keypoint
• This is a 3x3 linear system (3 dimensions)

15
Localization

• Derivatives approximated by finite differences,
• example
• If X is gt 0.5 in any dimension, process repeated

16
Filtering

• Contrast (use prev. equation)
• If D(X) lt 0.03, throw it out
• Edge-iness
• Use ratio of principal curvatures to throw out
  poorly defined peaks
• Curvatures come from Hessian
• Ratio of Trace(H)2 and Determinant(H)
• If ratio gt (r1)2/(r), throw it out (Lowe uses
  r10)

17
Orientation assignment

• Orientation of the keypoint is needed to achieve
  rotation invariance
• Computing angles relative to keypoint
• Orientations are precomputed along with
  magnitudes for all scale-space images
• Multiple orientations are assigned from an
  orientation histogram (strong angles)

18
Keypoint images
19
Keypoint images
20
Descriptor

• Descriptor has 3 dimensions (x,y,?)
• Orientation histogram of gradient magnitudes
• Position and orientation of each gradient sample
  rotated relative to keypoint orientation

21
Descriptor

• Weight magnitude of each sample point by Gaussian
  weighting function, s0.5width
• Distribute each sample to adjacent bins by
  trilinear interpolation (avoids boundary effects)

22
Descriptor

• Best results achieved with 4x4x8 128
  descriptor size
• Normalize to unit length
• Reduces effect of illumination change
• Cap each element to 0.2, normalize again
• Reduces non-linear illumination changes
• 0.2 determined experimentally

23
Application hands
4th octave
24
Application hands
25
Application hands
26
Summary

• Algorithm for finding image features
• Uses scale-space theory
• Keypoints are distinctive, invariant to rotation
  and scale
• Descriptor vector , 128 elements

27
References

• Brown, M. and Lowe, D. Invariant Features from
  Interest Point Groups. BMVC 2002, pp. 253-262.
• Lowe, D. International Journal of Computer
  Vision, 60, 2 (2004), pp. 91-110.
• Lowe, D. International Conference on Computer
  Vision, Corfu, Greece (September 1999), pp.
  1150-1157.