Approximate Correspondences in High Dimensions

1
Approximate Correspondences in High Dimensions

• Kristen Grauman
• Trevor Darrell
• MIT CSAIL
• () UT Austin

2
Key challenges robustness
Illumination
Object pose
Clutter
Occlusions
Viewpoint
3
Key challenges efficiency

• Thousands to millions of pixels in an image
• 3,000-30,000 human recognizable object categories
• Billions of images indexed by Google Image Search
• 18 billion prints produced from digital camera
  images in 2004
• 295.5 million camera phones sold in 2005

4
Local representations
Describe component regions or patches separately
Salient regions Kadir et al.
Harris-Affine Schmid et al.
5
How to handle sets of features?

• Each instance is unordered set of vectors
• Varying number of vectors per instance

6
Partial matching

• Compare sets by computing a partial matching
  between their features.

7
Pyramid match overview
8
Computing the partial matching

• Optimal matching
• Greedy matching
• Pyramid match

9
Pyramid match overview
Pyramid match measures similarity of a partial
matching between two sets

• Place multi-dimensional, multi-resolution grid
  over point sets
• Consider points matched at finest resolution
  where they fall into same grid cell
• Approximate optimal similarity with worst case
  similarity within pyramid cell

No explicit search for matches!
10
Pyramid match
Approximate partial match similarity
Grauman and Darrell, ICCV 2005
11
Pyramid extraction
12
Counting matches
Histogram intersection
13
Example pyramid match
14
Example pyramid match
15
Example pyramid match
16
Example pyramid match
pyramid match
optimal match
17
Approximating the optimal partial matching
x
Randomly generated uniformly distributed point
sets with m 5 to 100, d2
18
PM preserves rank
19
and is robust to clutter
20
Learning with the pyramid match

• Kernel-based methods
• Embed data into a Euclidean space via a
  similarity function (kernel), then seek linear
  relationships among embedded data
• Efficient and good generalization
• Include classification, regression, clustering,
  dimensionality reduction,
• Pyramid match forms a Mercer kernel

21
Category recognition results
ETH-80 data set
Accuracy
Time (s)
Mean number of features
Mean number of features
22

Category recognition results
23
Vocabulary-guided pyramid match

• But rectangular histogram may scale poorly with
  input dimension
• Build data-dependent histogram structure
• New Vocabulary-guided PM NIPS 06
• Hierarchical k-means over training set
• Irregular cells record diameter of each bin
• VG pyramid structure stored O(kL) stored once
• Individual Histograms still stored sparsely

24
Vocabulary-guided pyramid match
Uniform bins

• Tune pyramid partitions to the feature
  distribution
• Accurate for d gt 100
• Requires initial corpus of features to determine
  pyramid structure
• Small cost increase over uniform bins kL
  distances against bin centers to insert points

25
Vocabulary-guided pyramid match
W new matches _at_ level i
wij ( matches in cell j level i -
matches in children)
nij(X) hist. X level i cell j

• wij weight for hist. X level i cell j
• diameter of cell
• dij(X) dij(Y)
• (dij(H)max dist of Hs pts in cell i,j to
  center)

Mercer kernel
Upper bound
26
Results Evaluation criteria

• Quality of match scores
  How similar are the rankings produced by the
  approximate measure to those produced by the
  optimal measure?
• Quality of correspondences
  How similar is the approximate correspondence
  field to the optimal one?
• Object recognition accuracy
  Used as a match kernel
  over feature sets, what is the recognition output?

27
Match score quality
ETH-80 images, sets of SIFT features
d128
d8
Vocabulary-guided pyramid match
d8
d128
Uniform bin pyramid match
Dense SIFT (d128) k10, L5 for VG PM PCA for
low-dim feats
28
Match score quality
ETH-80 images, sets of SIFT features
29
Spearman correlation

• Correlation coefficient to measure how well two
  ordinal rankings agree

rank value in true ordering
corresponding rank assigned by approximate
ordering
30
Bin structure and match counts
Data-dependent bins allow more gradual distance
ranges
31
Approximate correspondences

• Use pyramid intersections to compute smaller
  explicit matchings.

32
Approximate correspondences
Use pyramid intersections to compute smaller
explicit matchings.
33
Correspondence examples
34
Approximate correspondences
ETH-80 images, sets of SIFT descriptors
35
Approximate correspondences
ETH-80 images, sets of SIFT descriptors
36
Impact on recognition accuracy

• VG-PMK as kernel for SVM
• Caltech-4 data set
• SIFT descriptors extracted at Harris and MSER
  interest points

37
Sets of features elsewhere
diseases as sets of gene expressions