Efficient Algorithms for Matching

1
Efficient Algorithms for Matching

• Pedro Felzenszwalb
• Trevor Darrell
• Yann LeCun
• Alex Berg

2
Efficient Algorithms for Matching

• Pedro Felzenszwalb
• Trevor Darrell
• Yann LeCun
• Alex Berg

Polynomial exact
3
Efficient Algorithms for Matching

• Pedro Felzenszwalb
• Trevor Darrell
• Yann LeCun
• Alex Berg

Polynomial exact
Multilinear approximate
4
Efficient Algorithms for Matching

• Pedro Felzenszwalb
• Trevor Darrell
• Yann LeCun
• Alex Berg

Polynomial exact
Multilinear approximate
Fast? but very good
5
Efficient Algorithms for Matching

• Pedro Felzenszwalb
• Trevor Darrell
• Yann LeCun
• Alex Berg

Polynomial exact
Multilinear approximate
Fast? but very good
Happy when things work
6
First Criticism

• Efficiently computing the wrong solution is not
  so useful

First Response

• Even if say, an algorithm does not solve object
  recognition, it can still be a useful tool

7
Why Matching?

• Ideas hatched before me
• Statistical Pattern Theory (Ulf Grenander)
• Deformable Templates
• Fischler Elshlager
• Etc. at least by the early 1970s
• transform and appearance parameters
• Matching to estimate transform

Used to be continuous, now often discrete Very
general, Translation / Diffeomorphism /
Assignment Image / Features / Parts / etc.
8
Why Matching?

• Ideas hatched before me
• Statistical Pattern Theory (Ulf Grenander)
• Deformable Templates
• Fischler Elschlager
• Etc. at least by the early 1970s
• transform and appearance parameters
• Matching to estimate transform

TRANSFORM
MODEL
IMAGE
Used to be continuous, now often discrete Very
general, Translation / Diffeomorphism /
Assignment Image / Features / Parts / etc.
9
Why Matching?

• Ideas hatched before me
• Statistical Pattern Theory (Ulf Grenander)
• Deformable Templates
• Fischler Elschlager
• Etc. at least by the early 1970s
• transform and appearance parameters
• Matching to estimate transform

TRANSFORM
MODEL
IMAGE
Used to be continuous, now often discrete Very
general, Translation / Diffeomorphism /
Assignment Image / Features / Parts / etc.
10
Why Matching?

• Ideas hatched before me
• Statistical Pattern Theory (Ulf Grenander)
• Deformable Templates
• Fischler Elschlager
• Etc. at least by the early 1970s
• transform and appearance parameters
• Matching to estimate transform
• Searching over diffeomorphisms difficult
• Searching over discrete assignments easier?

TRANSFORM
MODEL
IMAGE
Used to be continuous, now often discrete Very
general, Translation / Diffeomorphism /
Assignment Image / Features / Parts / etc.
11
Search for a Transformation
Image
Model of Car
?
12
Find Transformation Using Correspondence
Image
Model of Car
Sometimes

• Search through a discrete set of possible point
  correspondences
• Objective function should be close to cost of the
  original model
• Use the discrete correspondences to obtain a
  continuous transformation if needed

13
Find Transformation Using Correspondence
Image
Model of Car
Why it works Sometimes we can measure
consistency of model appearance locally Inspired
by branch and bound If local appearance is
inconsistent, any alignment with that appearance
is bad.
My preferred way of motivating
local features
14
Find Transformation Using Correspondence
Image
Model of Car
Sometimes local appearance is not enough, so we
model some version of spatial constraints.
Do not make the problem harder than it was
15
Linear Assignmente.g. Hungarian AlgorithmJust
Features, no Geometry
Individual feature matches provide most of the
solution.
16
Quadratic Assignment(Adding Geometric
Constraints)
Individual feature matches provide most of the
solution. Geometric consistency only has to clean
things up a little.
In this case we formulate the matching as an
Integer Quadratic Programming problem and look
for an approximate solution
17
Second Meta-Comment

• Even if a problem can have very difficult
  instances, the effective complexity of certain
  instances might be quite low. This can be quite
  difficult to verify formally.

18
Use Alignment to Compare
Back to the alignment
Model of Car

• Given alignment evaluate the model
• Note we might have been done already
• Grauman et al, Zhang et al
• Actually do some alignment and check
• the quality of the fit.

19
Humans can be very efficient
Thats all fine, but

• Simon Thorpe animal or not in lt150 ms
• Feed forward process
• Difficult to retrain (familiarization does not
  make a difference)
• Salient parts of images are actually processed
  more rapidly.
• Support for some styles of current algorithms
• Neurophysiological evidence for some mid-level
  vision (illusory contours, figure ground, etc.)
• von der Heydt et al

20
What is an Object?

• Apple yes, mist no
• A rule of thumb is that objects have some
  definite spatial support
• Image/Scene? Some context models treat scenes or
  images as objects
• Face, eyes, nose, eye-lashes
• We can build SPT models for all of these

21
Heuristics

• Take advantage of the data
• Sometimes a single feature is enough
• For efficiency need to weigh this against how
  often that feature is found
• Many (?) object recognition datasets allow easy
  discrimination between categories with only very
  simple features extracted from the whole image,
  eg Pascal and Caltech 101.
• Segmentation or Figure/Ground -- might as well
  see if there is an object there before trying to
  recognize it

22
An Approach

• Extract features from an image
• Look features up in a large database
• Approximate Nearest Neighbor algorithms can make
  this sub-linear.
• Each entry tells us about which hypotheses
  Object,position,pose,... might be present.
• Use a short list of these to check in more
  detail using a matching framework
• Simple matching can actually be indexed. Indyk et
  al, Grauman et al.
• Finally use a matching to align models and apply
  more expensive processing