CS 433/557 Algorithms for Image Analysis

1
CS 433/557 Algorithms for Image Analysis
Template Matching Acknowledgements Dan
Huttenlocher
2
CS 433/557 Algorithms for Image Analysis
Matching and Registration

• Template Matching
• intensity based (correlation measures)
• feature based (distance transforms)
• Flexible Templates
• pictorial structures
• Dynamic Programming on trees
• generalized distance transforms

Extra Material
3
Intensity Based Template MatchingBasic Idea
Left ventricle template
image
Find best template position in the image
4
Intensity-BasedRigid Template matching
5
Intensity-BasedRigid Template matching
s1
s2
Search over all plausible positions s and find
the optimal one that has the largest goodness of
match value Q(s)
6
Intensity-BasedRigid Template matching

• What if intensities of your image are not exactly
  the same as in the template? (e.g. may happen due
  to different gain setting at image acquisition)

7
Other intensity basedgoodness of match measures

• Normalized correlation
• Mutual Information (next slide)

8
Other goodness of match measures Mutual
Information

• Will work even in extreme cases

9
Other goodness of match measures Mutual
Information
Fix s and consider joint histogram of intensity
pairs
T
s1
I

• Mutual information between template T and image I
  (for given transformation s) describes
  peakedness of the joint histogram
• measures how well spatial structures in T and I
  align

10
Mutual Information(technical definition)
Assuming two random variables X and Y their
mutual information is
entropy and joint entropy e for random variables
X and Y measures peakedness of
histogram/distribution
marginal histogram (distribution)
joint histogram (distribution)
11
Mutual InformationComputing MI for a given
position s

• We want to find s that maximizes MI that can be
  written as

T
marginal distributions Pr(x) and Pr(y)
I
joint distribution Pr(x,y) (normalized
histogram) for a fixed given s
NOTE has to be careful when computing. For
example, what if H(x,y)0 for a given pair (x,y)?
12
Finding optimal template position s

• Need to search over all feasible values of s
• Template T could be large
• The bigger the template T the more time we spend
  computing goodness of match measure at each s
• Search space (of feasible positions s) could be
  huge
• Besides translation/shift, position s could
  include scale, rotation angle, and other
  parameters (e.g. shear)
• Q Efficient search over all s?

13
Finding optimal template position s

• One possible solution Hierarchical Approach

1. Subsample both template and image. Note that the
   search space can be significantly reduced. The
   template size is also reduced.
2. Once a good solution(s) is found at a corser
   scale, go to a finer scale. Refine the search in
   the neighborhood of the courser scale solution.

14
Feature Based Template Matching

• Features edges, corners, (found via filtering)
• Distance transforms of binary images
• Chamfer and Housdorff matching
• Iterated Closed Points

15
Feature-based Binary Templates/ModelsWhat are
they?

• What are features?
• Object edges, corners, junctions, e.t.c.
• Features can be detected by the corresponding
  image filters
• Intensity can also be a considered a feature but
  it may not be very robust (e.g. due to
  illumination changes)
• A model (binary template) is a set of feature
  points in
  N-dimensional space (also called feature space)
• Each feature is defined by a descriptor (vector)

16
Binary Feature Templates (Models) 2D example

• descriptor could be a 2D vector specifying
  feature position with respect to models
  coordinate system
• Feature spaces could be 3D (or higher). E.g.,
  position of an edge in a medical volumes is a 3D
  vector. But even in 2D images edge features can
  be described by 3D vectors (add edges angular
  orientation to its 2D location)

2D feature space
For simplicity, we will mainly concentrate on 2D
feature space examples
17
Matching Binary Template to Image
At fixed position L we can compute match quality
Q(L) using some goodness of match criteria.
Object is detected at all positions which
are local maxima of function Q(L) such that
where K is some presence threshold
Example Q(L) number of (exact) matches (in
red) between model and image features (e.g.
edges).
18
Exact feature matching is not robust
Counting exact matches may be sensitive to even
minor deviation in shape between the model and
the actual object appearance
19
Distance Transform
More robust goodness of match measures use
distance transform of image features

1. Detect desirable image features (edges, corners,
   e.t.c.) using appropriate filters
2. For all image pixels p find distance D(p) to the
   nearest image feature

20
Distance Transform
Image features (2D)
21
Distance Transform
Distance Transform can be visualized as a
gray-scale image
Image features (edges)
Distance Transform
22
Distance Transform can be very efficiently
computed
23
Distance Transform can be very efficiently
computed
24
Metric properties ofdiscrete Distance Transforms
Metric
Forward mask
Backward mask
Set of equidistant points
Manhattan (L1) metric
25
Goodness of Match viaDistance Transforms
At each model position one can probe distance
transform values at locations specified by model
(template) features
Use distance transform values as evidence of
proximity to image features.
26
Goodness of Match Measuresusing Distance
Transforms

• Chamfer Measure
• sum distance transform values probed by
  template features
• Hausdorff Measure
• k-th largest value of the distance transform at
  locations probed by template features
• (Equivalently) number of template features with
  probed distance transform values less than
  fixed (small) threshold
• Count template features sufficiently close to
  image features
• Spatially coherent matching

27
Hausdorff Matching

• counting matches with a dialated set of image
  features

28
Spatial Coherence of Feature Matches
50 matched
50 matched
29
Spatially Coherent Matching
Separate template/model features into three
subsets

• Matchable (red)
• -near image features
• Boundary (blue circle)
• -matchable but near un-matchable
• -links define near for model features
• Un-matchable (gray)
• -far from image features

Count the number of non-boundary matchable
features
30
Spatially Coherent Matching
Percentage of non-boundary matchable features
(spatially coherent matches)
31
Comparing different match measures

• Monte Carlo experiments with known object
  location and synthetic clutter and occlusion
• -Matching edge locations
• Varying percent clutter
• -Probability of edge pixel 2.5-15
• Varying occlusion
• -Single missing interval 10-25 of the
  boundary
• Search over location, scale, orientation

Binary model (edges)
5 clutter image
32
Comparing different match measuresROC curves
Probability of false alarm versus detection -
10 and 15 of occlusion with 5
clutter -Chamfer is lowest, Hausdorff (f0.8)
is highest -Chamfer truncated distance better
than trimmed
33
ROCs forSpatial Coherence Matching
34
Edge Orientation Information

• Match edge orientation (in addition to location)
• Edge normals or gradient direction
• 3D model feature space (2D location
  orientation)
• Extract 3D (edge) features from image as well.
• Requires 3D distance transform of image features
• weight orientation versus location
• fast forward-backward pass algorithm applies
• Increases detection robustness and speeds up
  matching
• better able to discriminate object from clutter
• better able to eliminate cells in branch and
  bound search

35
ROCs for Oriented Edge Pixels

• Vast Improvement for moderate clutter
• Images with 5 randomly generated contours
• Good for 20-25 occlusion rather than 2-5

Location only
Oriented Edges
36
Efficient search for good matching positions L

• Distance transform of observed image features
  needs to be computed only once (fast operation).
• Need to compute match quality for all possible
  template/model locations L (global search)
• Use hierarchical approach to efficiently prune
  the search space.
• Alternatively, gradient descent from a given
  initial position (e.g. Iterative Closest Point
  algorithm, later)
• Easily gets stuck at local minima
• Sensitive to initialization

37
Global Search Hierarchical Search Space Pruning
Assume that the entire box might be pruned out if
the match quality is sufficiently bad in the
center of the box (how? in a moment)
38
Global SearchHierarchical Search Space Pruning
If a box is not pruned then subdivide it into
smaller boxes and test the centers of these
smaller boxes.
39
Global Search Hierarchical Search Space Pruning

• Continue in this fashion until the object is
  localized.

40
Pruning a Box (preliminary technicality)
Location L is uniformly better than L if
for all
model features i
A uniformly better location is guaranteed to have
better match quality!
41
Pruning a Box (preliminary technicality)

• Assume that is uniformly better than any
  location
• then the match quality satisfies
  for any
• If the presence test fails (
  for a given threshold K) then any location
  must also fail the test
• The entire box can be pruned by one test at
  !!!!

42
Building for a Box of Radius n

• value of the distance transform changes at most
  by 1 between neighboring pixels

• value of can decrease by at most
  n (box radius) for other box positions

43
Global Hierarchical Search (Branch and Bound)

• Hierarchical search works in more general case
  where position L includes translation, scale,
  and orientation of the model
• N-dimensional search space
• Guaranteed or admissible search heuristic
• Bound on how good answer could be in unexplored
  region
• can not miss an answer
• In worst case wont rule anything out
• In practice rule out vast majority of template
  locations (transformations)

44
Local Search (gradient descent)Iterated Closest
Point algorithm

• ICP Iterate until convergence
• Estimate correspondence between each template
  feature i and some image feature located at F(i)
  (Fitzgibbons use DT)
• Move model to minimize the sum of distances
  between the corresponding features (like chamfer
  matching)
• Alternatively, find local move of the
  model improving DT-based match
  quality function Q(L)

45
Problems with ICPand gradient descent matching

• Slow
• Can take many iterations
• ICP each iteration is slow due to search for
  correspondences
• Fitzgibbons improve this by using DT
• No convergence guarantees
• Can get stuck in local minima
• Not much to do about this
• Can be improved by using robust distance measures
  (e.g. truncated Euclidean measure)

46
Observations on DT based matching

• Main point of DT allows to measure match quality
  without explicitly finding correspondence between
  pairs of mode and image features (hard problem!)
• Hierarchical search over entire transformation
  space
• Important to use robust distance
• Straight Chamfer very sensitive to outliers
• Truncated DT can be computed very fast
• Fast exact or approximate methods for DT (
  metric)
• For edge features use orientation too
• edge normals or intensity gradients

47
Rigid 2D templatesShould we really care?

• So far we studied matching in case of 2D images
  and rigid 2D templates/models of objects
• When do rigid 2D templates work?
• there are rigid 2D objects (e.g. fingerprints)
• 3D object may be imaged from the same view point
• controlled image-based data bases (e.g. photos of
  employees, criminals)
• 2D satellite images always view 3D objects from
  above
• X-Rays, microscope photography, e.t.c.

48
More general 3D objects

• 3D image volumes and 3D objects
• Distance transforms, DT-based matching criteria,
  and hierarchical search techniques easily
  generalize
• Mainly medical applications
• 2D image and 3D objects
• 3D objects may be represented by a collection of
  2D templates (e.g. tree-structured templates,
  next slide)
• 3D objects may be represented by flexible 2D
  templates (soon)

49
Tree-structured templates
Larger pair-wise differences higher in tree
50
Tree-structured templates

• Rule out multiple templates simultaneously
• - Speeds up matching
• - Course-to-fine search where coarse granularity
  can rule out many templates
• - Applies to variety of DT based matching
  measures
• Chamfer, Hausdorff, robust Chamfer

51
Flexible Templates
Flexible Template combines a number of rigid
templates connected by flexible strings

• parts connected by springs and appearance models
  for each part
• Used for human bodies, faces
• Fischler Elschlager, 1973 considerable recent
  work (e.g. Felzenszwalb Huttenlocher, 2003 )

52
Flexible TemplatesWhy?

• To account for significant deviation between
  proportions of generic model (e.g average face
  template) and a multitude of actual object
  appearance

53
Flexible TemplatesFormal Definition

• Set of parts
• Positioning Configuration
• specifies locations of the parts
• Appearance model
• matching quality of part i at location
• Edge for
  connected parts
• explicit dependency between edge-connected parts
• Interaction/connection energy
• e.g. elastic energy

54
Flexible TemplatesFormal Definition

• Find configuration L (location of all parts) that
  minimizes
• Difficulty depends on graph structure
• Which parts are connected (E) and how (C)
• General case exponential time

55
Flexible Templatessimplistic example from the
past

• Discrete Snakes
• What graph?
• What appearance model?
• What connection/interaction model?
• What optimization algorithm?

56
Flexible Templatesspecial cases

• Pictorial Structures
• What graph?
• What appearance model?
• -intensity based match measure
• -DT based match measure (binary templates)
• What connection/interaction model?
• -elastic springs
• What optimization algorithm?

57
Dynamic Programming forFlexible Template Matching

• DP can be used for minimization of E(L) for tree
  graphs (no loops!)

58
Dynamic Programming forFlexible Template Matching

• DP algorithm on trees
• Choose post-order traversal for
  any selected
  root site/part
• Compute for all leaf
  parts
• Process a part after its children are processed
• Select best energy position for the root and
  backtrack to leafs

If part i has only one child a
If part i has two (or more) children a, b,

59
Dynamic Programming forFlexible Template Matching

• DPs complexity on trees
• (same as for 1D snakes)
• n parts, m positions
• OK complexity for local search where m
  is relatively small (e.g. in snakes)
• E.g. for tracking a flexible model from
  frame to frame in a video sequence

60
Local SearchTracking Flexible Templates
61
Local SearchTracking Flexible Templates
62
Local SearchTracking Flexible Templates
63
Searching in the whole image (large m)

• m image size
• or
• m image sizerotations
• Then complexity is not good
• For some interactions can improve to
  based on Generalized Distance Transform
  (from Computational Geometry)

64
Generalized Distance Transform

• Idea improve efficiency of the key computational
  step (performed for each parent-child pair,
  n-times)
  ( operations)

65
Generalized Distance Transform

• Idea improve efficiency of the key computational
  step ( operations performed for each
  parent-child pair)
• Let
  (distance between x and y)
• 
  reasonable interactions model!
• Then is called a
  Generalized Distance Transform of

66
From Distance Transform toGeneralized Distance
Transform

• Assuming
• then
• is standard Distance Transform (of image
  features)

Locations of binary image features
67
From Distance Transform toGeneralized Distance
Transform

• For general and any fixed
• is called Generalized Distance Transform of

E(x) may represent non-binary image features
(e.g. image intensity gradient)
68
Algorithm for computing
Generalized Distance Transform

• Straightforward generalization of
  forward-backward pass algorithm for standard
  Distance Transforms

69
Flexible Template Matching Complexity

• Computing
• via Generalized Distance Transform
• previously
• (m-number of positions x and y)
• Improves complexity of Flexible Template Matching
  to in case of interactions

70
Simple Flexible Template ExampleCentral Part
Model

• Consider special case in which parts translate
  with respect to common origin
• E.g., useful for faces

NOTE for simplicity (only) we consider part
positions that are translations only
(no rotation or scaling of parts)
71
Central Part Model example

• Ideal location w.r.t. is given by
  where is a fixed translation
  vector for each igt1

72
Central Part Model Summary of
search algorithm
Matching cost

1. For each non-central part igt1 compute matching
   cost for all possible positions
   of that part in the image
2. For each igt1 compute Generalized DT of
3. For all possible positions of the central part
   compute energy
4. Select the best location or select all
   locations with larger then a fixed
   threshold

73
Central Part Modelfor face detection
74
Search Algorithmfor tree-based
pictorial structures

• Algorithm is basically the same as for Central
  Part Model.
• Each parent part knows ideal positions of
  child parts.
• String deformations are accounted for by the
  Generalized Distance Transform of the childrens
  positioning energies