Learning Decompositional Shape Models from Examples

1
Learning Decompositional Shape Models from
Examples

• Alex Levinshtein
• Cristian Sminchisescu
• Sven Dickinson
• University of Toronto

2
Hierarchical Models
Manually built hierarchical model proposed by
Marr And Nishihara (Representation and
recognition of the spatial organization of three
dimensional shapes, Proc. of Royal Soc. of
London, 1978)
3
Our goal
Automatically construct a generic hierarchical
shape model from exemplars

• Challenges
• Cannot assume similar appearance among different
  exemplars
• Generic features are highly ambiguous
• Generic features may not be in one-to-one
  correspondence

4
Appearance-based models
Automatically built appearance-based model from
video sequence (Ramanan, D. and Forsyth, D.A.,
Using Temporal Coherence to Build Models of
Animals, ICCV, 2003)
5
Layered Motion Segmentations Kumar, Torr and
Zisserman, ICCV 2005

• Models image projection, lighting and motion blur
• Models spatial continuity, occlusions, and works
  over multiple frames (cf. earlier work by Jojic
  Frey, CVPR 2001)
• Estimates the number of segments, their mattes,
  layer assignment, appearance, lighting and
  transformation parameters for each segment
• Initialization using loopy BP, refinement using
  graph cuts

6
Constellation models
Fergus, R., Perona, P., and Zisserman, A.,
Object Class Recognition by Unsupervised
Scale-Invariant Learning, CVPR 2003
7
Categorical features
Match
8
Constructing a Hierarchical Model from Examples
Input
Question What is it?
Output
9
Overview of the Approach
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
10
Blob Graph Construction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
11
Blob Graph Construction
The Representation and Matching of Categorical
Shape A. Shokoufandeh, L. Bretzner, D. Macrini,
M.F. Demirci, C. Jonsson, and S. Dickinson,
CVIU, Vol. 103, 2006, pp 139--154

• Edges are invariant to articulation
• Choose the largest connected component.

12
Blob Graph Construction
Perceptual grouping of blobs
Connectivity measure maxd1/major(A),
d2/major(B)
13
Feature matching
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
14
Feature matching
One-to-one matching. Rely on shape and context,
not appearance!
Many-to-many matching
15
A Many-to-Many Graph Matching Framework
1. Embed graphs with low distortion to yield
weighted point distributions. 2. Compute
many-to-many correspondences between the two
distributions using EMD. 3. The computed flows
yield a many-to-many node correspondence between
the two graphs.
Demirci, Shokoufandeh, Keselman, Bretzner, and
Dickinson (IJCV 2006)
16
Feature embedding and EMD
Spectral embedding
17
Returning to our set of inputs

• Many-to-many matching of every pair of exemplars.

18
Part Extraction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
19
Many-to-many matching results
100
100
100
50 50
20
Extracting parts

• Part a collection of blobs.
• Ideal part
• Represents blobs that occur frequently and
  participate in one-to-one correspondence across
  many exemplars.
• Finding parts
• From the pairwise matching results, find clusters
  (cliques) of blobs matching one-to-one.

21
(No Transcript)
22
Results of the part extraction stage
23
What is next?
24
Extracting attachment relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
25
Extracting attachment relations
Number of times blobs drawn from the two clusters
were attached
is high
Right arm is typically connected to torso in
exemplar images !
Number of times blobs from the two clusters
co-appeared in an image.
Torso
Right Arm
26
Extracting attachment relations
Number of times blobs drawn from the two clusters
were attached
Number of times blobs from the two clusters
co-appeared in an image.
Threshold PA to get part attachment
27
Extracting decomposition relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
28
Extracting decomposition relations
Left Arm
Upper
Lower
29
Extracting decomposition relations
Sum of all flows between blobs in two clusters
Number of flows between blobs in two clusters
Combine PA and PF to obtain a decomposition score
30
Model construction stage summary
Model Construction

• Clustering blobs into parts based on one-to-one
  matching results.
• Recovering relations between parts based on
  individual matching and attachment results.

31
Assemble Final Model
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Extract Decomposition Relations
Extract Attachment Relations
Model parts
Model decomposition relations
Model attachment relations
Assemble Final Model
32
Results
33
(No Transcript)
34
Experiments
35
List of system parameters
Parameter Description
Perceptual Grouping Threshold The threshold determining the pairwise connectivity between blobs in the exemplar images
D The dimensionality of the embedding space during matching
Embedding dimensionality during part extraction The dimensionality of the embedding space used for blob clustering.
K The maximum number of parts in the model
Tattach The threshold for accepting attachment between parts in the final model
Tchild The threshold for considering a part to be a potential child of another part
Tdecomp The threshold for accepting a decomposition
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
Conclusions

• Generic models must be defined at multiple levels
  of abstraction, as Marr proposed.
• Coarse shape features, such as blobs, are highly
  ambiguous and cannot be matched without
  contextual constraints.
• Moreover, features that exist at different levels
  of abstraction must be matched many-to-many in
  the presence of noise.
• The many-to-many matching results can be analyzed
  to yield both the parts and relations of a
  decompositional model.
• Preliminary results indicate that a limited
  decompositional model can be learned from a set
  of noisy examples.

43
Future work

• Construct models for objects other than humans
  objects with richer decompositional hierarchies.
• Automatically learn perceptual grouping relations
  between blobs from labeled examples.
• Develop indexing and matching frameworks for
  decompositional models.