Learning Decompositional Shape Models from Examples

1
Learning Decompositional Shape Models from
Examples

• Alex Levinshtein
• Cristian Sminchisescu
• Sven Dickinson

2
The Evolution of Object Recognition
3
Appearance-based models
Automatically built appearance-based model from
video sequence (Ramanan, D. and Forsyth, D.A.,
Using Temporal Coherence to Build Models of
Animals, ICCV, 2003)
4
Appearance-based models
Constellation model (Fergus, R., Perona, P., and
Zisserman, A., Object Class Recognition by
Unsupervised Scale-Invariant Learning, CVPR,
2003)
5
Hierarchical Models
Manually built hierarchical model proposed by
Marr And Nishihara (Representation and
recognition of the spatial organization of three
dimensional shapes, Proc. of Royal Soc. of
London, 1978)
6
Our goal
Automatically construct a generic hierarchical
shape model from exemplars

• Challenges
• Cannot assume similar appearance among different
  exemplars
• Generic features are highly ambiguous
• Generic features may not be in one-to-one
  correspondence

7
Automatically constructed Hierarchical Models
Input
Question What is it?
Output
8
Stages of the system
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
9
Blob Graph Construction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
10
Blob Graph Construction
On the Representation and Matching of Qualitative
Shape at Multiple Scales A. Shokoufandeh, S.
Dickinson, C. Jonsson, L. Bretzner, and T.
Lindeberg,ECCV 2002

• Choose the largest connected component.

11
Blob Graph Construction
Perceptual grouping of blobs
Connectivity measure maxd1/major(A),
d2/major(B)
12
Blob Graph Construction
Edge weights between connected blobs

• Edge weights between disconnected blobs are
  computed based on shortest path distances.
• Edge weights are invariant to articulation.

13
Feature matching
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
14
Feature matching
One-to-one matching. Rely on shape and context,
not appearance!
Many-to-many matching
?
15
Feature embedding
0 171 202 230
171 0 373 400
202 373 0 432
230 400 432 0
Spectral embedding
16
Matching using Earth Movers Distance
17
EMD under Transformation Algorithm
18
Returning to our set of inputs

• Many-to-many matching of every pair of exemplars.

19
Part Extraction
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
20
(No Transcript)
21
Results of the part extraction stage
22
What is next?
23
Extracting attachment relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
24
Extracting attachment relations
Right arm is typically connected to torso in
exemplar images !
25
Extracting attachment relations
Number of times blobs drawn from the two clusters
were attached
Number of times blobs from the two clusters
co-appeared in an image.
26
Extracting decomposition relations
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
27
Extracting decomposition relations
28
Extracting decomposition relations
Sum of all flows between blobs in two clusters
Number of flows between blobs in two clusters
29
Finding decompositions (example)
30
Model construction stage summary
Model Construction

• Clustering blobs into parts based on one-to-one
  matching results.
• Recovering relations between parts based on
  individual matching and attachment results.

31
Assemble Final Model
Exemplar images
Extract Blob Graphs
Blob graphs
Match Blob Graphs (many-to-many)
Many-to-many correspondences
Extract Parts
Model parts
Extract Decomposition Relations
Extract Attachment Relations
Model decomposition relations
Model attachment relations
Assemble Final Model
32
(No Transcript)
33
(No Transcript)
34
Experiments
35
List of system parameters
Parameter Description
Perceptual Grouping Threshold The threshold determining the pairwise connectivity between blobs in the exemplar images
D The dimensionality of the embedding space during matching
Embedding dimensionality during part extraction The dimensionality of the embedding space used for blob clustering.
K The maximum number of parts in the model
Tattach The threshold for accepting attachment between parts in the final model
Tchild The threshold for considering a part to be a potential child of another part
Tdecomp The threshold for accepting a decomposition
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
Failure Modes
43
(No Transcript)
44
(No Transcript)
45
Conclusions

• General framework for constructing a generic
  decompositional model from different exemplars
  with dissimilar appearance.
• Recovering decompositional relations requires
  solving the difficult many-to-many graph matching
  problem.
• Preliminary results indicate good model recovery
  from noisy features.

46
Future work

• Construct models for objects other than humans.
• Provide scale invariance during matching.
• Automatically learn perceptual grouping relations
  from labeled examples.
• Develop indexing and matching framework for
  decompositional models.