Jigsaws: joint appearance and shape clustering

1
Jigsaws joint appearance and shape clustering

• John Winnwith Anitha Kannan and Carsten Rother

Microsoft Research, Cambridge
2
Patch models

• Used for
• Object recognition/detection
• Object segmentation
• But also
• Stereo matching, photo stitching
• Texture synthesis
• Super-resolution
• Motion segmentation
• Image/video compression

3
Patch models

• Patch clustering/codebook (e.g. Leibe Schiele)

• Epitome (Jojic et al.)parameter sharing
  translation invariant

4
Issues with fixed patch size/shape

• Patch includes backgroundpatches containing the
  same object are not clustered together

• Patch excludes part of objectpatch is less
  discriminative

• Patch includes occlusionoccluded and unoccluded
  objects are not clustered together

5
Patch size?
More discriminative
More sharing
Less sharing
Less discriminative
Size
Small(single pixel)
Large(entire image)

• Depends on
• object size/shape
• object variability
• size of training set

6
Aims of jigsaw model

• Learn patches (jigsaw pieces) which are
• Shared each piece is similar in shape and
  appearance to many regions of the training
  images
• Discriminative each piece is as large as
  possible
• Exhaustive all parts of the training images can
  be reconstructed from the set of jigsaw pieces.

7
The Jigsaw model
Jigsaw J
8
The Jigsaw model
9
The Jigsaw model
Potts model
10
Toy example
Training image
Jigsaw
Learned using EM graph cuts
11
Dog example
Training image
12
Dog example
Reconstructed image
Epitome reconstruction
Learned segmentation
13
Faces example
128?128 Jigsaw mean

• 64?64 images
• Source Olivetti face database

14
Learning the pieces
Jigsaw J
15
Learning the pieces
Jigsaw J
16
Faces example

• Results of shape clustering on the face images

17
Object recognition (preliminary)

• Trained set 20 street images

Allow patches to deform (as in LayoutCRF, CVPR
2006).
18
Object recognition (preliminary)

• Trained set 20 street images (10 labelled)

Allow patches to deform (as in LayoutCRF, CVPR
2006).
Accuracy improves (1) if you include an
additional 10 unlabelled images when learning the
jigsaw.
19
Work in progress

• Training larger jigsaws on 100s of images
• Incorporating shape clustering into the
  probabilistic model
• Learning additional invariances e.g. to
  illumination
• Object recognition results on MSRC and other
  datasets

20
Conclusions

• Jigsaw model allows learning the shape and
  appearance of objects or object parts in images.
  Can also handle occlusion.
• Clustering shape and appearance much more
  powerful for recognition than appearance alone.
• Can be used as a plug-and-play replacement for
  fixed size patches in any existing patch-based
  system.

21
Thank you
jwinn_at_microsoft.com http//johnwinn.org