Part 4: combined segmentation and recognition

1
Part 4 combined segmentation and recognition
Li Fei-Fei
2
Aim

• Given an image and object category, to segment
  the object

Object Category Model
Segmentation
Cow Image
Segmented Cow

• Segmentation should (ideally) be
• shaped like the object e.g. cow-like
• obtained efficiently in an unsupervised manner
• able to handle self-occlusion

3
In this section brief paper reviews

• Jigsaw approach Borenstein Ullman, 2001, 2002
• Concurrent recognition and segmentation Yu and
  Shi, 2002
• Image parsing Tu et al. 2003
• Interleaved segmentation Liebe Schiele, 2004,
  2005
• OBJCUT Kumar et al. 2005
• LOCUS Winn and Jojic, 2005

4
Jigsaw approach Borenstein and Ullman, 2001, 2002
5
Jigsaw approach

• Each patch has foreground/background mask

6
Object-Specific Figure-Ground Segregation
Stella X. Yu and Jianbo Shi, 2002
7
Object-Specific Figure-Ground Segregation
Some segmentation/detection results
Yu and Shi, 2002
8
Image parsing Tu, Zhu and Yuille 2003
9
Image parsing Tu, Zhu and Yuille 2003
10
Implicit Shape Model - Recognition
Liebe and Schiele, 2003, 2005
11
Segmentation

• Interpretation of p(figure) map
• per-pixel confidence in object hypothesis
• Use for hypothesis verification

Liebe and Schiele, 2003, 2005
12
Cows Results

• Segmentations from interest points
• Single-frame recognition - No temporal continuity
  used!

Liebe and Schiele, 2003, 2005
13
OBJCUTshape prior -- Layered Pictorial
Structures (LPS)

• Generative model
• Composition of parts spatial layout

Layer 2
Spatial Layout (Pairwise Configuration)
Layer 1
Parts in Layer 2 can occlude parts in Layer 1
Kumar, et al. 2004, 2005
14
OBJCUT

• Probability of labelling in addition has
• Unary potential which depend on distance from T
  (shape parameter)

T (shape parameter)
Unary Potential Fx(mxT)
mx
m (labels)
my
Object Category Specific MRF
x
y
D (pixels)
Image Plane
Kumar, et al. 2004, 2005
15
OBJCUT Results
Using LPS Model for Cow
In the absence of a clear boundary between object
and background
Segmentation
Image
16
LOCUS model
Shared between images
Class shape p
Class edge sprite µo,so
Deformation field D
Position size T
Different for each image
Mask m
Edge image e
Object appearance ?1
Background appearance ?0
Image
Winn and Jojic, 2005
17
Summary

• Strength
• Explains every pixel of the image
• Useful for image editing, layering, etc.
• Issues
• Invariance issues
• (especially) scale, view-point variations