Unsupervised Image Layout Extraction

1
Unsupervised Image Layout Extraction

• David Liu, Datong Chen, and Tsuhan Chen
• Carnegie Mellon University

2
Segmentation
3
Segmentation

• a compact representation of the interesting
  image data that emphasizes the properties that
  make it interesting. Forsyth Ponce, 2003

• Interestingness should be automatically
  discovered from data, rather than defined a
  priori
• Use multiple images to discover interesting image
  data more objective

4
Overview

• Image representation
• Single topic model
• Multi-topic model
• Incorporating spatial information
• Results

5
Documents and Images Are Analogous
Documents
Images
Topic(Sport, Health, )
Topic (Face, Car, )
Word
Visual word
6
Visual Words
Difference of Gaussian interest point detector
SIFT descriptor (Lowe 99)
Vector quantization
7
Bag of Words Representation
count
words
8
Single Topic Model
count
words
words
w
w
Topic Appearance
zface
zcar
w
w
w
w
9
The Effect of Background Visual Words
10
A Multiple Topic Model --- PLSA

• Hofmann 01, Monay and Gatica-Perez 04, Sivic et
  al. 05, Quelhas et al. 05
• Models complex scenes

z
w
d
Document Characteristic
Topic Appearance
img 1
z
w

• Learning Maximum likelihood estimation using EM
  algorithm

• Unsupervised algorithm

z
w

• Segmentation

d
img 2
z
w
11
Segmentation using Multi-topic Model
12
Spatial Layout

• These two images are the same to the multi-topic
  model! Because it throws away position
  information.

count
count
Same
words
words
13
Spatial Layout

• Use bag of words and positions, rather than
  bag of words.

Different
14
Comparison
s
x
z
w
z
w
s spatial layout
d
d
x position
s
x
doc 1
doc 1
z
w
z
w
z
w
s
x
z
w
d
d
doc 2
s
x
z
w
doc 2
z
w
15
Image Layout Extraction

• EM algorithm
• s hypothesize where the interesting object is
  located at
• A number of S 10 fixed spatial distributions

position
s
x
d
z
w
Nd
s1 to s9
s10
D
appearance
16
Intermediate Results
17
Segmentation Results
18
Blobworld

• Blobworld, PAMI 2002
• Also a probabilistic approach
• Each pixel described by a feature vector
  (including color)
• Gaussian mixture model
• Operates per image. Each image has its own topic
  appearance.

19
Conclusion

• Use multiple images to discover the interesting
  properties within them
• Favor spatially contiguous objects
• Future extensions
• Multiple topics
• More complicated objects
• Special care at segmenting along boundary

20
Contact Information

• For further inquiries, please
  contact
• David Liu
  dliu_at_cmu.edu
• Thank you for your
  attention!