TEMPORAL EVENT CLUSTERING FOR DIGITAL PHOTO COLLECTIONS

1
TEMPORAL EVENT CLUSTERING FOR DIGITAL PHOTO
COLLECTIONS

• Matthew Cooper, Jonathan Foote, Andreas
  Girgensohn, and Lynn Wilcox
• ACM Multimedia
• ACM Transactions on Multimedia Computing ,
  Communications and Application

2
OUTLINE

• Introduction
• Feature extraction
• Clustering techniques
• Supervised event clustering
• Unsupervised event clustering
• Clustering goodness criteria
• Experimental result
• Conclusion

3
Introduction

• Users navigate their photos
• Temporal order
• Visual content
• Associate time and content with the notion of a
  specific event
• Photos associated with an event often exhibit
  little coherence in terms of either low-level
  image features or visual similarity
• photographs from the same event are taken in
  relatively close proximity in time

4
Basic concepts--- Event

• Events are naturally associated with specific
  times and places.
• Birthday party
• Vacation
• Wedding

5
Basic concepts--- EXIF CBIR
Metadata

• Exchangeable Image File (EXIF)
• Time, Location, Focal length, Flash, etc.
• gt Season, place, weather, indoor/outdoor,etc
• Content-based Image Retrieval (CBIR)
• Color, Texture, Shape, etc.
• gt Face Fingerprint Recognition,etc

6
FEATURE EXTRACTION

• EXIF headers are processed to extract the
  timestamp
• The N photos in the collection are then ordered
  in time so the resulting timestamps,
• tnn 1, . . . , N,satisfy t1 t2 tN
• Time difference between indices (photos) is
  nonuniform

7
FEATURE EXTRACTION

• Computing similarity matrices SK

temporal similarity matrix
8
FEATURE EXTRACTION

• Computing similarity matrix
• low-frequency discrete cosine transform (DCT)
  coefficients from each photo using the cosine
  distance measure

content-based similarity matrix
9
FEATURE EXTRACTION

• computing novelty scores

peaks in the novelty scores cluster boundaries
between contiguous groups of similar photos
K1000
K10000
K100000
10
CLUSTERING TECHNIQUES

• Supervised event clustering
• Based on LVQ
• Unsupervised event clustering
• Scale-space analysis of the raw timestamp data
• Temporal Similarity Analysis
• Combining Time and Content-Based Similarity

11
Supervised event clustering

• Let K take M values K K1, . . . , KM
• Define the M N matrix N(j,i) ?Kj (i)
• , where
• Based on LVQ (Learning Vector Quantization)
• Kohonen 1989
• LVQ codebook discriminates between the two
  classes event boundary and event interior.
• The codebook vectors for each class are used for
  nearest-neighbor classification of the novelty
  features for each photo in the test set.

12
Supervised event clustering

• In the training phase, a codebook is calculated
  using an iterative procedure
• Each step
• Nearest codebook vector to each training sample
  is determined
• shifted toward or away the training sample

If Nx and Mc are in the same class
If Nx and Mc arent in the same class
13
Supervised event clustering

• ALGORITHM 1 (LVQ-BASED PHOTO CLUSTERING).
• (1) Calculate novelty features from labeled
  sorted training data for each scale K
• (i) compute the similarity matrix SK
• (ii) compute the novelty score ?K
• (2) Train LVQ using the iterative procedure
• (3) Calculate novelty features for the testing
  data for each K
• (i) compute the similarity matrix SK
• (ii) compute the novelty score ?K
• (4) Classify each test samples novelty features
  Ni using the LVQ codebook and the
  nearest-neighbor rule.

14
Unsupervised event clustering

• scale-space analysis
• operate on the raw timestamps
• T0 t1, . . . , tN so that T0(i) ti
• ALGORITHM 2 (SCALE-SPACE PHOTO CLUSTERING).
• (1) Extract timestamp data from photo collection
  t1, . . . , tN.
• (2) For each s in descending order
• (i) compute Ts
• (ii) detect peaks in Ts , tracing peaks from
  larger to smaller scales (decreasing s).

15
UNSUPERVISED EVENT CLUSTERING

• Temporal Similarity Analysis
• Locate peaks at each scale by analysis of the
  first difference of each novelty scores ?K ,
  proceeding from coarse scale to fine (decreasing
  K)
• To build a hierarchical set of event boundaries,
  we include boundaries detected at coarse scales
  in the boundary lists for all finer scales.

checkerboard kernel used to compute the novelty
features
16
UNSUPERVISED EVENT CLUSTERING

• Combining Time and Content-Based Similarity
• constructed a content-based matrix SC using
  low-frequency DCT features and the cosine
  distance
• if ti-tj gt
  48h
• others
• if ti-tj gt
  48h
• others

17
CLUSTERING GOODNESS CRITERIA

• Peak detection at each scale K results in a
  hierarchical set of candidate boundaries
• Subset must be selected to define the final event
  clusters
• Three different automatic approaches
• Similarity-Based Confidence Score
• Boundary Selection via Dynamic Programming
• BIC-Based Boundary Selection

18
Similarity-Based Confidence Score

• Detected boundaries at each level K,
• BK b1, . . . , bnK ,
• indexed by photo BK ? 1, . . . , N

average intercluster similarity between photos in
adjacent clusters
average intracluster similarity between the
photos within each cluster
19
Boundary Selection via Dynamic Programming

• Reduced complexity
• Begin with the set of peaks detected from the
  novelty features at all scales
• Cost of the cluster between photos bi and bj

20
Boundary Selection via Dynamic Programming

• Optimal partitions with m boundaries based on the
  optimal partition with m-1 boundaries
• First, optimal partitions are computed with two
  clusters
• EF (j,m) is the optimal partition of the photos
  with cardinality m

21
Boundary Selection via Dynamic Programming

• Number of clusters increases, the total cost of
  the partition decreases monotonically
• Selecting the optimal number of clusters, M,
  based on the total partition cost

22
BIC-Based Boundary Selection

• This method is based on the Bayes information
  criterion (BIC) Schwarz 1978
• Assumption
• timestamps within an event are distributed
  normally around the event mean

Log-likelihood of the single segment model and
the penalty term
log-likelihood of the two segment model
? is 2 ,since we describe each segment using the
sample mean µ,and variance, s2
23
BIC-BASED BOUNDARY SELECTION

• Employ the hierarchical coarse-to-fine approach
• At each scale, we test only the newly detected
  boundaries (undetected at coarser scales)
• Add the boundaries for which the left side
  exceeds the right side

24
ALGORITHM 3 (SIMILARITY-BASED PHOTO CLUSTERING)

• (1) Extract and sort photo timestamps, t1, . . .
  , tn.
• (2) For each K in decreasing order
• (i) compute the similarity matrix Sk
• (ii) compute the novelty score ?K
• (iii) detect peaks in ?K
• (iv) form event boundary list using event
  boundaries from previous iterations and newly
  detected peaks
• (3) Determine a final boundary subset of
  collected boundaries over all scales considered
  according to one of the methods
• (a) the confidence score
• (b) the DP boundary selection approach
• (c) the BIC boundary selection approach

25
EXPERIMENTAL RESULT

• Run Times for Different Size Photo Collections
• The times are in seconds
• No Conf. indicates times for Steps 1 and 2
• BIC peak selection (BIC)
• Dynamic programming peak selection (DP)
• similarity-based peak selection (Conf.)
• Doubling the number of photos(N),the time for the
  segmentation step(No Conf.) increases linearly,
  while including the confidence measure (Conf.)
  incurs a polynomial cost.

26
EXPERIMENTAL RESULT

• Compare the event clustering performance of
  eleven systems on two separate photo collections
• Collection I consists of 1036 photos taken over
  15 months
• Collection II consists of 413 photos taken over
  13 months
• The first four algorithms in
• the table are hand-tuned
• to maximize performance.
• The remaining algorithms
• are fully automatic.

27
EXPERIMENTAL RESULT

• Precision indicates the proportion of falsely
  labeled boundaries
• Recall measures the proportion of true boundaries
  detected
• The F-score is a composite of precision and
  recall

28
EXPERIMENTAL RESULT
29
EXPERIMENTAL RESULT

• The adaptive-thresholding algorithms exhibited
  high recall and low precision on both test sets,
  even with manual tuning
• Scale-space and the two similarity-based
  approaches demonstrated more consistent
  performance and traded off precision and recall
  more evenly

30
CONCLUSION

• Employed the automatic temporal similarity-based
  method
• Does not rely on preset thresholds or restrictive
  assumptions
• As photo collections with location information
  become available, we hope to extend our system to
  combine temporal similarity, content-based
  similarity, and location-based similarity.
• The automatic methods performance exceeded that
  of manually tuned alternatives in our testing,
  and have been well received by users of our photo
  management application.