Survey on Photo Clustering

1
Survey on Photo Clustering
Zhihao Qiu qiuzhihao2007_at_163.com Nov.13.2007
2
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

3
Introduction

• Previous work in image browsing, search, and
  management has largely concentrated on solving
  the problem of a user interacting with a large,
  impersonal, possibly annotated image database.
• Unlike interacting with impersonal databases,
  users have very good memories about photographs
  within their personal collection.
• Facilitate organizing and viewing large photo
  collections by automatically dividing photos into
  meaningful episodes or events.

4
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

5
Basic concepts--- Clustering

• Clustering
• partition objects into clusters such
  that objects within a cluster have similar
  characteristics ,while objects in different
  clusters are most distinct from one another.
• Applied areas
• Text Clustering , Image Segmentation, Intelligent
  Video Processing, etc.
• Categories (algorithm)
• Division-Based, Hierarchy-Based, Density-Based,
  Grid-Based, etc.

6
Basic concepts--- EXIF CBIR

• Exchangeable Image File (EXIF) 1
• Time, Location, Focal length, Flash, etc.
• gt Season, place, weather, indoor/outdoor, etc
• Content-based Image Retrieval (CBIR)
• Color, Texture, Shape, etc.
• gt Face Fingerprint Recognition, etc

Metadata
1 http//www.exif.org/
7
Basic concepts--- Event

• Events are naturally associated with specific
  times and places.
• Birthday party
• Vacation
• Wedding
• etc.

8
Basic concepts---Event

• The photos associated with an event often exhibit
  little coherence in terms of both low-level image
  features and visual similarity.
• Generally, photographs from the same event are
  taken in relatively close proximity in time.
• Stanford researchers recently reported that
  organizing photos by time significantly improves
  users performance in a series of retrieval tasks
  2.

2 A. Graham, H. Garcia-Molina, A. Paepcke, and
T.Winograd. Time as the essence for Photo
Browsing Through Personal Digital Libraries.
Proc. Joint Conf. on Digital Libraries, 2002.
9
Basic concepts---Photo clustering systems

• Photo clustering systems
• help users to browse and organize digital photos
• automatic photo clustering in terms of metadata
  or content
• Categories (usage)
• 1) browsing photographs in the form of event
• 2) retrieving similar photograph
• 3) Facilitating annotation for later retrieval,
  etc.

10
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

11
System architecture

• Block Architectural Diagram 3

3 A. Loui and M. Wood. A software system for
automatic albuming of consumer pictures. In Proc.
ACM Multimedia, pages 159162, 1999.
12
System architecture

• The image quality analysis is responsible for
  excluding low-quality images
• underexposure overexposure
• red eye, blur photo, etc
• The event segmentation is responsible for
  determining event and sub-event boundaries
• clustering algorithm
• The layout engine is responsible for determining
  the placement of pictures on the page.

13
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

14
Clustering Techniques

• The various clustering techniques involved in
  clustering photos are listed as below
• Time-based
• Location-based
• Content-based
• TimeLocation-based
• TimeContent-based

15
Clustering -- Time-based

• Clustering along the time axis.
• Detecting noticeable gaps in the creation time.

Time-Based Clustering
16
Clustering --TimeLocation-based

• Clustering based on time.
• Clustering further based on their GPS values.

Time Location-Based Clustering
17
Clustering --TimeContent-based

• Clustering based on time.
• Clustering further based on content.
• Discrete Cosine
• Transform (DCT)
• Cosine Distance
• Measure

content-based similarity matrix
18
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

19
Photo clustering system examples

• PhotoTOC 4
• Using an adaptive local threshold applied to the
  inter-photo time interval.
• Researchers at Kodak 5
• Segment events by clustering time differences
  using the two class K-means algorithm and
  content-based post-processing.
• FXPAL Photo Application6
• Present similarity-based methods to cluster
  digital photos by time and image content.

20
1.PhotoTOC

• After sorting photos by time order.
• Where
• gi is the time difference between picture i and
  picture i1
• gN is considered a gap between events if it is
  much longer than a local log gap average
• K is a suitable threshold (empirically K log(17)
  )
• d is a window size (chosen to be d 10).

21
2. Researchers at Kodak

• Event segmentation algorithm

22
1st level cluster events by date/time

• Date/time clustering algorithm

23
2nd level cluster events by image content
Event clustering using block-based histogram
correlation.
24
3.FXPAL Photo Application

• Firstly, computing similarity matrices SK.

temporal similarity matrix
timestamp
DCT feature
content-based similarity matrix
25
Secondly, computing novelty scores.
peak
peak
similarity matrices SK for K 1000 (a), K
10000 (b), and K 100000 (c) minutes. Panels
(d), (e), and (f) show the corresponding novelty
scores
26
Thirdly, computing confidence scores.

• Select event boundary list for K maximizing the
  confidence score.

27
Hierarchical Photo Clustering

• 1. Extract and sort photo timestamps, t1 , . . .
  , tn. and compute
• DCT features V1,. . . ,VN.
• 2.For each K in decreasing order
• (a) Compute the similarity matrix SK
• (b) Compute the novelty score
• (c) Detect peaks in the novelty score.
• (d) Form event boundary list using event
  boundaries from previous iterations and newly
  detected peaks.
• 3. Compute confidence score using list of event
  boundaries,
• B(K) for each K
• 4. Select event boundary list for K maximizing
  the confidence
• score.

28
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

29
Performance evaluation

• The common way to evaluate effectiveness is by
  using variables called precision, recall and
  F-score
• F-score is the combined average measure of
  precision and recall
• F-score (2 x precision x recall)
  / (precision recall)
• The higher the percentage in both recall and
  precision, the better the performance of the
  system is.

30
Outline

• Introduction
• Basic concepts
• System architecture
• Clustering techniques
• Photo clustering system examples
• Performance evaluation
• Conclusion

31
Conclusion Future work

• The related research largely focuses on
  clustering based on the time, location and
  content, but treats each in an independent way.
• Incorporate multimodal metadata (time, content
  and camera parameters) and study their latent
  relationships, and consider more semantic
  features such as faces.
• Adopt or proposal a more suitable clustering
  algorithm to develop our system.

32
Reference

• 1 http//www.exif.org/
• 2 A. Graham, H. Garcia-Molina, A. Paepcke, and
  T.Winograd. Time as the essence for Photo
  Browsing Through Personal Digital Libraries.
  Proc. Joint Conf. on Digital Libraries, 2002.
• 3 A. Loui and M. Wood. A software system for
  automatic albuming of consumer pictures. In Proc.
  ACM Multimedia, pages 159162, 1999.
• 4 Platt, J., Czerwinski, M., and Field, B.
  (2002). PhotoTOCAutomatic Clustering for
  Browsing Personal Photographs.Microsoft Research
  Technical Report MSRTR-2002-17.
• 5 A. Loui and A. Savakis, Automatic image
  event segmentation and quality screening for
  albuming applications, in Proc. IEEE Int. Conf.
  Multimedia and Expo, New York, July 30Aug. 2
  2000.
• 6 M. Cooper, J. Foote, A. Girgensohn, and L.
  Wilcox, Temporal Event Clustering for Digital
  Photo Collections, In Proceedings of ACM
  Multimedia,2003.

33
Comments and Suggestions

• Thank you!