Survey on the methods of personbased photo clustering

1
Survey on the methods of person-based photo
clustering

• 
  
  Zhuojia LIANG
• 
  liangzj163_at_163.com
• 
  Nov.30.2007
• Software Engineering Laboratory
• Department of Computer Science, Sun Yat-sen
  University

2
Outline

• Introduction of person-based photo clustering
• Basic concept of Automation Face recognition
• Person-based photo clustering examples
• Conclusion

3
Introduction of person-based photo clustering

• person-based photo clustering
• Partition photos the same person
  presents in into clusters
• With vastly growing number of family photos,
  person-based photo clustering facilitates
  organizing and viewing someones photos

4
Outline

• Introduction of person-based photo clustering
• Basic concept of Automation Face recognition
• Person-based photo clustering examples
• Conclusion

5
Basic concept of Automation Face recognition (AFR)

• AFR system
• Face Detection
• Feature Extraction
• Face Recognition
• Adverse factors in AFR

6
A generic AFR system
Figure 1Configuration of a generic AFR system.1
1 W. ZHAO, R. CHELLAPPA, P. J. PHILLIPS AND A.
ROSENFELD Face Recognition A Literature
Survey ACM Computing Surveys, Vol. 35, No. 4,
December 2003, pp. 399458.
7
Introduction of face detection

• Given a single image, the goal of face detection
  is to identify all image regions which contain a
  face regardless of its three-dimensional
  position, orientation, and lighting
  conditions.2
• Four categories of face detection methods
• 1. Knowledge-based methods.
• 2. Feature invariant approaches.
• 3. Template matching methods.
• 4. Appearance-based methods.

2 Ming-Hsuan Yang, David J. Kriegman and
Narendra Ahuja Detecting Faces in Images A
Survey IEEE TRANSACTIONS ON PATTERN ANALYSIS
AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY
2002
8
The approach of face detection

• Table 1 Categorization of Methods for Face
  Detection in a Single Image2

2 Ming-Hsuan Yang, David J. Kriegman and
Narendra Ahuja Detecting Faces in Images A
Survey IEEE TRANSACTIONS ON PATTERN ANALYSIS
AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY
2002
9
Feature Extraction

• Normalization
• The image should be normalized to some
  extent. Normalization is usually a combination of
  linear translation, rotation and scaling,
  although the elastic matching method includes
  spatial transformations. 3
• Three types of feature extraction methods can be
  distinguished 1
• (1) generic methods based on edges, lines,
  and curves
• (2) feature-template-based methods that are
  used to detect facial features such as eyes
• (3) structural matching methods that take
  into consideration geometrical constraints on the
  features.

1 W. ZHAO, R. CHELLAPPA, P. J. PHILLIPS AND A.
ROSENFELD Face Recognition A Literature
Survey ACM Computing Surveys, Vol. 35, No. 4,
December 2003, pp. 399458. 3 William A.
Barrett A Survey of Face Recognition Algorithms
and Testing Results 1998 IEEE
10
Introduction of face recognition

• Recognition involves a match between the products
  of structural encoding and previously stored
  structural codes describing the appearance of
  familiar faces.4
• Three categories of face recognition methods1
• 1. Holistic matching methods.
• 2. Feature-based (structural) matching
  methods.
• 3. Hybrid methods.

1 W. ZHAO, R. CHELLAPPA, P. J. PHILLIPS AND A.
ROSENFELD Face Recognition A Literature
Survey ACM Computing Surveys, Vol. 35, No. 4,
December 2003, pp. 399458. 4 Bruce, Vicki
Young, Andy Understanding face recognition
British Journal of Psychology. 1986 Aug Vol 77(3)
305-327
11
The approach of face recognition

• Table2 Categorization of Still Face Recognition
  Techniques1

12
Adverse factors in AFR

• Adverse factors in AFR include lighting
  conditions, noise in the image, facial expression
  variations, glasses, hirsute changes, and
  posture. 3
• Face recognition continues to be a challenging
  topic in computer vision research. Most
  algorithms perform well under a controlled
  environment, while in the scenario of family
  photo management, the performance of face
  recognition algorithms becomes unacceptable due
  to difficult lighting/illumination conditions and
  large head pose variations. 5

3 William A. Barrett A Survey of Face
Recognition Algorithms and Testing Results
1058-6393/98 1998 IEEE 5 Y Tian, W Liu, R Xiao,
F Wen, X Tang A Face Annotation Framework with
Partial Clustering and Interactive Labeling
Computer Vision and Pattern Recognition, 2007.
CVPR'07. IEEE
13
Outline

• Introduction of person-based photo clustering
• Basic concept of Automation Face recognition
• Person-based photo clustering examples
• Conclusion

14
Person-based photo clustering examples

• I choose four typical examples
• FACE ANNOTATION FOR FAMILY PHOTO ALBUM MANAGEMENT
  (2003)
• Automatic Face-based Image Grouping for Albuming
  (2003)
• Leveraging Context to Resolve Identity in Photo
  Albums (2005)
• EasyAlbum An Interactive Photo Annotation System
  Based on Face Clustering and Reranking (2007)

15
Example 1 FACE ANNOTATION FOR FAMILY PHOTO ALBUM
MANAGEMENT6 (1/4)

• The method of person-based photo clustering
• When a face is detected in a photo and
  a user would like to annotate faces, the system
  will calculate a candidate list of names for the
  user to annotate the detected face, according to
  the similarities to the annotated faces. The user
  might accept the recommendation, or set a new
  name to that face. To further facilitate the face
  annotation, the user can specify a face and
  search the similar faces in the album, then
  annotate multiple faces in a batch way.
• Based on the time constrain and the
  low level features of the images, the photos can
  be grouped into several events. Usually the
  person in the same event will wear the same
  clothes. Combined with the date constraints, the
  body-related features will play an important role
  in the recognition.

6 L. Chen, B. Hu, L. Zhang, M. Li, and H.
Zhang. Face annotation for family photo album
management. International Journal of Image and
Graphics, pages 114, 2003.
16
Example 1 FACE ANNOTATION FOR FAMILY PHOTO ALBUM
MANAGEMENT (2/4)
Face Annotation Framework
Figure. 3. A popup menu in the face annotation
system.
Figure. 2. Framework overview.
17
Example 1 FACE ANNOTATION FOR FAMILY PHOTO ALBUM
MANAGEMENT (3/4)

• Evaluation
• Data set The photo album we used in our
  experiment is a typical family album. There are a
  total of 1707 photos in the album.

Table 4 Comparison of different learning
algorithms.
Table 3 Comparison of different features
Table 5. Body feature versus facial feature.
18
Example 1 FACE ANNOTATION FOR FAMILY PHOTO ALBUM
MANAGEMENT (4/4)
Figure 4. shows the target face and the first
eight resulting photos. The caption below the
photo are the file names and the distance values
between the resulting photos and the target photo
19
Example 2Automatic Face-based Image Grouping for
Albuming 7 (1/4)

• The steps of person-based photo clustering given
  below
• 1. Producing clusters of faces depicting the
  same person
• 2. In an interactive system, user will
  correct clusters and name them.

7 M. Das and A. Loui, "Automatic face-based
image grouping for albuming," Proc. ofIEEE
International Conf on Systems, Man and
Cybernetics, vol. 4, pp. 3726-3731, Oct. 2003
20
Example 2Automatic Face-based Image Grouping for
Albuming (2/4)
Table 3 Top five features selected for
classification by Ada-boost along with the
weights assigned
Any face similarity metric may be used, as
long as it produces a measure of similarity
between any two given faces within each of the
three classes - males, females and babies.
FIGURE 5 Steps in producing clusters of
faces depicting the same person
21
Example 2Automatic Face-based Image Grouping for
Albuming (3/4)

• The clustering follows the steps given below
• 0. Start with graph G containing N face
  nodes (no edges)
• 1. Sort all pair-wise similarity scores in
  decreasing order
• 2. For each Score in the sorted list
• 3. If (score lt threshold) exit, else
• 4. Add edge to G between the pair of face
  nodes with edge weight score

22
Example 2Automatic Face-based Image Grouping for
Albuming (4/4)

• Evaluation
• 1.Data set family photos from two families
  covering a time-span of 5 years, containing 1500
  images (of mainly 9 individuals) and 1200 images
  (of mainly 6 individuals) scanned from film
  negatives.
• 2.Face detection 92 of frontal faces that
  were large enough were detected (this covered
  about 65 of all faces in the images)
• 3. Baby/adult and gender The baby/adult
  classification rate was 92 on the detected faces
  (each family had one predominant baby who
  accounted for a disproportionate fraction of
  faces). The gender classification accuracy was
  89.
• 4.Person clustering 70 of the detected faces
  were clustered (the rest remained as singleton
  nodes). The precision of each cluster, defined as
  the ratio of the number of faces of the
  pre-dominant person and the total number of faces
  in the cluster, was 74.

23
Example 3 Leveraging Context to Resolve Identity
in Photo Albums 8 (1/6)

• The method of person-based photo clustering
• Based on time and location, the system
  automatically computes event and location
  groupings of photos. As the user annotates some
  of the identities of people in their collection,
  patterns of re-occurrence and co-occurrence of
  different people in different locations and
  events emerge. The system uses these patterns to
  generate label suggestions for identities that
  were not yet annotated. These suggestions can
  greatly accelerate the process of manual
  annotation and improve the quality of retrieval
  from the collection.

8 Naaman, M., Yeh, R.B., Garcia-Molina, H.,
Paepcke, A. Leveraging context to resolve
identity in photo albums. In JCDL. (2005) 178187
24
Example 3 Leveraging Context to Resolve Identity
in Photo Albums (2/6)

• We use the following intuitive guidelines
• Popularity. Some people appear more often than
  others.
• Co-occurrence. People that appear in the same
  photos may be associated with each other, and
  have a higher likelihood of appearing together in
  other photos.
• Temporal re-occurrence. Within a specific event,
  there tend to be multiple photos of the same
  person.
• Spatial re-occurrence. People that appear in a
  certain location have an elevated likelihood of
  appearing again in that same location, even
  during different events.

25
Example 3 Leveraging Context to Resolve Identity
in Photo Albums (3/6)

• Estimators (1/2)
• t(s) The time when the photo was taken.
• g(s) The geographic location coordinates where
  the photo was taken.
• L(s) The set of photos belonging to the
  location leaf node that contains photo s.
• E(s) The set of photos taken at the event that
  contains photo s.
• 1
• 2

Table 4. The basic estimators and the set of
photos each estimator considers when ranking
candidates to appear in photo s.
26
Example 3 Leveraging Context to Resolve Identity
in Photo Albums (4/6)

• Estimators (2/2)
• Estimating Co-occurrence PeopleRank
• The PeopleRank estimators aim to harvest
  the relationships between people. Such
  relationships naturally exist in personal photo
  collections due to the human nature of social
  interaction.
• Combining Estimators Padded and Weighted
• Padded When generating Hs, the system will
  choose the first candidates amongst the
  top-ranked candidates by a fine estimator, and
  the rest (padding the list until h candidates are
  found) from broader estimators.
• Weighted we assign a weight to each
  estimator.

27
Example 3 Leveraging Context to Resolve Identity
in Photo Albums (5/6)

• Evaluation

Figure 6. H-Hit rate for various estimators vs.
value of h
28
Example 3 Leveraging Context to Resolve Identity
in Photo Albums (6/6)

• Evaluation

Figure 7. 5-Hit rate or various estimators vs.
size of the important people set I
29
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking9(1/7)

• The method of person-based photo clustering
• Cluster annotation After automatic
  clustering, user can conduct photo/face
  annotation on clusters directly.
• Contextual re-ranking when the user clicks
  one photo/cluster, it indicates his/her attention
  on this one. system re-rank the photos/clusters
  and arrange the similar ones close to the clicked
  photo/cluster.
• Ad hoc annotation In this system users are
  allowed to annotate photos in an ad hoc manner
  when they are browsing or searching.

9. J. Cui, F. Wen, R. Xiao, Y. Tian, and X.
Tang. EasyAlbumAn interactive photo annotation
system based on face clustering and re-ranking.
Proc. CHI 2007. ACM Press, 2007.
30
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (2/7)
Figure 8. System framework
31
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (3/7)
Figure 9. The EasyAlbum Face Labeling User
Interface
32
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (4/7)

• Annotation

Figure 10. Annotation by selecting an existing
name or type a new one
33
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (5/7)

• Smart Cluster Merging and Splitting

Figure 12. After dragging, unselect misclustered
photos before merging with labeled clusters
Figure 11. Drag between FaceGroups in Group View
Area
34
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (6/7)

• Contextual re-ranking

Figure 13. Arrangement of FaceGroups before and
after Re-Ranking.Note that before re-ranking,
there are only two FaceGroups belong to the same
subject of the selected one in the neighborhood,
while after re-ranking, there are four, and all
are positioned near the selected one.
Figure 14. Comparison of orders of Faces before
(top) and after (bottom) Re-Ranking
35
Example 4EasyAlbum An Interactive Photo
Annotation SystemBased on Face Clustering and
Reranking (7/7)

• Evaluation

Figure 15.Scalability of EasyAlbum and Adobe
Photoshop Elements 4.0
36
Outline

• Introduction of person-based photo clustering
• Basic concept of Automation Face recognition
• Person-based photo clustering examples
• Conclusion

37
Conclusion (1/2)

• Some elements to person-based photo clustering
• Social content (time,location) Through the
  social content, we can get event-based cluster
  and estimate the likelihood for each person to
  appear in photo.
• Face detection Get faces.
• Face recognition Compare the faces, and
  calculate the similarity. Through the similarity,
  system can calculate a candidate list of names
  for the user to annotate the detected face,
  cluster or retrieval the photos.
• Visual content (body) In the same event, the
  body-related feature of the same person is always
  same, which is useful for clustering.
• Interaction techniques e.g. annotation, drag
  and drop the photos, merge and split the clusters.

38
Conclusion (2/2)

• Systems target
• How to lower the labor cost barrier for
  manual annotation, facilitate organizing and
  viewing someones photos.
• A good method
• Because each element has its advantage and
  disadvantage, a good method of person-based photo
  clustering is to integrate all elements.

39
References

• 1 W. ZHAO, R. CHELLAPPA, P. J. PHILLIPS AND A.
  ROSENFELD Face Recognition A Literature
  Survey ACM Computing Surveys, Vol. 35, No. 4,
  December 2003, pp. 399458.
• 2 Ming-Hsuan Yang, David J. Kriegman and
  Narendra Ahuja Detecting Faces in Images A
  Survey IEEE TRANSACTIONS ON PATTERN ANALYSIS
  AND MACHINE INTELLIGENCE, VOL. 24, NO. 1, JANUARY
  2002
• 3 William A. Barrett A Survey of Face
  Recognition Algorithms and Testing Results 1998
  IEEE
• 4 Bruce, Vicki Young, Andy Understanding
  face recognition British Journal of Psychology.
  1986 Aug Vol 77(3) 305-327
• 5 Y Tian, W Liu, R Xiao, F Wen, X Tang A Face
  Annotation Framework with Partial Clustering and
  Interactive Labeling Computer Vision and Pattern
  Recognition, 2007. CVPR'07. IEEE
• 6 L. Chen, B. Hu, L. Zhang, M. Li, and H.
  Zhang. Face annotation for family photo album
  management. International Journal of Image and
  Graphics, pages 114, 2003
• 7 M. Das and A. Loui, "Automatic face-based
  image grouping for albuming," Proc. ofIEEE
  International Conf on Systems, Man and
  Cybernetics, vol. 4, pp. 3726-3731, Oct. 2003
• 8 Naaman, M., Yeh, R.B., Garcia-Molina, H.,
  Paepcke, A. Leveraging context to resolve
  identity in photo albums. In JCDL. (2005)
  178187
• 9. J. Cui, F. Wen, R. Xiao, Y. Tian, and X.
  Tang. EasyAlbumAn interactive photo annotation
  system based on face clustering and re-ranking.
  Proc. CHI 2007. ACM Press, 2007.

40
Reference for representative work in table1 (1/2)

• 24 A.J. Colmenarez and T.S. Huang, Face
  Detection with Information-Based Maximum
  Discrimination, Proc. IEEE Conf. Computer
  Vision and Pattern Recognition, pp. 782-787,
  1997.
• 28 I. Craw, D. Tock, and A. Bennett, Finding
  Face Features, Proc. Second European Conf.
  Computer Vision, pp. 92-96, 1992.
• 32 Y. Dai and Y. Nakano, Face-Texture Model
  Based on SGLD and Its Application in Face
  Detection in a Color Scene, Pattern Recognition,
  vol. 29, no. 6, pp. 1007-1017, 1996.
• 79 R. Kjeldsen and J. Kender, Finding Skin in
  Color Images, Proc. Second Intl Conf. Automatic
  Face and Gesture Recognition, pp. 312-317, 1996.
• 86 A. Lanitis, C.J. Taylor, and T.F. Cootes,
  An Automatic Face Identification System Using
  Flexible Appearance Models, Image and Vision
  Computing, vol. 13, no. 5, pp. 393-401, 1995.
• 87 T.K. Leung, M.C. Burl, and P. Perona,
  Finding Faces in Cluttered Scenes Using Random
  Labeled Graph Matching, Proc. Fifth IEEE Intl
  Conf. Computer Vision, pp. 637-644, 1995.
• 89 M.S. Lew, Information Theoretic View-Based
  and Modular Face Detection, Proc. Second Intl
  Conf. Automatic Face and Gesture Recognition, pp.
  198-203, 1996.
• 98 S. McKenna, S. Gong, and Y. Raja, Modelling
  Facial Colour and Identity with Gaussian
  Mixtures, Pattern Recognition, vol. 31,no. 12,
  pp. 1883-1892, 1998.

41
Reference for representative work in table1 (2/2)

• 107 E. Osuna, R. Freund, and F. Girosi,
  Training Support Vector Machines An Application
  to Face Detection, Proc. IEEE Conf. Computer
  Vision and Pattern Recognition, pp. 130-136,
  1997.
• 123 A. Rajagopalan, K. Kumar, J. Karlekar, R.
  Manivasakan, M. Patil,U. Desai, P. Poonacha, and
  S. Chaudhuri, Finding Faces in Photographs,
  Proc. Sixth IEEE Intl Conf. Computer Vision, pp.
  640-645, 1998.
• 128 H. Rowley, S. Baluja, and T. Kanade,
  Neural Network-Based Face Detection, IEEE
  Trans. Pattern Analysis and Machine
  Intelligence,vol. 20, no. 1, pp. 23-38, Jan.
  1998.
• 140 H. Schneiderman and T. Kanade,
  Probabilistic Modeling of Local Appearance and
  Spatial Relationships for Object
  Recognition,Proc. IEEE Conf. Computer Vision and
  Pattern Recognition, pp. 45-51,1998.
• 154 K.-K. Sung and T. Poggio, Example-Based
  Learning for View-Based Human Face Detection,
  IEEE Trans. Pattern Analysis and Machine
  Intelligence, vol. 20, no. 1, pp.39-51, Jan.
  1998.
• 163 M. Turk and A. Pentland,Eigenfaces for
  Recognition, J. Cognitive Neuroscience, vol. 3,
  no. 1, pp. 71-86, 1991.
• 172 J. Yang and A. Waibel, A Real-Time Face
  Tracker, Proc. Third Workshop Applications of
  Computer Vision, pp. 142-147, 1996.
• 178 K.C. Yow and R. Cipolla, Feature-Based
  Human Face Detection, Image and Vision
  Computing, vol. 15, no. 9, pp. 713-735, 1997.

42
Comments and Suggestions

• Thank you!