SemiAutomatic Image Annotation

1
Semi-Automatic Image Annotation

• Liu Wenyin, Susan Dumais, Yanfeng Sun,
• HongJiang Zhang, Mary Czerwinski
• and Brent Field
• Microsoft Research

2
Outline

• Introduction What, Why, and How
• Our Approach
• Semi-Automatic
• Processes and Algorithms
• Automated Performance Evaluation
• Usability Studies
• Concluding Remarks

3
What it is and Why

• Image Annotation is a process of labeling images
  with keywords to describe semantic content
• For image indexing and retrieval in image
  databases
• Annotated images can be found more easily using
  keyword-based search

4
Image Annotation Approaches

• Totally Manual Labeling (Gong et al., 1994)
• Enter keywords when image is loaded/registered/bro
  wsed
• Accurate but labor-intensive, tedious, and
  subjective
• Direct Manipulation Annotation (Shneiderman and
  Kang 2000)
• Drag and drop keywords (from a predefined list )
  onto image
• Still manual, also limited to predefined keywords
  (cant be many)
• Automatic Approaches Efficient but less
  reliable and not always applicable compared to
  human annotation---how to grab this when no text
  context?
• By Image Understanding/Recognition (Ono et al.
  1996)
• By Associating with environmental text (Shen et
  al. 2000 Srihari et al. 2000 Lieberman 2000)

5
Our Proposed Approach

• Semi-Automatic Approach
• User provides initial query and relevance feed
  back.
• Feedback used to semi-automatically annotate
  images
• Trade-off between manual and automatic
• Achieve both accuracy and efficiency
• Increase productivity
• Employ Content-Based Image Retrieval (CBIR), text
  matching, and Relevance Feedback (RF)

6
CBIR and RF Process and Framework
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
Algorithms for Matching

• Visual Similarity Measurement
• Features color histogram/moments/coherence,
  Tamura coarseness, pyramid wavelet texture, etc
• Distance model Euclidean distance
• Semantic (Keywords) Similarity Measurement
• Features keyword vectors, TFIDF
• Metrics dot product and cosine normalization
• Overall similarity weighted average of the above
  two

11
Algorithms to Refine Search

• Image Relevance Feedback Algorithms
• There are many algorithms can be used
• Cox et al. (1996)
• Rui and Huang (2000)
• Vasconcelos and Lippman (1999)
• Lu et al. 2000 is employed in MiAlbum for text
  and images
• Modified Rocchios Formula
• Uses both semantics (keywords) and image-based
  features during relevance feedback

12
Semi-Automatic Annotation During Relevance
Feedback

• In each keyword-query search cycle
• When positive and negative examples provided,
• Increase the weight of the keyword for all
  positive examples
• Decrease the weight of the keyword for all
  negative examples
• Relevance feedback algorithm refines and puts
  more relevant images in top ranks for further
  selection as positive examples
• Repeat the feedback process

13
Possible Future Automatic Annotation

• When a new image is added
• Find top N similar images using image metrics
• Most frequent keywords among annotations of these
  top N similar images are potential annotations,
  and could be automatically added with low weight
  or presented to user as potential annotations
• TBD--Need to be confirmed in further RF process

14
Automated Performance Evaluation

• Test Ground Truth Database
• 12,200 images in 122 categories from Corel DB
• Category name is ground truth annotation
• Automatic Experimental Process
• Use category name as query feature for image
  retrieval
• Among first 100 retrieved images, those belonging
  to this category are used as positive feedback
  examples others as negative
• Performance Metrics
• Retrieval accuracy and annotation coverage

15
Image retrieval accuracy and annotation coverage
16
Usability Studies

• Objectives
• 2 studies examined overall usability of MiAlbum
• The usability of the semi-automatic annotation
  strategy
• Tasks
• Import pictures, annotate pictures, find
  pictures, and use relevance feedback
• Questionnaires including but not limited to
• Overall ease of entering annotations for images
• Impact of annotation on ease of searching for
  images
• Satisfaction of search refinement relevance
  feedback

17
(No Transcript)
18
(No Transcript)
19
Questionnaire Results

• Overall ease of entering annotations 5.6/7.0
• Ease to search annotated photos 6.3/7.0
• Intuitiveness of refining search 4.1/7.0
• Other Comments
• Positive on semi-automatic (1) When using the
  up and down hands the software automatically
  annotated the photos chosen. (2) The ability to
  rate pictures on like/dislike and have the
  software go from there.
• Negative difficulties in understanding the
  feedback process and how the matching algorithm
  operated.

20
Concluding Remarks

• A Semi-automatic Annotation Strategy Employing
• Available image retrieval algorithms and
• Relevance feedback
• Automatic Performance Evaluation
• Efficient compared to manual annotation?
• More accurate than automatic annotation
• Usability Studies
• Preliminary usability results are promising
• Need to improve the discoverability of the
  feedback process and the underlying matching
  algorithm