How to Determine Atomic Rankings

1
How to Determine Atomic Rankings?

• Content-based Retrieval of Multimedia

William Conner, CS 497 KCC Context Survey
2
Definitions

• Atomic ranking
• A single, indivisible ranking of objects from
  one subsystem (i.e., no combining functions used)
• e.g., color subsystem, texture subsystem
• Content-based retrieval of multimedia
• Querying multimedia based on its content rather
  than its meta-data
• e.g., searching for a song based on its melody
  rather than its title

3
Relevance to Top-k

• Many attributes of multimedia content are
  inherently fuzzy
• e.g., redness, fine texture, loudness
• Examples cited in core stage papers
• Fagin mentioned IBMs Garlic and QBIC
• Nepal and Ramakrishna mentioned CHITRA
• Will become even more apparent later

4
Taxonomy

• Classification of the various multimedia
  retrieval techniques
• Dimensions
• Type of multimedia
• User interface
• Similarity measurement used

5
Types of Multimedia

• Image retrieval
• Music retrieval
• Video retrieval

6
User Interfaces

• Query by meta-data
• Filename, captions, title, etc.
• e.g., Yahoo, Google, p2p file sharing
• Not the focus of this context survey
• Query by content
• User creates query (sketching, humming, etc.)
• User selects example (image, song, etc.)
• User selects pre-defined query

7
Measuring Similarity

• Direct measurement of low-level features
• Multimedia retrieval algorithms operate in the
  same multimedia type as the objects being
  compared
• Indirect measurement of low-level features
• Multimedia retrieval algorithms transform
  low-level features into another multimedia type
  and plug in that types retrieval algorithm
• Incorporating user information
• Considers user feedback for future queries

8
Representative Approaches

• Rest of presentation will discuss some
  representative approaches encountered thus far in
  my survey
• Each approach will be classified according to the
  taxonomy presented (there will be some overlap)
• High-level details of algorithms discussed

9
Color Histograms

• Type of multimedia images, video
• User interface creation, selection
• User can sketch query image or select colors
• User can provide example query image
• Similarity measurement direct
• Very similar to Euclidean distance where each
  color is treated as one dimension

10
Color Histograms

• Global color histogram
• One large histogram is created for an entire
  image
• Not very good at discriminating images
• Local color histograms
• Divide an image into cells and create a histogram
  for each cell
• Better than global approach

11
Color Histograms
Example Local Histograms
Example Image
12
Codebook

• Type of multimedia images
• User interface selection
• User provides example query image
• Similarity measurement indirect
• Reduces image retrieval problem to text retrieval
  problem and applies document similarity
  measurement techniques

13
Codebook

• Partitions all the images in a database into
  blocks (or cells)
• Uses a training set of blocks and a clustering
  algorithm to generate a codebook of
  representative blocks for database
• Codebook of representative blocks acts as a
  dictionary for images

14
Codebook

• Images in the database are represented by a
  matrix of indices from the codebook
• Each block is replaced by closest match
• Query images are also partitioned and have their
  blocks replaced
• Images become a sequence of keyblocks (which are
  like keywords for a document)
• Text retrieval techniques can then be used

15
Codebook
16
Objects and Spatial Relationships

• Type of multimedia images
• User interface creation
• User diagrams spatial relationships of color
  regions
• Similarity measurement direct
• Compare low-level image features of
  objects/regions and their spatial relationships
  to each other

17
Objects and Spatial Relationships

• Identify similar objects/regions based on color
  and other low-level image features
• These similarity measures are relatively
  inexpensive
• Prune images that do not contain objects
• Measure spatial relationships between objects in
  remaining candidate images
• Compare spatial relationships to query

18
Objects and Spatial Relationships
19
Melodic Contours

• Type of multimedia music
• User interface creation
• User provides example query by humming
• Similarity measurement indirect
• Reduces music retrieval problem to finding best
  string matches

20
Melodic Contours

• Sequence of relative differences in pitch between
  successive notes
• Feature that humans use to distinguish melodies
  (i.e., most familiar part of a song)
• Songs in database and humming are converted into
  a sequence of symbols that represent pitch
  differences
• e.g., U,D,S (basic) or u,U,d,D,S (enhanced)

21
Melodic Contours

• Songs in database are pre-processed
• Approximate string matching
• Brute force search for all substrings
• Could improve performance by restricting search
• e.g., Search only prefixes and/or suffixes

22
Melodic Contours
Note Sheet music appeared in Zhu and Shasha
2003
23
Time Series

• Type of multimedia music
• User interface creation
• User provides example query by humming
• Similarity measurement direct
• Pitch is viewed as a time series and difference
  time series are compared to each other

24
Time Series

• Addresses the problem with error-prone note
  segmentation
• Used by melodic contour
• Difficulty determining discrete notes
• Uses dynamic time warping
• Accounts for variance in absolute pitch, tempo,
  relative pitch, and local time variation

25
Time Series
Note Sheet music and times series appeared in
Zhu and Shasha 2003
26
Representative Frames

• Type of multimedia video
• User interface creation, selection
• User sketches an image or selects colors
• User selects example query image
• Similarity measurement indirect
• Reduces video retrieval problem to finding
  similar images

27
Representative Frames

• Break video up into clips called shots
• Same scene, single camera operation, event
• Shot (or boundary) detection algorithms
• Generate representative frame for each shot
• Representative frames represent videos
• Returned as query result for browsing
• R-frames could be chosen from the clip or
  generated as an average of frames in shot

28
Representative Frames
Frame 3
Frame 2
R-frame
Frame 1
Frame 0
29
Specific Events

• Type of multimedia video
• User interface selection
• User can only select pre-defined events
• e.g., all shots on goal in a soccer game
• Very limited
• Similarity measurement direct
• Maps low-level video features to high-level
  semantics

30
Specific Events

• Basketball example
• Goal detection (i.e, a shot that was made)
• low-level feature / high-level meaning
• Motion vector change / shot was attempted
• Loudness in audio / crowd cheering
• Text appearance / scoreboard display after goal
• Sports channels can easily retrieve the most
  exciting clips from one or more basketball videos

31
Specific Events
32
ShotRank

• Type of multimedia video
• User interface selection
• User does not really form query
• User browses most interesting representative key
  frames chosen from video clips
• Similarity measurement direct, user info
• Considers both low-level video features and user
  logs from previous users

33
ShotRank

• Measure interestingness and importance of video
  shots for a particular video
• Playback most interesting 2 minutes of clips
• Reorganize key frame browsing order
• Video features link shots to each other
• e.g., Two similar shots or consecutive shots
• User log links users to shots
• e.g., User visits (or interacts) with a video clip

34
ShotRank

• The importance of a shot is determined by
  importance of shots that link to it and engaged
  users that visit it
• The engagement of a user (i.e., a users
  importance) is determined by the importance of
  shots that the user visits
• Sound familiar? Hence, the name.

35
ShotRank

• s vector of ShotRank of all shots of a video
  clip
• u vector of user log importance (or UserRank)
• A adjacency matrix between shots
• V visiting matrix between users and shots
• ß tuning parameter
• s ß AT s (1 - ß) VT u
• u (1 - ß) V s

36
ShotRank

• By the way .
• Developed in large part by Bin Yu during a summer
  internship
• UIUC student from the MONET group who interned at
  Microsoft Research Asia
• Appearing in MM03 in November

37
QBIC Demo

• Query by Image Content
• Developed at IBM
• Image and video retrieval
• The State Hermitage Museum Digital Collection
• Select example color histogram