Using Relevance Feedback in Multimedia Databases

1
Using Relevance Feedback in Multimedia Databases

• Chotirat Ann Ratanamahatana
• Eamonn Keogh

7th International Conference on VISual
Information Systems at 10th International
Conference on Distributed Multimedia
Systems September 9, 2004
2
Roadmap

• Time series in multimedia databases and their
  similarity measures
• Euclidean distance and its limitation
• Dynamic time warping (DTW)
• Global constraints and R-K Band
• Relevance Feedback and Query Refinement
• Experimental Evaluation
• Conclusions and future work

3
What are Time Series

• A collection of observations made sequentially in
  time.
• People measure things
• and thingschange over time

• Their blood pressure
• George Bush's popularity rating
• The annual rainfall in San Francisco
• The value of their Google stock

4
Time Series in Multimedia Databases
Image data may best be thought of as time series
5
Image to Time Series
6
Video to Time Series
Steady pointing
Hand moving to shoulder level
Hand moving down to grasp gun
Hand moving above holster
Hand at rest
7
Time Series in Multimedia Databases
Video
George Washingtons Manuscript
8
Classification in Time Series
Pattern Recognition is a type of supervised
classification where an input pattern is
classified into one of the classes based on its
similarity to these predefined classes.
9
Euclidean Distance Metric
Given 2 time series Q q1, , qn and C
c1, , cn their Euclidean distance is defined
as
Q
C
10
Limitations of Euclidean Metric
Very sensitive to some distortion in the data
Training data consists of 10 instances from each
of the 3 classes
Perform a 1-nearest neighbor algorithm, with
leaving-one-out evaluation, averaged over 100
runs.
Euclidean distance Error rate 29.77
DTW Error rate 3.33
11
Dynamic Time Warping (DTW)
Euclidean Distance One-to-one alignments
Time Warping Distance Non-linear alignments are
allowed
12
How Is DTW Calculated? (I)
Warping path w
13
How Is DTW Calculated? (II)
Each warping path w can be found using dynamic
programming to evaluate the following recurrence
where ?(i, j) is the cumulative distance of the
distance d(i, j) and its minimum cumulative
distance among the adjacent cells.
14
Global Constraints (I)

Prevent any unreasonable warping
Sakoe-Chiba Band
Itakura Parallelogram
15
Global Constraints (II)
A Global Constraint for a sequence of size m is
defined by R, where Ri d 0 ? d ? m, 1
? i ? m. Ri defines a freedom of warping above
and to the right of the diagonal at any given
point i in the sequence.
Ri
Itakura Parallelogram
Sakoe-Chiba Band
16
Ratanamahatana-Keogh Band (R-K Band)
Solution we create an arbitrary shape and size
of the band that is appropriate for the data we
want to classify.
17
How Do We Create an R-K Band?
First Attempt We could look at the data and
manually create the shape of the bands.
(then we need to adjust the width of each band as
well until we get a good result)
100 Accuracy!
18
Learning an R-K Band Automatically
Our heuristic search algorithm automatically
learns the bands from the data. (sometimes, we
can even get an unintuitive shape that give a
good result.)
100 Accuracy as well!
19
R-K Band Learning With Heuristic Search
20
R-K Band Learning in Action!
21
Classification Examples with R-K Bands
Error rate
Euclidean 32.13
DTW 10 4.52
R-K Bands 0.9
22
Face Classification
23
Relevance Feedback

• A well-known and effective method in improving
  the query performance, especially in text-mining
  domains.
• Refining the query based on users reaction
• Only relatively little research has been done on
  relevance feedback in images or multimedia data.

24
Query Refinement

• Averaging a collection of time series using DTW,
  according to their weights and warping (DTW)
  alignments.

25
Experiment Datasets

1. Gun Problem
2. Leaf Dataset
3. Handwritten Word Spotting data

26
Experimental Design

• Given an initial query, we measure the precision
  and recall for each round of the relevance
  feedback retrieval.
• Show the 10 best matches (k-nearest neighbors).
• User ranks each result.
• Accumulatively build the training set.
• Learn an R-K band according to the current
  training data.
• Generate a new query (query refinement), and
  repeat.

27
(No Transcript)
28
Results Gun
29
Results Leaf
30
Results Wordspotting
31
Conclusions

• Different shapes and widths of the band
  contributes to the classification accuracy /
  precision.
• We have shown that incorporating R-K Band into
  relevance feedback can reduce the error rate in
  classification, and improve the precision at all
  recall levels in video and image retrieval.

32
Future Work

• Investigate other choices that may make envelope
  learning more accurate.
• Heuristic functions
• Search algorithm (refining the search)
• Is there a way to always guarantee an optimal
  solution?
• Examine the best way to deal with multi-variate
  time series for more complex data.
• Explore other utilities of R-K Band and relevance
  feedback, specifically on real-world problems
  music, bioinformatics, biomedical data, etc.

33
Questions?
Contact ratana_at_cs.ucr.edu eamonn_at_cs.ucr.edu
Homepage http//www.cs.ucr.edu/ratana All
datasets are publicly available at
UCR Time Series Data Mining Archive
http//www.cs.ucr.edu/eamonn/TSDMA