WP3 Progress CrossMedia Indexing Michael Oakes University of Sunderland

1
WP3 Progress Cross-Media Indexing Michael
Oakes University of Sunderland

•  CERTH-ITI, CWI, FhG, INRIA 

2
Outline

• WP progress summary
• WP objectives
• Major WP achievements
• Subtask status
• Subtask status against plans
• Subtask technical achievements
• Deliverables status
• Milestone status
• WP Issues
• WP dependencies
• Open issues and Deviations
• Detailed WP future plan
• WP technical goals

3
WP3 Progress summary

• WP3 objectives
• Investigate novel approaches to cross-media
  indexing, particularly in the following areas
• Cross-media representation techniques to
  control the granularity of concepts in a
  cross-media document representation, allowing our
  system to scale to 1000 - 3000 concepts.
• Cross-media fusion, where we aim to develop new
  techniques that allow us to combine knowledge
  from all mono-media features in an optimal way in
  order to produce indexing software that can
  rapidly distinguish between different concepts.

4
WP3 Progress summary

• Major WP3 achievements
• D3.1.1 Literature review looking at the
  state-of-the-art in cross-media indexing
• D3.1.2 Implementation of a state-of-the-art
  cross-media indexing tool (audio, text and visual
  features combination using SVM)

5
Subtask status against plans
6
Subtask technical achievements

• ST 3.1.1 Literature Review
• UoS, CERTH-ITI, CWI, FhG, INRIA all contributed
  
• ST 3.1.2 Implement state-of-the art
• CERTH-ITI, FhG, UoS all contributed
• Audio, text and visual feature extraction
• Replication of MediaMill cross-media indexing
  using the TRECVID data set
• Combination of features using SVM Early and Late
  Fusion.

7
Early and Late Fusion Experiments
8
Experiment 5, Concepts 76-101
9
Deliverables status
10
Milestone status
11
WP Issues

• WP dependencies
• WP3 builds on the mono-media feature sets that
  come out of WP2
• Techniques for scalability will come from WP2
• Dependency on WP1 annotation of ground truth data
  for evaluation
• WP4 Task 4.1 depends on indexing terms identified
  by WP3
• WP3 cross-media indexing subsystem will be
  integrated in the whole Vitalas system in WP6.
• Open issues and Deviations
• Tasks 3.2 to 3.4 are redefined with respect to
  previous versions of the TA.

12
WP3 future plans

• T3.2 Automatic extraction of concept vocabularies
  (T10 to T14)
• T3.3 Algorithms for cross-media fusion (T13 to
  T30) explore the use of machine learning to
  identify concepts which are difficult to
  differentiate within a mono-medium, then
  selecting additional feature sets from the same
  or other media to differentiate them.
• T3.4 Development of Vitalas cross-media indexing
  component (T13 to T30) to be integrated into the
  whole Vitalas framework through WP6.