Direct Video

1
Direct Video Audio Content Search Engine

• IST-2-045081

Advanced search technologies for digital
audio-visual content
2
Introduction

• Divas represents the combined efforts of eight
  companies and institutions to
• design and develop a multimedia search engine
• based on advanced direct video and audio search
  algorithms applied directly on encoded
  (compressed) content.
• IST DIVAS (FP6 IST-2-04582) was officially
  launched at the 1st of January 2007, with a
  duration of 24 months.

3
The Current Situation

• Availability of huge and ever expanding
  distributed repositories of media in various
  formats
• how can a system efficiently and reliably
  identify content fragments captured from various
  streams?
• Only techniques for indexing and searching raw
  (uncompressed) content are available today (and
  text-based techniques)

4
Overcome Metadata Issues

• provides the capability to the user to locate
  captured video and audio feeds with missing
  additional context
• like title, filename, origin, location, service
  provider etc.
• or in situation where metadata based queries are
  inapplicable

5
Avoid Decompression

• Search of multimedia libraries using the
  techniques for uncompressed content is a heavy
  duty/ costly solution
• because for the search each item has to be
  decompressed.

6
Avoid Annotation

• Metadata annotation may be a heavy duty/costly
  solution to content owners.
• A complementary solution should therefore also be
  made available

7
Prospects

• Audio-visual signature/ fingerprint extraction
  directly from compressed resources
• Extend Search Techniques
• By supporting content queries, DIVAS extends the
  state of the art beyond nowadays pursued search
  techniques based on metadata.
• Improve the reliability of audio-visual content
  detection
• By its multimodal (video audio content)
  approach, and by combining the query results
  obtained from both modalities.

8
The Consortium
9
Characterization and direct search of compressed
video

• DIVAS proposes characterization, feature
  extraction and direct search of compressed video
• as opposed to cognitive-level metadata annotation
  from uncompressed video streams.
• Video fingerprinting,
• as envisaged (but not extensively exploited) in
  the MPEG-7 standard is the term approximately
  fitting to our approach.
• DIVAS will pursue
• Mpeg-2 compliant implementation
• a H.264 compliant implementation.

10
Characterization and direct search of compressed
audio

• Already a relatively mature technology on
  uncompressed audio
• Based on the extraction of fingerprints, which
  capture the characteristic features of an audio
  clip.
• These fingerprints are then compared to the
  fingerprint of a query (an audio clip to search
  for).

11
DIVAS Environment

• DIVAS system search techniques incorporate in
  parallel both audio and video based searching.
• In terms of functional decomposition the system
  will address audio and video in a different way.
• DIVAS system utilizes two different engines
• a/generate unique indexes from each clip
• b/search among the aforementioned identifiers,
  providing a match/no match answer to the user.

12
Architectural Goals

• Open architecture
• Future-proof design
• Scalability
• Interoperability
• Expandability
• Modularity

13
High Level Decomposition
14
Functional Overview
15
Approach
Compressed Audio Signal
DIVAS
Conventional
Decoding
Direct conversion into the suitable
time/frequency domain
Conversion to suitable time/frequency domain
Feature Extraction
Speech Recognition
Music Information Retrieval
16
General Aspects
Tool A Content uploading
Result of content search
DIVAS ENGINE
Indexes (fingerprints) DB

Reading
Tool C Administration
Updating
Writing
Tool B Content search
Content index
17
Content features extraction engine
Video/audio/text content
Video/audio/text indexes
Video features extraction engine
Content demultiplexer
Index multiplexer
Multiplexed indexes
Multiplexed Content
Audio features extraction engine
Engine of text/meta features extraction
Plug-ins
Plug-ins
18
Video features extraction engine
Video Decoder (Transcoder)
Features extractor
Video index
Video content
Plug-ins
Plug-ins
19
Searching
CONTENT FEATURES EXTRACTION ENGINE
COMPARISON ENGINE
Query content
Query index
Search result
20
Comparison engine
Query index
Query index
Search result
Index comparer
Search result
Searched index
Plug-ins
Index reader
Indexes (fingerprints) DB
Read
Plug-ins
21
Case of On-line Comparison
Query index
Monitoring result
Query index
Index comparer
Index
Content stream
Plug-ins
CONTENT FEATURES EXTRACTION ENGINE
Monitored content
Content stream reader
Read