Semantic Media Retrieval UC From the Analysis perspective

1
Semantic Media Retrieval UC(From the Analysis
perspective)
Institute of Informatics Telecommunications NCSR
Demokritos, Athens, Greece
Ioannis Pratikakis
MMSEM F2F meeting Athens, 08 December 2006
2
Contents

• UC aim
• Motivation wrt interoperability
• Example 1 solution sketching
• Example 2 solution sketching
• Concluding remarks

3

• UC Goal
• Deal with semantic interoperability for
• Automatic Media interpretation / understanding

Linking of low-level features to high-level
semantics
4
Linking LLF -gt HLS

• Two main trends
• (i) using machine learning techniques to infer
  the required mapping
• (ii) using ontology-driven approaches to both
  guide the semantic analysis and infer high-level
  concepts using reasoning

5
An efficient multimedia retrieval system MUST
(i) be able to handle the semantics of the
query (ii) unify multiple modalities in a
homogeneous framework (iii) abstract the
relationship between low level media features and
high level semantic concepts to allow the user to
query in terms of these concepts rather than in
terms of examples.
6
Examples

• Example 1
• Lack of semantics in low-level descriptors
• (single media)
• Example 2
• Interchange of semantics between media
• (multiple media)

7
Example 1
It is NOT related to (i) Different ways of
describing the same semantics See (MPEG-7
metadata interoperability UC) (ii) Lack of
formal grounding due to XML schema It is related
to Interchange of low-level descriptors for
semantic interpretation
8
Lack of semantic description Any Low-level
descriptor (MPEG-7 or proprietary) is
represented as a vector of numerical values
lt?xml version'1.0' encoding'ISO-8859-1'
?gt ltMpeg7 xmlns "http//www.mpeg7.org/2001/MPEG-
7_Schema" xmlnsxsi "http//www.w3.org/2000/10/X
MLSchema-instance"gtltDescriptionUnit xsitype
"DescriptorCollectionType"gtltDescriptor xsitype
"ColorLayoutType"gtltYDCCoeffgt43lt/YDCCoeffgt ltCbDCCoe
ffgt21lt/CbDCCoeffgt ltCrDCCoeffgt43lt/CrDCCoeffgt ltYACCo
eff5gt16 16 16 16 16 lt/YACCoeff5gt ltCbACCoeff2gt1
6 16 lt/CbACCoeff2gt ltCrACCoeff2gt16 16
lt/CrACCoeff2gt lt/Descriptorgt lt/DescriptionUnitgt lt/M
peg7gt
A non-expert cannot attribute to an object
with particular low-level descriptors even
semantics that concern Color, shape, texture, etc.
The ColorLayout Color descriptor
Useful for content-based retrieval
9
The classification process
Extract features
Transform to a feature vector
Compare with a model
Exchanging classifiers would be more interesting
to users than exchanging media descriptions
M. Asbach and J-R Ohm, Object detection and
classification based on MPEG-7 descriptions
Technical study, use cases and business models,
ISO/IEC JTC1/SC29/WG11/MPEG2006/M13207, April
2006, Montreaux, CH.
10
Interoperability problem

• A user might want to exchange class descriptions
• between an imaging device, a set-top box
• and a personal computer.
• Also, he might want to buy or rent classifiers
• over the Internet or stored on a disc

The classification process itself has to be
described in a standardized form
11
Solution (Example 1)

• In the classification process an exchange
  concerns
• classifier parameters
• class descriptions

• A classifier description MUST
• Specify on what kind of data (modality) it
  operates
• Specify whether concerns the complete media or a
  segment
• Contain a description of the feature extraction
  process
• ( Custom descriptors are needed to improve
  classification
• results achieved with MPEG-7 descriptors )
• Contain the transformation to generate feature
  vectors
• Contain a model that associates a feature vector
  to a class

Absence of media descriptor ontology
12
Solution Example (1)
Content layer
13
Scenario - Example (1)
Maria is an architect who wishes to retrieve
available multimedia material of a particular
architecture style like Art Nouveau, Art
Deco, Modern among the bulk of data that she
has already stored using her multimedia
management software. Due to her particular
interest, she plugs in the Architecture style
classifier kit that enables the retrieval of
all images or videos that correspond to this
particular style she queries for (eg. retrieve
all shots about the House of V. Horta, a major
representative of Art Nouveau style in Brussels,
which includes visual instances of the style).
14

• Example 2
• Interchange of semantics between media
• (multiple media)

15
The subject of this page is the victory of the
athlete Reiko Sosa at the Tokyos marathon.
What analysis steps are required if we would
like to enable a semantic retrieval in the query
show me images with the athlete Reiko Sosa ?
16
Semantics extraction from textual content
Person Names Naoko Takahashi, Reiko
Sosa Places Tokyo Athletics type Womens
Marathon and activity runs away from
17
Semantics extraction from visual content
Atheletes face detection ltSpatial relationsgt
One leads against the other
18
Extracted Semantics from textual visual content
may give a very good support to reasoning
mechanisms to reach a conclusion that we have
images with the athlete Reiko Sosa
19
To fulfil such a scenario we should solve the
problem how to interchange semantics from
different modalities
20
Solution Example (2)
This scenario manifestates the basic requirement
that is to extract rules of prioritising
particular modalities for the concept extraction
Cross-modality layer
Knowledge framework
21
Concluding remarks

• Example 1
• Lack of semantics in low-level descriptors
• (single media)
• Example 2
• Interchange of semantics between media
• (multiple media)

Solutions 1, 2 New ontological descriptions or
layers in existing ontologies that permit to
extract the appropriate rules