Outline

1
-- Outline --

• Objectives of the MPEG-7 standard
• A MPEG-7 application example
• Main elements of MPEG-7
• Conformance of new Descriptors /Description
  Schemes

2
Why do we need Mpeg-7?

• Allow accurate access to audio-visual content
• Achieve the maximum interoperability
• Content modalities, feature aspects
  applications
• Environments content creation, management,
  distribution consumption

3
What is Mpeg-7?

• Is NOT a standard for feature extraction /
  matching (search engines)
• Is NOT compression standard
• Is known as Multimedia Content Description
  Interface
• structural, detailed descriptions of AV content
  at different granularity in different
  application areas.

4
Mpeg-7 Application example

• Region color descriptor proposal (IBM)
• To query images based on the color of one or more
  of their regions (e.g. find me skin-colored
  regions ).
• Various colored surfaces can be modelled and
  clustered into surface classes.
• The region color descriptor of a region in an
  image can be represented by a surface class with
  a semantic class label a class identifier.

5
Creating region color descriptor
6
Querying region color descriptor
7
Main Elements of Mpeg-7 (1)

• Descriptors (Ds)
• syntax semantics of feature representation
• Description Schemes (DSs)
• semantics structure of the relationships
  between components (Ds DSs)
• Description Definition Language (DDL)
• creation of new Ds DSs
• modification / extension of existing Ds DSs
• Systems tools
• issues of synchronisation, transmission
  mechanisms...

8
Main elements of Mpeg-7 (2)
DDL
9
Mpeg-7 specifications

• Part 1 Systems (System tools)
• Part 2 Description Definition Language (DDL)
• Part 3 Visual (Ds)
• Part 4 Audio (Ds)
• Part 5 Multimedia Description Schemes (DSs
  Ds)
• Part 6 Reference Software
• Part 7 Conformance

10
Description Definition Language is ...

• XML Schema (W3C) Mpeg-7 extensions
• Reusable Mpeg-7 Schema
• Importing to import type declarations of Mpeg-7
  schemes
• Redefinition To modify existing Ds or DSs
• Restriction To restrict certain aspects of
  existing Ds or DSs
• Extension To extend existing Ds or DSs

11
Mpeg-7 Audio

• Low-level / generic tools
• Scale Tree (20 Ds) temporal envelope, spectral
  envelope, harmonicity
• Silence segment (1 D) levels of silence
• Application-specific tools
• Sound effects (5 Ds)
• Musical instrument timbre (3 Ds)
• Spoken content (12 Ds)
• Melody contour (4 Ds), Melody (5 Ds)

12

• SoundEffectCategoryRef a category label defined
  in Controlled Terms
• ProbabilityModel statistical model used for
  content classification
• AudioSpectrumBasis a projection matrix to reduce
  the dimentionality of a frequency spetra

13
Mpeg-7 Visual (1)

• Basic structures (5 Ds)
• Grid layout
• Time series
• Multiple view
• Spatial 2D co-ordinates
• temporal interpolation

• Color (7 Ds)
• Color Space Color Quantization
• Scalable Color HSV color space Haar
  transformation
• Dominant Color
• Color Layout, Color structure
• Group-of-Frames/Group-of-Pictures color

14
Mpeg-7 Visual (2)

• Texture (3 Ds)
• Homogenous directionality, coarseness and
  regularity of patterns
• Non-Homogenous (Edge Histogram)
• Shape (3 Ds)
• Contour-based Curvature Scale-Space (CCS)
• Region-based Angular Radial Transformation
• 3D

• Motion (4 Ds)
• Motion Activity intensity, direction, spatial
  distribution
• Camera Motion
• Motion Trajectory
• Localization (2 Ds)
• Region locator
• Spatial-temporal locator
• Face recognition (1 D)

15

• GlobalCurvatureVector global parameters of the
  contour, i.e.Eccentricity Circularity
• PrototypeCurvatureVector eccentricity
  circularity of the prototype contour
• HighestPeak the parameters of the filter
  corresponding to the highest peak
• Peak the parameters of the remaining prominent
  peaks

16
Overview of MDS
17
Overview of MDS
Title, creator, classification
Format,Coding, instance
Usage Rights, Usage Record
Event, Object, relation
Segment, segment relation graph
18
Overview of MDS
Filter, search, browse
Hierarchical / sequential summary
Collection of semantic concepts
Network condition, resolution
19
Structural aspects of a video clip
A video clip is broken down into segments
sub-segments. Each segment may be described by
a set of visual or audio Ds and DSs.
20
An example of textual description of structural
aspects in Mpeg-7
21
Hierarchical summary
22
Conformance of extended D/DS

• If a new DS defined by restriction(not by
  extension) of an existing one through DDL, it can
  be said to be compliant to Mpeg-7.
• If descriptions instantiated from a new DS
  (defined by extension redefinition), they can
  be said to be compliant to DDL, but not Mpeg-7.
• It is assumed that consuming terminals understand
  the semantic of the new DS in a non-normative way.

23
What can we do about Mpeg-7 based on our current
system?

• Sports hierarchical summary, maybe combined with
  users preference
• Simple structural video segments, maybe combined
  with text/semantic annotation
• Partial description generated from our specific
  visual/audio feature analysis tools for
  similarity retrieval

24
Summary of a given program based on users
preference
25
What to do if we cannot find the right DS/D for
our needs?

• Extension by using DDL
• Extension without using DDL
• through amendments (Mpeg-7 Version 2)
• through private tools
• through private tools defined within a consortium
  for a specific application domain

26
Conclusion

• The Mpeg-7 standards cover a wide range of
  generic application needs.
• The XML-based DDL is used to define various
  Mpeg-7 functional models. Also provided are to
  modify/create/extend Ds DSs.
• Mpeg-7 visual/audio parts allow similarity
  retrieval.
• Mpeg-7 MDS provides various metadata structures
  to describe AV content.