Multimedia Semantic Web and MPEG-7

1
Multimedia Semantic Web and MPEG-7

• Ana B. Benitez
• ana _at_ ee.columbia.edu
• Image and Advanced Television Lab (ADVENT)
• Department of Electrical Engineering
• Columbia University

2
Semantic Web and Multimedia

• Motivation
• Rapid explosion of available multimedia on the
  Web
• Extract semantics from multimedia is even harder
  than from text!
• Multimedia Semantic Web
• Describe multimedia documents
• Enable interoperable, scalable and intelligent
  applications
• Deal with multimedia and interact with users at
  human levels
• MPEG-7 Multimedia Content Description
  Interface
• Describe different aspects of multimedia
  documents at different abstraction levels
• Enable multimedia applications in general

3
MPEG-7

• Suite of tools for describing
• Structure and semantics XML and RDF for generic
  data
• Audio and visual features color, texture,
  melody and timbre
• Content management information creator, media
  format and rights
• Free and structured annotations who, when and
  where
• Text and graph classification schemes thesauri
  and ontologies
• Summaries and variations key frame hierarchy
  and abstracts
• Multimedia collections and models statistical
  models and classifiers
• XML-Schema as description definition language
• ISO standard in September 2001!

4
MPEG-7 Description
5
Abstraction Levels
6
Structure and Semantic Tools

• Framework comparable to ER Modeling and Semantic
  Networks
• Entities, attributes and relationships
• Structure descriptions
• Segment entities video, audio, multimedia
  segments,
• Attributes AV descriptors, spatio-temporal
  localization,
• Relationships temporal, spatial,
• Semantic descriptions
• Semantic entities objects, events, concepts,
  states, places, times
• Attributes label, definition, abstraction level,
  media occurrences
• Relationships object-event, concept-semantic
  entity,
• Multiple abstraction levels media and formal
• Encoding of rules based on alphabet and rule
  graphs in the works

7
RDF/DAMLOIL vs MPEG-7

• Pros RDF/DAMLOIL
• Cardinality of properties different than zero or
  one
• Arbitrary union and intersection combinations of
  classes
• Pros MPEG-7
• Richer and more flexible graphical structure
• Total or partial graph morphisms to relate graphs
• Discover and reuse sub-descriptions but not as
  nodes, e.g., modern adaptation of Hamlet
  (poisson?heart medicine drown?step front of car)
• Closer to how humans construct descriptions
• More types of entities (segments, object, events,
  ), attributes (AV features, definition,
  abstraction, ), relations (composition,
  dependency, spatio-temporal), and abstraction
  levels (media, formal)
• Mechanism to encode rules to enable inference in
  progress
• Non-structure/semantic information (management,
  summaries, )

8
Some Questions

• What is the Semantic Web doing about multimedia?
• What is the position of the Semantic Web on
  MPEG-7?

9
Some Pointers

• To know more about MPEG-7
• MPEG http//www.cselt.it/mpeg
• MPEG-7 Industry Forum http//www.mpeg7.org
• To know more about our work at MPEG-7
• MPEG-7 Project at Columbia University
  http//www.ctr.columbia.edu/ana/MPEG7
• ADVENT http//www.ee.columbia.edu/advent,
  Research

10
The End

• Thanks for you attention!

11
Content Management Description

• ltCreationInformationgt
• ltCreationgt
• ltCreatorgt
• ltRolegtltNamegtPhotographerlt/Namegtlt/Rolegt
• ltPersongt ltNamegt ltGivenNamegtSeungyuplt/GivenNa
  megt lt/Namegt lt/Persongt
• lt/Creatorgt
• ltCreationCoordinatesgt
• ltCreationLocationgt ltName xmllang"en"gtColum
  bia Universitylt/Namegt lt/CreationLocationgt
• ltCreationDategt ltTimePointgt1998-09-19lt/TimePo
  intgt lt/CreationDategt
• lt/CreationCoordinatesgt
• lt/Creationgt
• lt/CreationInformationgt
• ltMediaInformationgt
• ltMediaProfile master"true"gt
• ltMediaFormatgt ltContentgtimagelt/Contentgt
• ltVisualCodinggt
• ltFormat colorDomain"color
  href"urnmpegVisualCodingFormatCS1"gtJPGlt/Format
  gt
• ltFrame height"480" width"704"/gt
• lt/VisualCodinggt

12
Content Structure Description

• ltStillRegion id"SR1"gt
• ltTextAnnotationgt
• ltFreeTextAnnotationgt
• Alex shakes hands with Ana
• lt/FreeTextAnnotationgt
• lt/TextAnnotationgt
• ltSpatialDecomposition overlap"false"
  gap"true"gt
• ltStillRegion id"SR2"gt
• ltTextAnnotationgt ltFreeTextAnnotationgt Alex
  lt/FreeTextAnnotationgt lt/TextAnnotationgt
• ltVisualDescriptor xsitype"ColorStructureTy
  pe"gt ... lt/VisualDescriptorgt
• lt/StillRegiongt
• ltStillRegion id"SR3"gt
• ltTextAnnotationgt ltFreeTextAnnotationgt Ana
  lt/FreeTextAnnotationgt lt/TextAnnotationgt
• ltMatchingHintgt
• ltHint value"0.455" xpath../../VisualDes
  criptor"/gt
• lt/MatchingHintgt
• ltRelation xsitype"DirectionalSpatial
  SegmentRelationType name"left

13
Locator Description

• ltSpatioTemporalLocatorgt
• lt!-- Spatio-temporal component 1 --gt
• ltFigureTrajectory type"1"gt
• ltMediaTimegt
• ltMediaTimePointgtT000015lt/MediaTimePointgt
• ltMediaDurationgtPT1M15Slt/MediaDurationgt
• lt/MediaTimegt
• lt/FigureTrajectorygt
• ltVerticesgt
• ltCoordinatesgt 4.34 1.43 4.33 lt/Coordinatesgt
• ltCoordinatesgt 10.3 5.03 .33 lt/Coordinatesgt
• ltCoordinatesgt 5.34 .43 2.37 lt/Coordinatesgt
• lt/Verticesgt
• ltVerticesgt
• ltCoordinatesgt 4.34 1.43 4.33 lt/Coordinatesgt
• ltCoordinatesgt 10.3 5.03 .33 lt/Coordinatesgt
• ltCoordinatesgt 5.34 .43 2.37 lt/Coordinatesgt
• lt/Verticesgt
• ltVerticesgt

14
Content Semantics Description

• ltSemanticgt
• ltLabelgtltNamegtAlex shakes hands with Ana
  lt/Namegtlt/Labelgt
• ltSemanticBase xsitype"EventType" id"EV1"gt
• ltLabelgtltNamegtShake handslt/Namegtlt/Labelgt
• ltRelation xsitype"ObjectEventRelationType
  name"hasAgentOf
• target"AO1"/gt
• ltRelation xsitype"ObjectEventRelationType
  name"hasAccompanierOf
• target"AO2"/gt
• ltRelation xsitype"ConceptSemanticBaseRelatio
  nType
• name"hasPropertyOf"
  target"C1"/gt
• ltSemanticPlacegt ltLabelgtltNamegtColumbia
  Universitylt/Namegtlt/Labelgt
• lt/SemanticPlacegt
• ltSemanticTimegt ltLabelgtltNamegtSeptember 9,
  1998lt/Namegtlt/Labelgt
• lt/SemanticTimegt
• lt/SemanticBasegt
• ltSemanticBase xsitype"AgentObjectType"
  id"AO1"gt
• ltLabelgtltNamegtAlexlt/Namegtlt/Labelgt
• ltAgent xsitype"PersonType"gt
  ltNamegtltGivenNamegtAlexlt/GivenNamegtlt/Namegt lt/Agentgt
• lt/SemanticBasegt