CS 586 Distributed Multimedia Information Management

1
CS 586 Distributed Multimedia Information
Management

• Prof. Dennis McLeod

2
About the paper . . .

• Towards Ontology-Driven Discourse From Semantic
  Graphs to Multimedia Presentations
• In Proceedings of the 2nd International Semantic
  Web Conference, 2003
• By Joost Geurts, Stefano Bocconi, Jacco van
  Ossenbruggen, and Lynda Hardman
• Presented by Gabriela Gutierrez, February 11,
  2004

3
Overview

• Introduction
• Example Scenario Rembrandt
• Process 1 From Semantic Graph to Structured
  Progression
• Process 2 From Structured Progression to
  Multimedia Presentation
• Conclusion

4
Introduction

• Traditionally
• Applying Semantic Web technology to multimedia
  information systems focuses on using annotations
  and ontologies to improve retrieval process
• Presentation of data is detail best left to CSS
  or XSLT style sheets
• In this paper
• Claim that importance and complexity of effective
  presentation design is grossly underestimated
• Concentration on improving the presentation of
  the retrieval results

5
Introduction

• Human professional designers must understand
• Underlying semantics of the clients information
• Most effective order, grouping and priorities for
  structuring this information
• Most effective means of using the chosen medium
  to convey the information

6
Introduction

• Information presentation design is a
  knowledge-driven process. It requires
• Sufficient knowledge about domain
• Knowledge on ordering, grouping and prioritizing
  information
• Knowledge about media design
• Selection of most appropriate medium
• Understanding of medium characteristics in order
  to choose an effective means to achieve the
  communication goal

7
Introduction

• Problem
• Professional designers can only design
  data-driven web sites if the underlying data, its
  semantics and target audience are relatively
  homogeneous.
• Variety of data sources, semantic relations,
  output devices, and user profiles forces content
  providers to adopt one-size-fits-all approach.
• Automation is needed in order to make the
  presentation of information knowledge-driven.

8
Introduction

• Assumptions
• Multimedia items are properly annotated
• Annotations represent domain relations in a
  semantic graph (e.g. RDF)
• Graph has associated Domain ontology
• There is a Discourse ontology containing
  information about different document genres and
  building blocks for creating documents for each
  genre
• There is a Design ontology containing media
  design knowledge

9
Introduction
10
Example Scenario Rembrandt

• Web query life and work of Rembrandt
• User-selected type of structured progression
  discBiography
• User-selected output medium non-interactive
  multimedia presentation
• Semantic graph retrieval components results
  domain ontology semantics relations
• Structured progression typical facts (name,
  DOB,) career facts personal life info

11
(No Transcript)
12
Process 1 From Semantic Graph to Structured
Progression

• CSS and XSLT operate purely on the XML level of
  RDFs serialization syntax w/o any understanding
  or support for semantics of RDF data model
• Transformation process needs access to knowledge
  on RDF Schema level
• For querying underlying domain ontology
• For access to its own operating knowledge

13
Process 1 From Semantic Graph to Structured
Progression

• Several transformations prototyped in Java and
  Prolog environments
• Direct access to a Sesame RDF Schema-based
  repository
• Can use any query language supported by Sesame
  (RQL, RDQL, SeRQL) to gain direct access on the
  RDF instance level and the RDF Schema Level
• Transformation process uses (declarative) domain
  and discourse-specific knowledge, while
  (procedural) transformation code remains generic

14
Process 1 From Semantic Graph to Structured
Progression

• Transformation code uses RQL query to retrieve
  classes that Rembrandt instance belongs to . . .
  domArtist
• Discourse ontology defines instance of
  discArtistBiography that has discSubject
  property with value domArtist
• Structured progressions have a discnarrativeUnits
  property that specifies the discNarrativeUnits
  that can be used to construct it (e.g.
  discPersonalData, discPrivateLife and
  discCareer)

15
Process 1 From Semantic Graph to Structured
Progression

• Narrative Units have associated rules used to
  select matching content
• Example discPrivateLife
• Rules to select information about family
  relations from semantic graph
• Graph includes relation domisMarried between
  Rembrandt and Saskia_Uylenburgh
• Rule 3 in following table can use domain
  relation to select Saskia in the discRole of
  discSpouse
• Rules can be applied recursively
• Rule 3 specifies that PrivateLife is the
  narrative unit that can be used for a subsequent
  nested story line
• Process continues until no more rules can be
  applied or a rule specifies that no further
  expansion should happen

16
(No Transcript)
17
Process 1 From Semantic Graph to Structured
Progression

• After all rules have been applied
• Biography w/ 3 narrative units
• discPersonalData (Rembrandt in role of
  discMainCharacter)
• discCareer (Chiaroscuro in role of
  discTechnique)
• DiscPrivateLife (Saskia_Uylenburgh in role of
  discSpouse)

18
Process 2 From Structured Progression to
Multimedia Presentation

• Two-step process
• Structured progression transformed into Document
  Structure
• Decisions on output medium (e.g. text,
  interactive hypermedia, passive multimedia)
• Document Structure transformed into a tree of
  formatting objects
• Detailed layout and formatting decisions (e.g.
  timing of presentation, transition effects)

19
Process 2 From Structured Progression to
Multimedia Presentation

• Advantage
• Mapping discourse-specific narrative units to
  more general document elements allows for more
  commonly applicable formatting rules (e.g.
  discPrivateLife can be mapped to document
  section element, relying on common formatting
  rules for section-level elements)
• Disadvantage
• There is always a level that can no longer be
  specified in terms of document structure (e.g. a
  figure w/ too much detail)
• Solution detailed structures copied directly
  into document structure in step 1 in order to
  define specific rules in step 2 to deal w/
  formatting directly

20
From Structured Progression to Document Structure
21
Conveying Document Structure

• Transforming a document structure into
  presentation constructs uses Cuypers library
• Uses constraint solving techniques to verify that
  a presentation construct conforms to
  delivery-context constraints (e.g. screen size)
• Allows alternative formatting specification if
  constraints are violated
• A rule that transforms a document structure into
  presentation construct has 2 discourse
  parameters
• discNarrativeType
• discRole
• Parameters allow system to adapt formatting of
  presentation to convey message more effectively

22
Conveying Discourse Semantics Directly

• Depending on their function, we need to define
  formatting for different media types
• Rembrandt self-portrait (discPortrait in
  discPersonalData vs. discPainting illustrating
  Chiarocuro)
• Awareness of impact of different media modalities
• Fall-back rules
• Image not identified as either discPortrait or
  discPainting should be applied generic
  formatting for images since mmPainting and
  mmPortrait are subclasses of mmImage

23
Conclusion

• Only short presentations have been generated to
  date, based on restricted domain ontology
• Focus has been on single discourse structure
  (biography) and single document structure
  (multimedia presentation)
• Additional research required to scale the system
  to more realistic scenarios
• Under investigation how knowledge about the user
  interacts w/ discourse and design knowledge used
  in current prototype