Event-Based Fusion of Distributed Multimedia Data Sources

1
Event-Based Fusion of Distributed Multimedia Data
Sources

• Vincent Oria
• Department of Computer Science
• New Jersey Institute of Technology
• Newark, NJ 07102

2
Outline

• Classical Data Integration Problem
• Multimedia Data
• An Architectural approach to Multimedia Data
  Integration
• Event-Based Integration of Data Sources
• Conclusion

3
Classical Data Integration
Borrowed from M. Lenzerini
4
Classical Data Integration Issues

• How to construct the global schema?
• (Automatic) source wrapping
• How to discover mappings between the sources and
  the global schema?
• Limitations in the mechanisms for accessing the
  sources
• Data extraction, cleaning and reconciliation
• How to process updates expressed on the global
  schema, and updates expressed on the sources?
• The modeling problem How to model the mappings
  between the sources and the global schema?
• The querying problem How to answer queries
  expressed on the global schema?
• Query optimization

5
Multimedia Data

• Multimedia data management is more than physical
  server design
• Logical data modeling is important
• Multimedia data management is more than
  similarity search
• Show me all the images that are similar to this
  one in terms of color, texture, shape.
• Querying is much more complicated
• Give me all the news items on Baghdad over the
  last 2 weeks

6
Multimedia Data

• Multimedia data is heterogeneous in both format
  and in access primitives and this has to be
  accommodated
• You cannot store all the data in a single DBMS
  the system has to be open
• Query-based access to multimedia data is
  important as well as browsing and some
  transactional access
• Some DBMS-like interface and control over
  multimedia data should be provided

7
Multimedia Data

• Multimedia data management is not data model
  independent
• The complexity of the primitive data types and
  the required extensibility necessitate certain
  functionality
• It does not make sense to completely ignore
  standardization or to be slave to them
• Follow, and perhaps extend, standards (e.g.,
  XML, MPEG, )

8
Multimedia Database Processing
MM Data Pre- processor
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
Meta-Data
Recognized components
Additional Information
Query Interface
MM Data
MM Data Instance
MM Data Instance
Users
Multimedia DBMS
Multimedia Data Preprocessing System
Database Processing
9
Document Database Architecture
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
DTD/ XML Manager
Schema Parser
DTD or XML Schema files
DTD/ XML Schema
Type Generator
Query Interface
lt!ELEMENT ..gt ..... lt!ATTLIST...gt
Document content
Document Parser
DTD/ XML Schema
XML or SGML Document Instance
Documents
Parse Tree
Types
Users
Document DBMS
Instance Generator
Objects
Document Processing System
Database Processing
10
Image Database Architecture
Semantic Objects
Syntactic Objects
Image Content Description
Meta-Data
Query Interface
Image Annotation
Image
Users
Image
Image DBMS
Image Processing System
Database Processing
11
Video Database Architecture
Key Frames
Video Content Description
Meta-Data
Query Interface
Video
Video Annotation
Video
Users
Video DBMS
Video Processing System
Database Processing
12
Multimedia Data Integration An Architectural
Perspective

• Simple Client-Server
• Integrated Server
• Database Server
• Middleware and Mediation

13
Simple Client-Server
Client
Image Server
Database Server
Text Server
CM Server

• Heavy-duty client
• Synchronization, user interface, QoS,
• Client has to access each server
• Scalability problems
• client code has to be updated when new servers
  come on-line

14
Integrated Server
Client
DBMS Functions
Image Server
CM Server
Object Storage Server

• Heavy-duty server
• DBMS should be able to handle multiple storage
  systems
• Real-time constraints on CM

15
Database Server
Client
Database Server
Image Server
CM Server
Text Server

• Lighter client
• Client has to access only one server
• Scalability problems
• server may become a bottleneck - distribute and
  interoperate

16
Document Server
Structured Document DBMS
Image DBMS
CM DBMS

• Document-centric view
• Multimedia objects are parts of documents
• Might be suitable for, e.g., e-commerce catalogs

17
Interoperable System
Client
Client
Wrapper
Wrapper
Wrapper
Wrapper
Wrapper
18
Event-Based Multimedia Data Integration

• An event aims at modeling any happening
• Facts, context
• An event has 3 components
• Time
• Space (location)
• Objects

19
Events Temporal Dimension

• Time Line and Temporal relationships

Event2
Time Line
Image
Video
20
Events Spatial Dimension

• GIS (Location and Spatial Relationships)

Event2
Event1
Event3
Directional and Topological relationships
21
Events Object Dimension

• Which real world objects are involved in the
  event?
• Object Recognition
• Classical Data Integration

22
Event Spatio-Temporal Dimension

• Moving Objects and their Trajectories
• Raw representation
• The trajectory T of a moving object is defined as
  a sequence of vectors
• Tt1, , tn
• Each ri show the successive positions of the
  moving object over a period of time.
• Movement sequence
• The trajectory of a moving object is represented
  by a sequence of (movement direction, distance
  ratio) pairs. This representation is not affected
  by rotation, shifting or scaling.
• Mm1, , mn-1
• Each mi is a pair of (movement direction,
  distance ratio).

23
Event Model

• Events model interpretation context
• Example KIMCOE 2006 is an event
• Participants are objects
• Location Hilton Garden Inn, Suffolk, Virginia
• Date/Time October 24 - 27, 2006
• Has sub-events like sessions or visit of Lockheed
  Martin's Center For Innovation
• Event Properties
• Discrete or continuous
• Local or distributed
• Simple or composite
• Descriptors Data (classical and multimedia)

24
Event Querying
Time
Objects RDBM, XML
Space GIS
25
Event Querying
Time
Objects RDBM, XML
Space GIS
26
Event Querying
Time
Objects RDBM, XML
Space GIS
27
Event Operators

• Temporal Operator
• Spatial Operators
• Spatio-Temporal Operator
• Aggregation

28
Aggregation and Concept Hierarchy

• Dimensions are hierarchical by nature total
  orders or partial orders
• Example Location(continent ? country ?
  province ? city)
• Time(year?quarter?(month,week)?day)

Industry Country Year Category Region
Quarter Product City Month Week
Office Day
29
Aggregation and Concept Hierarchy Operators

• roll-up (increase the level of abstraction)
• drill-down (decrease the level of abstraction)
• slice and dice (selection and projection)
• pivot (re-orient the multi-dimensional view)
• drill-through (links to the raw data)

30
Aggregation and Concept Hierarchy Roll-up

• Use of aggregation to summarize at different
  levels of a dimension hierarchy
• Ex if we are given total sales per city we can
  aggregate on the market to obtain sales per state

Time (Quarters)
Q1
Time (Quarters)
Q2
Q4
Q3
Market (city, state)
Newark
Drama
S. Orange
Q1
Q2
Q4
Q3
N. York
Market (States,, USA)
New Jersey
Comedy
Category
Drama
New York
Dayton
Horror
Arizona
Comedy
Ohio
Category
Sci. Fi..
Horror
Sci. Fi..
Roll-up on Market
31
Aggregation and Concept Hierarchy Drill-down

• Inverse of roll-up
• Given a total sales by state, we can ask for more
  detailed presentation by drilling down on market

Q1
Time (Quarters)
Q2
Q4
Q3
Market (city, state)
Newark
Drama
S. Orange
Q1
Q2
Q4
Q3
N. York
Market (States,, USA)
New Jersey
Comedy
Category
Drama
New York
Dayton
Horror
Arizona
Comedy
Ohio
Category
Sci. Fi..
Horror
Sci. Fi..
Drill-down on Market
32
Aggregation and Concept Hierarchy Dice and
Slice
33
Conclusion

• Event model A data Integration model
• This is a work in progress We need to fully
  define the event model
• We want to build on existing Technology (RDBMS,
  XML, GIS,..)