Semantic Contentbased Access To Hypervideo Databases

1
Semantic Content-based Access To Hypervideo
Databases
Haitao Jiang Major Professor Ahmed K.
Elmagarmid Computer Science Department Purdue
University 1998
2
Organization Of The Talk

• Introduction And Review Of Related Work
• Logical Hypervideo Data Model (LHVDM)
• Semantic Content-based Video Queries
• A Web-based Logical Hypervideo Database (WLHVDB)
• Conclusion

3
Unique Characteristics Of Video Data

• Semantics rich and ambiguous
• Relationship ill-defined
• Structure unclear
• Dimension spatial and temporal
• Volume huge

4
Video Data Content

• Visual Content
• Audio Content
• Text Content
• Semantics Content

5
Research Problems

• Video Data Modeling
• Video Data Indexing
• Video Data Query
• Video Browsing

6
Video Data Model Requirements

• Content-based Data Access
• Video Data Abstraction
• Variable Data Access Granularity
• Dynamic And Incremental Video Annotation

7
Video Data Model Requirements (Con.)

• Video Data Independence
• Spatial And Temporal Characteristics
• Video And Meta-data Sharing And Reuse

8
Related Work

• Video Data Modeling, Indexing, And Querying
• Video Objects
• Video Browsing

9
Video Data Modeling, Indexing, and Querying

• Traditional Database Approach
• Visual Content Or Segmentation-based Approach
• Stratification Or Annotation Layering Approach

10
Traditional Database Approach

• Categorize And Predefine Video Data
  Attributes/Values
• Use Traditional Databases And SQL
• Inflexible And Limited
• Examples VISION, Video Database Browser

11
Segmentation-based Models

• Parse And Segment Video Streams
• Index On Visual Features Of RFrames
• Extract High Level Logical Structure And
  Semantics By Classifying Against Domain Models

12
Segmentation-based Models (con.)

• Can Be Fully Automated
• Lack Of Flexibility
• Limited Semantics
• Video Streams Need To Be Well-structured
• Examples JACOB, QBIC, Informedia

13
Stratification

• Segment Video Semantics
• Concept Of Logical Video Data
• Allows For Semantic Content-based Video Access
• Annotation Can Be Tedious And Biased
• Examples VideoStar, Algebraic Video

14
Stratification(con.)

• Existing Models
• Has Limited Temporal Queries
• Has Limited Video Browsing Mechanism
• Lack Multi-user Views And Data Sharing
• Lack Modeling Of Video Objects
• Lack Spatial And Spatial-Temporal Query
  Capabilities

15
Different Forms Of Video Annotation

• Multi-layer Icons - MediaStream
• Keywords
• Free Text Documents
• Other Types Of Annotation?

16
Sources Of Video Annotations

• Closed Caption
• Text In Video Frames highlight detection and OCR
• Voice Recognition
• Manual Annotation

17
Annotation Support In A Video Data Model

• Annotation of Arbitrary Sequence
• Incremental Creation, Deletion, And Modification
• Multi-user Annotation Sharing
• Arbitrary Overlap Of Annotations

18
Video Objects

• Index On Spatial And Temporal Information
• MBR as the Spatial Representation
• Narrow Focus And Lack Of Data Abstraction
• Limited Video Queries
• Example AVIS, CVOT

19
Video Browsing

• Visual Content-based Browsing
• Film Strips
• Salient Images
• Scene Clustering Graph
• Need Semantic Content-based Browsing
• Need Inter-Video Navigation

20
Research Motivations

• Visual Content-based Video Access IS Important
  BUT Lack Semantics
• Users Often Prefer Semantic Content-based Video
  Data Access
• Lots Applications Digital Video Library And
  Distance Learning etc.
• Web Is An Emerging Way Of Information Sharing

21
Research Goal

• Goal To Provide Effective And Flexible Semantic
  Content-based Video Data Access In A Distributed
  and Multi-user Sharing Environment
• Both Spatial And Temporal Video Queries
• Heterogeneous Applications And User Views
• Semantic Content-based Browsing

22
Logical Hypervideo Data Model (LHVDM)

• Definition
• Hierarchical Video Abstractions
• Hot Video Object Modeling
• Video Indexing
• Video Semantic Association And Hypervideo
• A Generic Video Database Architecture

23
Logical Hypervideo Data Model (con.)

• (PV, PVS, LV, LVS, HO, CD, LINKS, UV, MAP)
• PV Set Of Physical Video Streams
• PVS Set Of Physical Video Segments
• LV Set Of Logical Video Streams
• LVS Set Of Logical Video Segments
• HO Set Of Hot Objects
• CD Set Of Content Descriptions
• LINK Set Of Video Hyperlinks
• UV Set Of User Views
• MAP Set Of Mapping Relations

24
Logical Hypervideo Data Model (con.)

• MAP includes
• PV PVS Easy Data Manipulation
• PVS LV Data Independence And Data Reuse
• LV LVS Multi-user View
• LV,LVSHOEffective Query
• LV,LVS,HO,CDUV Multi-user View Sharing
• LV,LVS,HO,LINKSCD Semantic Content-based
  Access
• Video Hyperlinks Effective Video Browsing

25
Hierarchical Video Abstractions
User Views (UVs)
Logical Hypervideo Data Model (LHVDM)
Logical Video Segments (LVSs)
Hot Objects (HOs)
Video Hyperlinks
Logical Video Streams (LVs)
Physical Video Segments (PVSs)
Physical Video Streams (PVs)
26
Hot Video Objects

• What Is A Hot Video Object
• A Logical Video Abstraction
• A Sub-Frame Region That Is Hot In A Set Of
  Logical Frame Sequence
• Why Call Them Hot Object?
• Target Of Interest
• Hyperlink Property (Hot Video Spot)

27
Hot Video Objects (con.)

• Implicit Hot Video Object
• Why Hot Object Modeling Is Important?
• More Precise Video Annotation And Query
• Capture Spatial Characteristics of Video Data

28
Hot Video Object Model
29
Hot Object Tracking

• Template Matching
• Active Contour
• E ? Eint (1- ?) Eext
• Eint(Vi) ?Vi 2 ?Vi 2
• Eext ?c (-N(V (t))?I(V (t)))dt
• Eext (Vi) 1 - (D(Vi)/255 1) (-N(Vi)?I(Vi ))

30
Experiments With Hot Video Object Tracking
31
Video Indexing

• Main Indexes Semantic Content Descriptions
• Content Descriptions
• Hot Objects v.s. LVSs v.s. LVs v.s. LINKSs
• Systems v.s. Users
• Sharing Access Control
• Auxiliary Indexes Various Mappings

32
Video Semantic Association and Hypervideo

• Why Semantic Association Is Important?
• More Effective Video Data Access
• Video Hyperlinks Represent Semantic Associations
• Hypervideo And Hypervideo Databases
• Flexible And User Adaptive Hypervideo Databases

33
A Generic Video Database System Architecture
Users
Video Indexes
Knowledge Inference Engine
GUI

• Knowledge
• Base

Hot Objects
Spatial Information
Geometric Engine
Integrator
LVS
Information Retrieval Engine
Video Annotations
Video Hyperlinks
LV
Video Data
Video Database Engines
Video Database
34
Semantic Content-based Video Query Language

• (Expr, Granularity, Scope, Space)
• Expr video query expression
• Granularity
• Logical Video Streams (v)
• Logical Video Segments (s)
• Hot Objects (o)

35
Semantic Content-based Video Query Language
(con.)

• Scope
• System Annotations Only (s)
• Including User Annotations (u)
• Including Sharable Other Users Annotations (a)

36
Semantic Content-based Video Query Language
(con.)

• Space
• Subset Of VDB
• Can Be Result Of Another Query
• Allows Recursive Query Refinement
• Example Q2 (expr, o, u, Q1)

37
Boolean and IR Query Operators

• AND, OR, NOT
• NOT Is Not Safe
• ADJ (adjacent)
• Regular Expression And Approximate Matching

38
Temporal Query Operators

• Thirteen Interval Relations Allen83

Before
After
Ends
Starts
Equal
Meets
Overlaps
During
39
Temporal Query Operators (con.)

• Operators For LVS
• Interval Temporal Operators
• Operators For HO
• Instance (or Point) Temporal Operators more
  precise query specification
• Interval Temporal Operators

40
Spatial Query Operators

• Directional
• Topological
• Distance

North
West
East
South
41
Query Processing

• Recursive And Top-Down/Bottom-UP
• Support Distribute Evaluation
• Close World Assumption?
• Answer No (Raw Video Data) And Yes (Within
  Users View)
• Reason Video Data Is Semantically Rich

42
Query Search Space

• User Definable
• System Owned Subset of VDB Are Always Searched
• Users Queries Are Processed Within Ones View
• Determined By A Querys Granularity, Scope,
  Space, And User

43
Efficient Query Processing

• Query Augmentation or Pre-filtering
• Query Evaluation Order
• Query Caching And Knowledge Base

44
Query Examples

• Simple Queries
• Find video clips that has a red BMW Z3 in it
• Q1 ((red Ù BMW Z3), o, -, -)

45
Query Examples (con.)

• Temporal Query
• Find video clips in which a scene with a bird
  flying appears after the scene with a child
  eating ice cream
• Q3 ((bird Ù flying) Tafter (child Ù eating Ù
  ice cream), o, -,-)
• Q3 ((bird Ù flying) Iafter (child Ù eating Ù
  ice cream), s, -,-)

46
Query Examples (con.)

• Spatial Query
• Find video clips in which the Vice President Al
  Gore standing to the right of President Clinton
  who is giving his Union speech at Washington DC
• Q2a (Union speech Ù Washington Ù DC, s, -,-)
• Q2b (((Vice President ? Gore) Sright (President
  ? Clinton)), o, -, Q2a)

47
Query Examples (con.)

• Spatial Queries
• Find video clips in which a blue bird is flying
  over a kids head
• Q4 ((blue Ù bird Ù flying) Sabove ((child ? kid
  ? boy ? girl) Ù head), o, -,-)

48
Query Examples (con.)

• Spatio-Temporal Query
• Find video clips in which a police car with
  siren on is chasing a red Porsche and hit on it
• Qa ((police car siren), o, -,-)
• Qb (red Prosche), o, -,-)
• Q ((Qa Sapproach Qb) Ibefore (Qa Stouch Qb),
  o, -,-)

49
Web-based Logical Hypervideo Database System
(WLHVDB)

• System Architecture
• Video Wrapper And Lazy Delivery
• Populating The Video Database
• Distributed Query Processing
• Access Control And User Profiling

50
System Architecture
Physical Video Data
Video Parser
Logical Video Data
Account Manager
Video Annotations
Video Annotation Engine
Server
User Profile Manager
Video Hyperlinks
Query Processing Server
User Views
Access Control Manager
IR Engine
Video Indexes
User Profiles
Server Cache Manager
Various Tools and Scripts
Server Query Cache

• Internet

Query Input
Client Query Cache
Query Result Presentation
Media Player
Client Cache Manager
Client
Data Editor
Client
Client
51
Glimpse (GLobal IMPlicit SEarch)

• Small Index 2-4
• Full Text Boolean Queries
• Arbitrary Approximate And Regular Expression
  Matching
• Efficient (occurrences among 4500 files of total size of 69MB

52
Video Wrapper And Lazy Delivery

• Why Need Them?
• Huge Date Volume v.s. Limited Bandwidth
• Why Lazy Delivery?
• Avoid Sending Video Data Information
• What is a Video Wrapper?
• Multi-resolution Video Representation
• Adaptive Local Refinement Based On Interest

53
Current Video Wrapper Implementation
Video Poster
Clip Posters
RFrames
PVSs

54
Populating the VDB
Video Parsing and Segmentation
Video Representation Construction
Video Wrapper
Video Capture
LVSs and Annotations
Video Indexes
LVs
Closed Caption Capture
Hot Objects
Object Tracking
DBA
Users
Analog Video
55
Query Processing
Client
Server
Video Query
Query Parsing and Syntax Checking
Show Error Message
Processing IR Sub-queries
No Error
Search Users View
Sending IR Sub-queries
Processing Boolean and Spatio-temporal Operators
Sending Partial Results
Result
56
Server and Client-Side Query Caching
IR Sub-queries
Server
Client
Miss
Miss
Client-side Cache
Server-side Cache
Query IR Engine
Hit
Hit
Update Cache
Get Results
Get Result
Send Results
Update Cache
Results
57
Video Browsing

• Loop of Query-Browsing-Play
• Inter- And Intra-Video Browsing
• User Adaptive
• Video Wrapper Refinement Process

58
Video Browsing (con.)
Video Query
Video Posters
Video Board
RFrames
Annotations
Audio Stream
Video Hyperlinks
Video Stream
59
Access Control and User Profiling

• Different Categories Of Users And Groups
• Different Permissions On Video Data And Metadata
• Users Need To Authenticate Themselves
• User Activities And Local Environment Information
  Are Recorded

60
Summary of Major Contributions

• A Novel Video Data Model (LHVDM) That Supports
• Multi-level Video Abstraction
• Video Data Independence
• Multi-user Data Sharing
• Dynamic And Incremental View Update
• Variable Access Granularities

61
Summary of Major Contributions (con.)

• A Novel Video Data Model (LHVDM) That Represents
• Both Spatial And Temporal Video Characteristics
• Hot Video Objects
• Video Semantics And Semantic Associations

62
Summary of Major Contributions (con.)

• A Novel Video Data Model (LHVDM) That
• Supports User Adaptive Video Browsing
• Hyperlinks Video Entities For More Efficient
  Browsing
• Can Be Extended To Other Multimedia Data Such As
  Audio Data

63
Summary of Major Contributions (con.)

• A Video Query Language That Allows
• Easy Query Formulation
• Video Semantic Content-based Queries
• Both Spatial And Temporal Constraints
• Hot Object-based Video Queries
• User Selectable Granularity, Space, and Scope

64
Summary of Major Contributions (con.)

• A Generic Video Database System Architecture
  That Is
• Modular, Flexible, And Scalable
• Readily To Be Distributed
• Easy To Be Implemented

65
Summary of Major Contributions (con.)

• The Design And Implementation Of A Web-based
  Prototype That Uses
• A Novel Video Wrapper And Lazy Evaluation
  Approach
• Distributed Query Processing And Sub-query
  Caching Schema
• Multi-user Data Access Control And View Sharing
• User Profiling

66
Future Work

• Identify New Applications And Perform More
  Extensive Tests
• Explore And Integrate Other Forms Of Video
  Annotations Such As Visual Features
• Extended To Other Multimedia Data Such As Slides,
  Images, And Audio

67
Future Work(con.)

• Knowledge-based Video Access
• Automatic Generation of Video Wrappers
• Video Data Security and Access Control

68
Related Publications

• Survey and Books
• A. K. Elmagarmid and H. Jiang. Multimedia Video
  (chapter), Encyclopedia of Electrical and
  Electronics Engineering. John Wiley Sons. 1998,
  In press.
• A. K. Elmagarmid, H, Jiang and et al. Video
  Database Systems Issues, Products and
  Applications. Kluwer Academic Publishers, 1997.
• H. Jiang, A. Helal, A. K. Elmagarmid, and A.
  Joshi. Scene Change Detection Techniques for
  Video Database Systems. ACM Multimedia Sys.,
  6186-195, May 1998.
• H. Jiang and A. K. Elmagarmid. Video Databases
  State of the Art, State of the Market and State
  of Practice. Proc. 2nd Intl. Workshop on
  Multimedia Info. Sys., Page 87-91, West Point,
  New York, September 26-28, 1996.

69
Related Publications (con.)

• Video Analysis And Computer Vision
• H. Jiang and A. K. Elmagarmid. Extract Visual
  Content Representation in Video Databases, Proc.
  of Intel Conf. on Imaging Sci., Sys., and Tech.
  (CISST'97), Las Vegas, Nevada, June 30 - July 3,
  1997.
• H. Jiang and J. Dailey. A Video Database System
  for Studying Animal Behavior. Proc. SPIE
  Photonics East'96 - Multimedia Storage and
  Archiving Sys. Intl. Conf., Page 162-173, Volume
  SPIE-2916, Boston, MA, November 18-19, 1996.

70
Related Publications (con.)

• Video Data Model, Indexing, and Access
• H. Jiang, D. Montesi, and A. K. Elmagarmid.
  Integrate Video and Text for Content-based
  Accesses to Video Databases. J. of Multimedia
  Sys. and Tools. 1998, accepted.
• H. Jiang and A. K. Elmagarmid. WVTDB - A
  Web-based VideoText Database System. Special
  Issue on Data and Knowl. Management in Multimedia
  Sys., IEEE Trans. on Data and Knowl. Eng.. 1998,
  accepted.

71
Related Publications (con.)

• Video Data Model, Indexing, and Access
• H. Jiang and A. K. Elmagarmid. Spatial and
  Tempora Content-based Queries in Hypervideo
  Databases. Special Issue on Multimedia Data
  Management, The Very Large Database J.. 1998,
  submitted.
• F. Kokkoras, H. Jiang, I. Valhavas, A. K.
  Elmagarmid, and E. N. Houstis. Smart VideoText
  An Intelligent Video Database System, CSD-TR
  97-049, Department of Computer Sciences, Purdue
  University, West Lafayette, IN 47907, USA, 1997.

72
END OF THE PRESENTATION

• THANK YOU VERY MUCH