Multimedia Databases

1
Multimedia Databases
2
Questions for you

• Where do you think potential multimedia database
  applications exist?
• Think of local examples and national examples
• Recent examples in the news? see photocopy of
  article
• Multimedia database driven systems
• Multimedia intelligent querying database systems

3
Answers

• Local examples of potential examples
• Football clubs - SCFC/PVFC
• Theatres - New Vic / Regent
• University - SU
• Museums - City
• Newspaper Sentinel
• National examples include
• Airports
• Police databases

4
Introduction to multimedia databases

• What is an multimedia database?
• A multimedia database can also be known as a
  multimedia database management system (MMDBMS)
• A multimedia database/MMDBMS is a framework that
  manages different types of data potentially
  represented in wide diversity of formats on a
  wide array of media sources
• What are the features of this type of database?

5
Features of a MMDBMS

• Ability to uniformly query data (media data,
  textual data) represented in different formats
• Ability to simultaneously query different media
  sources and conduct classical database operations
  (create, read, update and delete etc) across them
• Ability to retrieve media objects from a local
  storage device in a continuous manner
• Ability to take the answer generated by a query
  and develop a presentation of that answer in
  terms of audio-visual media
• Ability to deliver this presentation in a way
  that satisfies various user requirements

6
Review of media types

• Text/Document
• Image
• Video
• Audio
• Classical Data (e.g. relations, flat files,
  object bases etc)

7
Review of media types

• Video and audio differ from the other media types
  listed above because of their temporal nature
• Ability to take the answer generated by a query
  and develop a presentation of that answer in
  terms of audio-visual media
• Ability to deliver this presentation in a way
  that satisfies various user requirements
• Video/audio retrievals must appear to be
  continuous, hiccup free presentations
• Video/audio support operations like fast-forward,
  rewind and pause, that were not supported by
  classical data types
• Let us briefly consider how this data could be
  used in a business multimedia scenario

8
Sample multimedia scenario

• Consider current terrorism investigation by the
  USA/UK security bodies
• Investigation may generate the following types of
  data sources
• Video data captured by surveillance cameras that
  record activities at various locations
• Audio data captured by legally authorized
  telephone wiretaps

9
Sample multimedia scenario

• Image data consisting of still photos taken by
  investigators
• Document data seized by the police during raids
  on one or more places
• Structured relational data containing background
  information, bank records etc of the suspects
• Geographical information systems data
• What could we do with this data?
• Answer - raise queries

10
Example image queries

• I have a photograph/still image e.g.
• I want to know the identity of the person in the
  picture
• The image has a name attribute attached to it
• Query 1 retrieve all images from the image
  library (database) in which the person appearing
  in the currently displayed photograph appears

11
Example image query

• I want to examine pictures of Chris Mayer
• Query 2 retrieve all images from the image
  library in which Chris Mayer appears
• This could be done by either some sort of key
  match or using an image match

12
Issues raised

• If follows that there are two basic kind of
  queries for images
• Image based queries
• Keyword based queries
• In the first query we gave an image as input
  (query image)
• We expect output as a ranked list of images that
  are similar to the query image
• What does similar mean? How confident can we be
  with the result? What action rests on the result?

13
Issues raised

• To support this we need to know what similarity
  means
• We need to know what ranking means
• A multimedia database driven system needs to be
  able to efficiently support these operations

14
Issues raised

• In the 2nd query we gave a keyword as input (name
  of suspect Chris Mayer)
• We want as output those photographs that are
  known to contain an image object whose name
  attribute is Chris Mayer
• To support this we need to know how to associate
  different attributes with images (or parts of
  images)
• We need to index and retrieve images based on
  such attributes

15
Example Audio (sound) query

• An investigation officer is listening to an audio
  surveillance tape
• The tape contains a conversation between
  individual A under surveillance and another
  individual B meeting A
• Query1 Find the identity of individual B given
  that individual A is Chris Mayer

16
Example Audio (sound) query

• Officer wants to review all audio logs that Chris
  Mayer participated in during some specified
  period of time
• Query2 Find all audio tapes in which Chris Mayer
  was a participant

17
Example Text query

• Investigating officer is browsing an archive of
  text documents - newspaper archives, police
  department files on old terrorist cases, witness
  statements etc
• Query Find all documents that deal with the
  Mayer Gangs financial transactions with
  Britannia Building Society

18
Example Video query

• Officer is examining a surveillance video of a
  particular person being assaulted by an hooligan.
  However, the hooligans face is obscured and
  image processing algorithms return very poor
  matches.
• The officer thinks the assault was by someone
  known to the victim
• Query Find all video segments in which the
  victim of the assault appears
• By examining the answer we hope to find other
  people who have previously interacted with the
  victim

19
Simple Textual example

• Query Find all individuals who have been
  convicted of terrorism in the UK and who have had
  electronic fund transfers made into their bank
  accounts from Britannia Building Society
• The answer is problematic
• Determining all people convicted of different
  crimes may require accessing a wide variety of
  databases belonging to different police
  jurisdictions etc
• Britannia may have accounts in hundreds of banks
  worldwide each of which uses different formats
  and different database systems

20
Heterogeneous query

• All queries discussed so far involve one media
  type i.e. image, audio, video or text
• Each query accesses only image or audio or video
  data but does not access a mix of these media
  types
• Complex queries will mix and match data from
  these different media sources
• Mix and match is difficult!

21
Heterogeneous multimedia query

• Query Find all individuals who have been
  photographed with Chris Mayer and who have been
  convicted of security offences in the UK and who
  have recently had electronic fund transfers made
  into their bank accounts from Britannia
• This query requires
• We find all people satisfying the conditions of
  the simple query before

22
Heterogeneous multimedia query

• We access a mug shot database containing names
  and pictures of various individuals
• We access surveillance photograph database of
  still images
• We access a surveillance video database to see if
  a meeting between the suspect and other people
  recorded on the video
• Access image processing algorithms to determine
  who occurs in which video/still

23
Requirements Issues - Queries

• We need a single language within which multimedia
  data of different types can be accessed
• Language must be able to specify combination
  operations across different media types/merge and
  manipulate
• Language must be able to access
• Meta data describing the content of media sources
• Raw data supported by the different media sources

24
Requirements Issues - Queries

• As well as the language we need techniques to
• Optimise queries by planning
• Develop servers that can optimize processing of a
  set of queries

25
Requirements Issues - Content

• What is content of media source? Under what
  conditions can content be described textually and
  under what conditions must it be described
  directly through the original media type?
• How should we extract the content of
• an image?
• an video clip?
• an audio clip?
• a free/structured text document?

26
Requirements Issues - Content

• How should we index the results of this extracted
  content?
• What is retrieval by similarity?
• What algorithms can be used to efficiently
  retrieve media data on the basis of similarity?

27
Requirements Issues - Storage

• How do these storage devices work?
• Disk systems
• CD-ROM and DVD
• Tape systems and tape libraries
• How is data laid out on such devices?
• How to design servers using the above devices
  when they use playbackrewindfast fwd and pause

28
Requirements Issues -Presentations and Delivery

• How do we specify the content of multimedia
  presentations?
• How do we specify the form (layout) of this
  content?
• How to deliver a presentation to users when there
  is the need to
• How to interact with remote (distributed) servers
  and convergence/compatibility issues
• What are the bandwidth issues

29
Directed Reading

• IEEE paper on MMDBMS
• Multimedia Databases Lynne Dunckley
• Chapter 1, 2, 5
• There is one short term loan and one 24 hour loan
  copy
• Computing and Computing Weekly in Thompson
  library
• See if there is any news in this area