Media Retrieval

1
Media Retrieval
2
Outlines (revisited)

• Introduction
• Intellectual Foundation of Multimedia Information
  Retrieval
• Retrieval Models
• Text Retrieval
• Database, Bibliographic, and Keyword Searches
• Content-based Retrieval
• Object-matching and beyond
• Indexing
• Inverted file and MPEG-7
• Searching
• Multimodality and Query adaptation
• Introduction to Concept-based Retrieval
• SoloArt

3
Content-based Retrieval(the conventional
approaches)

• Images, Audio, and Video
• Caption Text
• Multimodalities
• Search methods and search engines
• Let us get started with a tour!

4
Touring the MMR systems

• Text-based Querying
• Images
• Googles Image Search
• Altavista Image Search
• Audio
• Compaqs SpeechBot
• Video and Multiple modalities
• Lycoss Multimedia Search
• FAST Multimedia Search
• StreamSearch.com
• MediaSearch.com

5
Googles Image Search
6
Googles Image Search -2
7
Alta Vistas Image Search
8
Alta Vistas Image Search -2
9
Compaqs SpeechBot
10
FASTs AllTheWeb Search
11
The Extended Text-based Systems

• Processing textual information to infer on
  audio-visual content.
• Often rely on the filename.
• koala.jpg
• Using meta information embedded in the web page.
• HEIGHT"60" ALTkoala watching TV"
• Further processing of the container page.
• The webpage is called Trailer for Starwars
  episode I Phantom the Menace, and in there you
  find a link to the phantom.mpg file.

12
Speech Pre-processing

• In SpeechBot, a speech processing mechanism is
  applied to convert audio data into textual data,
  then conventional text-based indexing and
  searching could be applied.

13
Touring the CBR systems

• Visual-based Querying
• Columbia Univ.s VisualSeek
• University of California at Irvines MARS
  (associated with Univ. of Illinois)
• IBMs QBIC

14
VisualSEEK
15
MARS
16
QBIC at Hermitage Museum
17
QBIC at Hermitage Museum -2
18
QBIC at Hermitage Museum -3
19
What did you observe?

• What has been extended?
• Which applications may be supported?
• What is the constraint?

20
CBR a brief history

• In 1970s, image data are commonly archived
  independently and indexed using text-based
  databases.
• Indexing carries subjectivity of annotation.
• Manual and expensive.
• Into 1990s, efforts are taken to remove person
  indexing and to automate the mechanism.
• Image data themselves are used as indexes.
• Full automation is idealized.
• Content-Based Retrieval was born.

21
CBR a brief history -2

• The CBR was founded on the computer vision and
  pattern recognition idealism and their
  techniques.
• CBR employs object-matching mechanisms.
• Earlier systems are characterized by Query by
  Example approach.
• Features commonly used are Color, Texture, and
  Shape.
• Motion features in case of videos.

22
CBR a brief history -3

• As Computer vision ideal is yet to achieve, CBR
  found itself to be constrained in a green-house.
• Many set-off for sophisticated feature
  exploration and computational reasoning
  adventures.
• A strong call for abandoning the full-automation
  idealism for returning to human interaction.
• HCI techniques such as relevant feedbacks started
  to roll-out.

23
Content-based Retrieval?
24
Between Yesterday Tomorrow

• An interesting question to ask today may well be
  is the object-matching idea all we want?
• Typical current CBR query would mostly read
  retrieve all documents that presents a blonde
  wearing blue dress driving a red porsche.
• Now let us suggest some offsets How about
  retrieve all paintings created using chiaroscuro
  technique? What differences are there between
  Christmas Cards and Birthday Cards?
• Where do you want to go today?

25
The Passage

• Go Deeper. Enhancing what we have got.
• Human-centered computing.
• Distributed Computing.
• Multi-modalities querying and processing.
• Go Sidewise. Call for better multimedia
  understanding.
• Efforts to find out which information and
  semantics are useful and how they could be
  derived and managed Concept-based (or semantic)
  retrieval.
• Transcending the Object-Matching Idealism.