Content Based Image Retrieval

1
Content Based Image Retrieval

• Images generated by
• satellites, military reconnaissance/surveillance
  flights, fingerprinting, mug shots,
• scientific experiments, biomedical imaging
• Applications are
• Art Galleries and Museum Management
• Interior Design, Remote Sensing, Weather
  forecasting, Picture Archiving, Fabric and
  Fashion Design, Law Enforcement and Criminal
  Investigation, Scientific/Medical Database
  Management

2
Previous Approaches

• Set of attributes extracted manually and managed
  by standard DBMS
• Feature extraction / object recognition system
• Automated approaches to object recognition are
  computationally expensive, difficult and tend to
  be domain specific

3

• 2 categories of features
• Primitive/Low level
• Logical
• Images can be retrieved by
• colour texture
• sketch shape
• motion text
• objective attributes subjective attributes
• browsing

4
CBIR

• CBIR must also address broader info retrieval
  context
• medical images / diagnosis / treatment
• Query specification
• needs natural language
• show me images of snow covered mountains
• sketch shape required
• similarity based

5
Query by Image Content(QBIC)

• QBIC allows queries on large image and video
  databases based on
• example images
• user sketches and drawings
• colour and texture patterns
• demo at http//wwwqbic.almaden.ibm.com/

6
QBIC

• Database population
• images and videos are processed to extract
  features - colours, textures, shapes, camera and
  object motion
• Database query
• User comprises graphical query
• Features generated from query and input to
  matching engine that finds image or video with
  similar features

7

• Illustration of a histogram for a grey scale
  image
• This is what information the computer sees
• You can tell from the histogram that the image is
  dark but nothing about the content.

8
Histogram matching

• Store histogram for each image
• Compare images using some statistical method
• Return sorted matches best fit first
• Problem - These would have the same histogram

9
Red Histogram
10
Green Histogram
11
Blue Histogram
12
Colour match

• Average colour
• adds up red, green, blue components of each pixel
• Colour position
• 6x8 or 9x12 grid overlays picture. For each block
  average Munsell colour and 5 most frequently
  occurring colours

QBIC interprets the virtual canvas as a grid of
coloured areas, then matches this grid to other
images stored in the database.
13
Shape matching
Aggregate shape measurement
Easier to automatically compute shape if
background is plain This was produced by
PaintShopPro
14
Textures

• Textures can be rough or smooth, vertical or
  horizontal etc

15
QBIC

• Queries can be
• Find images with approximately 30 red, 15 blue
• Find images with red round object

Drawn query
Best match
Matches in decreasing order of correlation
16
QBIC

• Queries can be
• Find images similar to this one
• Find sunsets with 30 red, 30 yellow
• Find images with 30 red and blue textured object
• Find images like this sketch

User sketch
Best match
17
Problems with shapes

• What would you input to retrieve both of these?
• Other problems
• Draw a horse returns dogs, cows etc
• Draw side view of cars doesnt retrieve front
  views
• Objects may be partially obscured

18
Colour similarity
All histograms should be similar to picture 1
19
Colour similarity and keyword
All histograms should be similar to picture
1 but keyword man was included. The picture of
the lion has the keyword MANE! The
following picture has keyword MANUFACTURING!
20
Colour Layout
Colour position should be similar to picture 1
21
Colour percentage
34 yellow 30 red 36 dont care
22
Colour percentage and keyword
34 yellow 30 red 36 dont care keyword SUNSET
23
Keyword only
keyword SUNSET
24
Custom Paint Query
Top two thirds of picture blue Lower
third yellow
25
Texture search
Texture search like first picture
26
Texture search
Texture search like first picture
27
Performance and usability

• Effectiveness
• measure of relevance of retrieved images
• Efficiency
• system responsiveness and interactivity of system
• Usability
• measuring and evaluating human performance and
  preference

28
Sound retrieval by content

• Current applications, Soundfisher for example
  search using the following attributes
• Amplitude
• Pitch
• Duration
• Sample rate
• These attributes are not very user friendly
• Soundfisher finds similar soundbites but does not
  work well for voice or musical data

29
Sound retrieval interface

• How can you input your search data
• Use audio file (WAV, MP3, MID)
• Use keyboard (assumes some musical ability)
• Whistle / hum/ sing (assumes some musical
  ability)
• A major problem with searching a sound file is
  that the same melody played on different
  instruments will give a different sound wave

30
Sound retrieval by content

• Same music different instruments
• How can you perform a search for this melody?

31
Video Searching

• How to locate individual sequences from
• a database of video clips
• a long piece of video ( a whole movie)
• Can be done by
• using a textual database of the clips
• fast-forwarding through movie

32
Using Storyboards

• One approach is to use a storyboard representing
  the movie.
• Storyboard consists of a series of video stills
  representing the movie but much smaller in size.
• Storyboards can be browsed manually or using a
  query by example

33
Reverse Engineering the Video

• Storyboards need to be generated from the
  finished video sequence
• Each shot requires
• start point, end point and most representative
  picture
• first two can be found by comparing the amount
  and constancy of change between individual frames

34
Most Representative Frame

• The average frame can be generated by using an
  average colour for each pixel
• The best representative frame was the one that
  differed most from the average
• Storyboards could be used over networks when it
  is time consuming to download a complete video

35
Storyboard example
Original storyboard from The Wrong Trousers
Storyboard generated from the video The Wrong
Trousers