Query by Image Content

1
Query by Image Content
2
Content Based Image Retrieval

• Images generated by
• satellites, military reconnaissance/surveillance
  flights, fingerprinting, mug shots,
• scientific experiments, biomedical imaging
• Applications are
• Art Galleries and Museum Management
• Interior Design, Remote Sensing, Weather
  forecasting, Picture Archiving, Fabric and
  Fashion Design, Law Enforcement and Criminal
  Investigation, Scientific/Medical Database
  Management

3
Previous Approaches

• Set of attributes extracted manually and managed
  by standard DBMS
• Feature extraction / object recognition system
• Automated approaches to object recognition are
  computationally expensive, difficult and tend to
  be domain specific

4

• 2 categories of features
• Primitive/Low level
• Logical
• Images can be retrieved by
• colour texture
• sketch shape
• motion text
• objective attributes subjective attributes
• browsing

5
CBIR

• CBIR must also address broader info retrieval
  context
• medical images / diagnosis / treatment
• Query specification
• needs natural language
• show me images of snow covered mountains
• sketch shape required
• similarity based

6
Query by Image Content(QBIC)

• QBIC allows queries on large image and video
  databases based on
• example images
• user sketches and drawings
• colour and texture patterns
• demo at http//wwwqbic.almaden.ibm.com/
• (stamps database has large number of images)

7
QBIC

• Database population
• images and videos are processed to extract
  features - colours, textures, shapes, camera and
  object motion
• Database query
• User comprises graphical query
• Features generated from query and input to
  matching engine that finds image or video with
  similar features

8

• Illustration of a histogram for a grey scale
  image
• This is what information the computer sees
• You can tell from the histogram that the image is
  dark but nothing about the content.

9
Histogram matching

• Store histogram for each image
• Compare images using some statistical method
• Return sorted matches best fit first
• Problem - These would have the same histogram

10
Colour match

• Average colour
• adds up red, green, blue components of each pixel
• Colour position
• 6x8 or 9x12 grid overlays picture. For each block
  average Munsell colour and 5 most frequently
  occurring colours

QBIC interprets the virtual canvas as a grid of
coloured areas, then matches this grid to other
images stored in the database.
11
Shape matching
Aggregate shape measurement
Easier to automatically compute shape if
background is plain This was produced by
PaintShopPro
12
Textures

• Textures can be rough or smooth, vertical or
  horizontal etc

13
QBIC

• Queries can be
• Find images with approximately 30 red, 15 blue
• Find images with red round object

Drawn query
Best match
Matches in decreasing order of correlation
14
QBIC

• Queries can be
• Find images similar to this one
• Find sunsets with 30 red, 30 yellow
• Find images with 30 red and blue textured object
• Find images like this sketch

User sketch
Best match
15
Problems with shapes

• What would you input to retrieve both of these?
• Other problems
• Draw a horse returns dogs, cows etc
• Draw side view of cars doesnt retrieve front
  views
• Objects may be partially obscured

16
Colour similarity
All histograms should be similar to picture 1
17
Colour similarity and keyword
All histograms should be similar to picture
1 but keyword man was included. The picture of
the lion has the keyword MANE! The
following picture has keyword MANUFACTURING!
18
Colour Layout
Colour position should be similar to picture 1
19
Colour percentage
34 yellow 30 red 36 dont care
20
Colour percentage and keyword
34 yellow 30 red 36 dont care keyword SUNSET
21
Keyword only
keyword SUNSET
22
Custom Paint Query
Top two thirds of picture blue Lower
third yellow
23
Texture search
Texture search like first picture
24
Texture search
Texture search like first picture
25
Performance and usability

• Effectiveness
• measure of relevance of retrieved images
• Efficiency
• system responsiveness and interactivity of system
• Usability
• measuring and evaluating human performance and
  preference

26
Video Searching

• How to locate individual sequences from
• a database of video clips
• a long piece of video ( a whole movie)
• Can be done by
• using a textual database of the clips
• fast-forwarding through movie

27
Using Storyboards

• One approach is to use a storyboard representing
  the movie.
• Storyboard consists of a series of video stills
  representing the movie but much smaller in size.
• Storyboards can be browsed manually or using a
  query by example

28
Reverse Engineering the Video

• Storyboards need to be generated from the
  finished video sequence
• Each shot requires
• start point, end point and most representative
  picture
• first two can be found by comparing the amount
  and constancy of change between individual frames

29
Most Representative Frame

• The average frame can be generated by using an
  average colour for each pixel
• The best representative frame was the one that
  differed most from the average
• Storyboards could be used over networks when it
  is time consuming to download a complete video

30
Storyboard example
Original storyboard from The Wrong Trousers
Storyboard generated from the video The Wrong
Trousers