Content Based Image Retrieval

1
Content Based Image Retrieval

• Jafar Muhammadi
• Muhammadi _at_ ce.sharif.edu

2
Outline

• Introduction
• Text-based approach
• Visual features-based approach
• Query based on text and content
• Region based approach
• Semantic-sensitive approach
• Open research issues

3
Finding Latent Semantics

• We want to transform low-level features to a
  higher level of meaning
• Used for dimension reduction (Searching in
  high-dimensional spaces)
• More importantly, it creates clusters of
  co-occurring features
• So-called concepts

4
Text based Retrieval

• Index images using keywords or descriptions
  (google, piction)
• Sources of keywords when images have been
  extracted from a web
• page
• The URL and filename
• The alt attribute ltimg alt"..." ...gt
• Other words from the web page, its title,
  description (meta tags), etc.
• Easier to design and implement, fast
  execution
• Accepted approach for high value pictures
• A picture is worth, and can require too many
  words
• Query word may not appear as keyword
• Surrounding text may not describe the image

5
Piction retrieval based on captions

• Collection several hundred news photographs,
  mostly of people
• Use captions to identify whos in the picture and
  where, then find matching faces
• Relies heavily on text

6
Feature based Retrieval

• Features that human visual system use
• Color suitable for color images
• Texture visual patterns, surface propertis
• Shape boundary of real world objects, edges
• Motion
• handles low level semantic queries
• Many Features can be extracted
• - can not handle higer level queries

7
WebSeek search for images on the Web

• Collection 500,000 Web images and videos
• Filename and surrounding text determine subject
• Simple content-based techniques
• Global color histograms
• Color in localized regions

8
QBIC image content plus annotation

• QBIC searching multimedia databases
• Collections
• General collections, Video clips, Fabric samples,
  etc.
• Retrieval based on color, texture, shape, sketch
• Relies (sometimes) on human annotation of image
  or objects and Textual information like title,
  subject, object labels
• Wide range of applications
• Bridging gap between database- and visual aspects
• Direct query query by example

9
Indexing

• Sequential scanning very slow for large databases
• Solution treat feature vectors as points in n-d
  space, and employ a multi-dimensional indexing
  method
• Two problems
• Distance function quadratic size of feature
  vector (and cross-talk renders multi-key indexing
  inapplicable)
• Multi-dimensional indexing requires space and/or
  time exponential on n

10
Reducing problem complexity

• By mapping the feature vector to a lower
  dimensional space, we can obtain a distance
  function underestimating the actual distance.
  This reduces the problem complexity with the
  sacrifice of introducing some false hits.

11
Region based approach

• Extract objects from images first
• Handle object based queries (e.g., Find images
  with objects that are similar to given object )
• Reduce feature storage adaptively
• - Object Segmentation is very difficult
• - User interface region marking, feature
  combination

12
Blobworld represent image regions

• to find objects ? look for coherent regions

13
Group pixels into regions

• Expectation-Maximization (EM)
• Assign each pixel to a Gaussian in color/texture
  space ? segmentation

• Describe regions color, texture, shape
• Color Color histogram within region
• Texture Contrast and directionality
• ? stripes vs. spots vs. smooth
• Shape Fourier descriptors of contour

14
Example Query
15
Interaction with user
16
Result of Query
17
Semantic-sensitive approach

• Motivation
• Human Object segmentation relies on knowledge
• Precise computer image segmentatoin is a very
  difficult open problem
• Hypothesis It is possible to build a robust
  computer matching algorithms without first
  segmentation the image accurately.
• SIMPLIcity Semantic-sensitive Integrated
  Matching for Picture Libraries
• Wavelet-based feature extraction for fast
  segmentation
• Integrated Region Matching

18
Semantic classes

• Features that are used (Feature vectors)
• Color
• Texture
• Shape
• Location
• Semantic types of the images
• Natural photograph vs. artificial generated by
  computer
• Textured or untextured
• Indoor vs. outdoor
• Objectionable or benign
• SIMPLIcity also uses the similarity measure
  between images on the image semantics
• flowers often appear with green levels boats
  often appear with water

19
Segmentation Matching

• Fast Image Segmentation
• Partition an image into 4x4 blocks
• Extract wavelet-based features from each block
• Use k-means to cluster feature vectors into
  Regions
• IRM Integrated Region Matching
• IRM defines an image-to-image distance as a
  weighted sum of region-toregion distances
• Reduce the influence of inaccurate segmentation
• Matching algorithm use a fuzzy approach

20
Robustness to image Alterations

• 10 brighten on average
• 8 darken
• Bluring with a 15x15 Gaussian filter
• 70 sharpen
• 20 more saturation
• 10 less saturation
• Shape distortions
• Cropping
• Shifting
• Rotation

21
Results

• query image is a landscape image on the
  upper-left corner of the block of images

22
Results

• The query image is a photo of food

23
Results

• The query image is a portrait image that probably
  depicts life in Africa

24
Results

• The query image is a textured image

25
Results

• Robustness to Scaling, Shifting, and Rotation

26
Results

• The robustness of the SIMPLIcity system to image
  cropping and scaling

27
Results

• The retrieval results made by the SIMPLIcity
  system with shifted query images

28
Results

• retrieval results made by the SIMPLIcity system
  with a rotated query image

29
Ongoing Research

• Automatic modeling and learning of concepts for
  image indexing
• Training concepts Basic building blocks in
  determining the semantic meaning of images
• Basic object flower, beach,
• Object Composition Buildinggrassskytree
• Location Asia, Venice
• Time night sky, winter frost
• Abstract Sports, Sadness
• 600 concepts can be learned automatically

30
Future works

• Explore new methods for better accuracy
• Refine statistical modelling of images
• Learning from 3D
• Refine matching schemes
• Apply these methods to
• special image databases (e.g., art, biomedicine)
• Very large databases

31
Open Research Issues

• Study of human perception of image content from a
  pschologysical level
• High Dimensional indexing
• Human in the loop of retrieval
• High level concept extraction from low leve
  visual features
• Web oriented retrieval
• Learning models
• New models for image regionization based on
  feature vectors
• New methods for matching regions and feature
  vectors

32
Any Question?

• ?

33

• Thank you