Content-based Image Retrieval

1
Content-based Image Retrieval

• Hai Le
• Supervisor Sid Ray

2
Content-based Image Retrieval

• Introduction
• CBIR fundamentals
• Overview of the system
• Relevance feedback
• Results and Discussion
• Conclusion

3
Introduction

• Search and retrieval of image databases
• Retrieval by query, such as an example image
• Application areas include, weather
  forecasting, iiiiiiiimedical research, fabric
  design, WWW search iiiiiiiijust to name a few..
• CBIR systems which are commercially
  iiiiiiiiavailable include IBMs QBIC, Blobworld,
  iiiiiiiiVisualSeek, Virage etc.

4
CBIR Fundamentals
Y
N
5
CBIR Fundamentals
Y
N
6
CBIR Fundamentals

• User is searching for a specific target item
  from iiiiiiiiia known database
• User is searching for a class of similar items
• Query by example image, drawing etc.
• Exploiting multiple queries to refine searches
• Use of segmentation or object selection to
  iiiiiiiiisearch for image specifics

7
CBIR Fundamentals
Y
N
8
CBIR Fundamentals

• Low level image features used as classifiers
• Most commonly used features include
• Colour
• Texture
• Shape
• Transform domain features
• DCT
• Wavelet filters

9
CBIR Fundamentals
Y
N
10
CBIR Fundamentals

• Features are represented as a numerical value
• Similarities between images are determined by
  iiiiiiiiidifferences between values
• Typically, queries are viewed as points in a
  iiiiiiiiimultidimensional feature space, and the
  iiiiiiiiidistance between points is determined
• Common distance functions include
• Euclidean
• City block
• Images are indexed in terms of their distance
  iiiiiiiiifrom the query point

11
CBIR Fundamentals
Y
N
12
CBIR Fundamentals

• Top N most similar images are retrieved
• Searches can be refined by user feedback
• Data which is fed back is usually used to
  iiiiiiiidetermine significant features
• A features influence on a query can be
  iiiiiiiiincreased/decreased by applying a
  weighting iiiiiiiifacture during distance
  calculations
• Images are re-indexed

13
Overview of the System

• Query by example image
• 3 selectable features for querying, with 189
  iiiiiiiisub features
• Selection between 4 weighting schemes
• Incorporates user interaction, retrieved
  images iiiiiiiiare selected as either relevant or
  non-relevant

14
Overview of the System

• Features used in the system
• Colour histogram
• Colour moments
• Edge histogram

15
Overview of the System

• Colour histogram using linear quantised HSV
  iiiiiiiiicolour space
• 18 hues (20 degrees separation), 3
  iiiiiiiiisaturations, 3 values, total of 162
  colours
• Each bin is proportional ie.
• Colour moments calculated by taking the
  iiiiiiiiiaverage, standard deviation and cube
  root of iiiiiiiiithe third moment, of each of the
  HSV channels

16
Overview of the System

• Edge histogram is derived by the number of
  iiiiiiiiedge pixels present in the image and the
  iiiiiiiidirection of the edge pixels
• Sobel edge detector is applied on the image,
  if iiiiiiiithe edge strength surpasses a
  threshold the iiiiiiiidirection is recorded and
  quantised
• A histogram is generated from all the
  iiiiiiiidirectionality values
• Each bin represents a sub feature

17
Overview of the System

• Euclidean metric used to measure distance
• Features which constitute large values
  iiiiiiiiovershadow those with small values
• Gaussian normalisation is applied to give
  equal iiiiiiiiemphasis on all features

18
Overview of the System

• Most similar images are those closest to the
  iiiiiiiiquery image in the feature space
• Images are indexed by their distance values
• The top N images are retrieved and displayed

19
Relevance Feedback

• An image object can be modeled as
• D is the image data, F fi , features, R
  rij iiiiiiiia set of representation or
  subfeatures, iiiiiiiieg.
  Histogram bins
• An objects similarity to the query is
  calculated iiiiiiiiby rij and its corresponding
  weight wij and a iiiiiiiisimilarity measure S
• Weighted Euclidean

20
Relevance Feedback

• Different queries are more reliant on certain
  iiiiiiiiifeatures than others
• Varying wijs changes emphasis of features
  iiiiiiiiiduring distance calculation
• Term weighting using relevance feedback
  iiiiiiiiiinclude
• Density estimation
• Support vector machine learning
• Self-organising maps

21
Relevance Feedback

• System training data uses binary feedback,
  iiiiiiiiieither positive example or negative
  examples
• Number of retrieved images is 4 plus original
  iiiiiiiiiquery image
• Of the 4 examples, images marked as relevant
  iiiiiiiiiare placed in a positive subset,
  unselected iiiiiiiiiimages are placed in a
  negative subset
• Unretrieved images are left indifferent

22
Relevance Feedback

• Weighting scheme devised by Rui et. al
• Form an M x N matrix, M number of positive
  iiiiiiiiimages, N number of features
• For each column of the matrix calculate the
  iiiiiiiistandard deviation
• The weight wij is calculated by taking

23
Relevance Feedback

• Using both positive and negative images
• If means of rij positive and rij negative are
  iiiiiiiiisimilar, feature is insignificant
• Using variance

24
Relevance Feedback

• Positive images are similar because of a
  iiiiiiiiispecific characteristic, negative images
  are iiiiiiiiidifferent because of a number of
  iiiiiiiiicharacteristics
• Using negative standard deviation

25
Results and Discussion

• Image database of 200, divided into individual
  iiiiiiiigroups ranging from 4-10 images per group
• group of 10 images selected for test case
• each image selected as query, precision and
  iiiiiiiiirecall calculated for each of the 4 term
  iiiiiiiiweighting schemes after 1 iteration
• Precision and recall averaged over the 10
  iiiiiiiiimages

26
Results and Discussion
27
Results and Discussion
28
Results and Discussion

• Employing negative examples, showed
  iiiiiiiiiimprovement
• Type 3 showed slowest degradation
• Type 4 showed significant improvement

29
Conclusion

• CBIR system developed incorporating
• variety of features
• functional interface
• user friendly feedback mechanism
• Improved term weighting scheme over used
  iiiiiiiiin the MARS CBIR system
• Maintains same simplistic interface as MARS
• Future work
• image classification
• segmentation/local features
• better feature representation eg. Gabor
  filtering