Image Search

1
Image Search

• Using Color and Texture

2
Project NETRA

• http//www-iplab.ece.ucsb.edu/netra/Netra.html
• Examples edge flow http//www-iplab.ece.ucsb.edu
  /netra/examples/Netra.html

3
Efficient Color Representation

• For efficiency, Yining Deng,et.al.1 proposed
  using a dominant color descriptor, which is the
  peak histogram value for each of three colors
  RGB.
• Previous implementations used colors from 10s to
  100s of bins, time and storage were excessive
• Singular Value Decomposition used to reduce
  computational complexity, however the reduction
  is not related to features and will produce
  errors
• Hilbert curve fitting- features close together in
  the original may be far apart on the curve.
• Color Moment Descriptors (mean, variance, and
  third order moment) perform slightly worse than a
  high level histogram. The mean colors may be
  quite different from any of the original colors
• Since they perform satisfactorily even with this
  disadvantage, I decided to implement this
• The Dominant color descriptor takes the dominant
  colors from a histogram and searches for colors
  within a distance of these colors (each of which
  have a percentage of each of RGB)

4
Efficient Color Representation

• Image is first segmented using the edgeflow
  algorithm
• Color clustering is performed on each segment to
  obtain representative colors
• The dominant color descriptors is then
• F ci, pi, i 1, , N
• N is the number of color clusters
• ci is the color vector
• pi is its percentage where ?pi 1
• Three to four colors are sufficient to describe
  an image region
• Complexity of a search is O(m n) where n is the
  number of colors in a query, m is the number of
  nodes accessed per query color

5
Texture Features

• B.S Manjunath, W.Y. Ma2,3 suggest using a Gabor
  function and its frequency components, the
  Fourier transform
• Higher frequency components may be sufficient to
  retrieve images
• Calculating the entire Gabor function for a 512 x
  512 image takes 9.3 seconds on a SUN Spark 20,
  search and retrieval takes 1.02 seconds.
• Using the adaptive filter that they suggest using
  four feature components takes 2.3 seconds for
  feature extraction and 0.7 seconds for adaptive
  filter selection and 0.1 seconds for retrieval
• This was too long
• Used edge flow to segment images\
• Mean and variance, presentation by Dr Lu
• Using the Mean, Standard Deviation, Third and
  Forth Moment
• Compatible with the color calculations,
  efficient, I have adopted this method

6
Implementation Parameters

• Web based so that anyone can upload pictures and
  try it out
• Use of a robust language
• Java
• Easy interface with web pages
• Capable of performing appropriate calcs on the
  server
• Used several Java Servlets
• ImageInfo http//www.geocities.com/SiliconValley/L
  akes/6686/, gets width, height, etc of an image
• MultipartRequest http//www.geocities.com/jasonpel
  l/programs.html, parses a multipart form and
  writes out any files found
• ServletMultipartRequest http//www.millstone.org/d
  ocumentation/apidocs/org/millstone/webadapter/Serv
  letMultipartRequest.html, wrapper for
  MultipartRequest for use in Servlets

7
Implementation Parameters

• JSP
• Java compiled on the fly
• Java, SQL, and HTML mixed on the same page
• SQL
• Used Hypersonic SQL, an all Java implementation
• Find a server that has all of these
• Although I already have a web server (and FTP) on
  2 home machines, determining how to set up a JSP
  and HSQL server would be too time consuming (plus
  Oracle SQL uses busy-waiting and takes up all
  your clock cycles)
• http//www.mycgiserver.com/
• Free, although they email requests for donations
  for an upgrade
• Sometimes slow, they are planning an upgrade to
  speed things up
• http//www.webappcabaret.com/ only 1 month free
• http//www.brinkster.com/
• Moved the start page here because it is faster
  (and also free)

8
Implementation

• I didnt get as far as I had expected, but have
  images being uploaded and added to a DB
• http//www25.brinkster.com/kenwschmidt/search/inde
  x.html

9
Future Implementation

• Texture
• By Mean and variance
• Mean x 1/n ? xi
• Standard Deviation (second moment)
• 1/n ? (xi x )2
• Third Moment 1/n ? (xi x )3
• Fourth Moment 1/n ? (xi x )4
• Search the DB for images within Y of these four
  texture parameters
• Color
• Record the mean of each of the three colors
• Search for images within Y of these

10
How to Implement?

• Found some implementations which would be easy
  but they were not standard SUN Java, therefore
  the server that I had chosen probably doesnt
  have them available
• Searching SUNs documentation, I found several
  ways to access pixels
• DataBuffer
• SampleModel
• BandedSampleModel
• ColorModel

11
How to Implementfootnote
Java has some built in functions that are
interesting
float elements 0.0f, -1.0f,
0.0f,-1.0f, 4.f, -1.0f, 0.0f, -1.0f, 0.0f
BufferedImage bimg new
BufferedImage(bw,bh,BufferedImage.TYPE_INT_RGB)
Kernel kernel new Kernel(3, 3,
elements) ConvolveOp cop new
ConvolveOp(kernel,
ConvolveOp.EDGE_NO_OP,null)
cop.filter(bi,bimg)
12
How to Implementfootnote
byte reverse new byte256
for (int j0 jlt200 j)
reversej(byte)(256-j)
ByteLookupTable blutnew
ByteLookupTable(0, reverse)
LookupOp lop new LookupOp(blut, null)
lop.filter(bi,bimg)
13
How to Implementfootnote
RescaleOp rop new RescaleOp(1.5f, 1.0f,
null) rop.filter(bi,bimg)
14
How to Implementfootnote
Sharpening with Convolution float elements
0.0f, -1.0f, 0.0f, -1.0f, 5.f, -1.0f,
0.0f, -1.0f, 0.0f   Kernel kernel new
Kernel(3,3,elements) ConvolveOp cop new
ConvolveOp(kernel, ConvolveOp.EDGE_NO_OP, null)
cop.filter(bi,bimg)
15
How to Implement

• Each of these did too much, I just wanted access
  to the raw pixels
• java.awt.image.PixelGrabber

16
Pseudo Code
for (int j 0 j lt h j) for (int i 0 i
lt w i) pixel pixels j w
i red (pixel gtgt 16) 0xff green
(pixel gtgt 8) 0xff blue (pixel )
0xff redSum2 pow ( ( red - redMean ), 2
) greenSum2 pow ( ( green- greenMean ), 2
) blueSum2 pow ( ( blue- blueMean ), 2
) redSum3 pow ( ( red - redMean ), 3
) greenSum3 pow ( ( green- greenMean ), 3
) blueSum3 pow ( ( blue- blueMean ), 3
) redSum4 pow ( ( red - redMean ), 4
) greenSum4 pow ( ( green- greenMean ), 4
) blueSum4 pow ( ( blue- blueMean ), 4
) redMoment2 redSum2 / noOfPixels greenMo
ment2 greenSum2 / noOfPixels blueMoment2
blueSum2 / noOfPixels redMoment3 redSum3 /
noOfPixels greenMoment3 greenSum3 /
noOfPixels blueMoment3 blueSum3 /
noOfPixels redMoment4 redSum4 /
noOfPixels greenMoment4 greenSum4 /
noOfPixels blueMoment4 blueSum4 /
noOfPixels overallMoment2 ( redMoment2
greenMoment2 blueMoment2 ) / 3 overallMoment3
( redMoment3 greenMoment3 blueMoment3 ) /
3 overallMoment4 ( redMoment4 greenMoment4
blueMoment4 ) / 3

• for (int j 0 j lt h j)
• for (int i 0 i lt w i)
• pixel pixels j w i
• red (pixel gtgt 16) 0xff
• green (pixel gtgt 8) 0xff
• blue (pixel ) 0xff
• redSum red
• greenSum green
• blueSum blue
• redMean redSum / noOfPixels
• greenMean greenSum / noOfPixels
• blueMean blueSum / noOfPixels
• overallSum redSum greenSum blueSum
• overallMean overallSum / noOfPixels

17
References

1. An Efficient Color Representation for Image
   Retrieval, Yining Deng, B. S. Manjunath, Charles
   Kenney, Michael S. Moore, Hyundoo Shin, IEEE
   TRANSACTIONS ON IMAGE PROCESSING, VOL. 10, NO. 1,
   JANUARY 2001
2. Texture Features for Browsing and Retrieval of
   Image Data, B.S Manjunath, W.Y. Ma, IEEE
   TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE
   INTELLIGENCE, VOL. 18, NO. 8, JANUARY 1996
3. A Texture Thesaurus for Browsing Large Aerial
   Photographs, Wei-Ying Ma and B. S. Manjunath,
   JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION
   SCIENCE. 49(7)633648, 1998
4. Texture Features and Learning Similarity,
   Wei-Ying Ma and B. S. Manjunath, 1063-6919/96IEEE