Image Comparison Method for MultiImage Matching

1
Image Comparison Method for Multi-Image Matching
Vision, Imaging, Video Autonomous Research
Laboratory
University of Ottawa

• By
• Colince Donfack
• April 11th, 2005 ELG 5378
  Image Processing

2
Objectives

• Use image comparison techniques to improve
  features-based multi-image matching.
• Find the right sequence of a set of images.

3
Outline
1. Introduction 2. Content-Based Image Retrieval
(CBIR) 3. Review of some CBIR techniques
4. Method 5. Experimental results 5. Conclus
ion 6. References
4
Cubic Panoramas

• Multi-sensor Ladybug1 camera captures 6
  overlapping images.
• The images are stitched to form a large spherical
  panorama.
• Cube images are obtained by rendering
  perspective views with 90o field of view 1

The Ladybug Camera1

• Advantages
• 360o field of view
• Reduced cost of storage
• Fast rendering

1 Point Grey Research - www.ptgrey.com
5
Problem Description

• Some facts
• Regular sizes images have in average N 500
  features
• Distinctive feature vectors have over 100
  elements
• One cubic panorama has around 3000 features
• Feature similarity often measured using the
  Euclidian distance
• Main issue matching computation cost
• Increases quadratic function of the number of
  features
• If cube faces are matched separately, we will
  need to compute 6N2 distances instead of (6N)2
  ( 1.5 M vs. 9 M if N 500)
• Other issues with matching panoramic images
  directly (more features)
• Probability of having false matches increases
• Enforcing matching distinctiveness seriously
  reduce the number of matches
• Applying uniqueness and symmetry constraint ?
  multiply matching time

6
Project motivation

• We did a small experiment in MATLAB to justify
  the project.
• We used the SIFT Lowe to find and match
  features
• First we reformat cubic images to get rid of
  useless images

7
Project motivation

• We compared the time needed to compute match
  features in the 6 images together and separately

• Theoretically the time saving is around
• Practically we obtain

8
Content-Based Image Retrieval (CBIR)

• Generally used to retrieve images from a database
  that is the most similar to given query.

Fig. Diagram for content-based image retrieval
system 3
9
Color Statistics

• Simple color histogram
• A way to represent the color of an image.
• Many color spaces (RGB, CIE Lab, CIE L,u,v,
  HSV ) but for image comparison RGB is sufficient
• Pros invariant to rotation and translation
• Cons different images can have the same color
  histogram.
• Similarity measure

• Color Coherence Vector 4
• Add spatial information to the histogram
• For each color, pixels are classified as coherent
  (ai) or non coherent (bj)
• Requires more computation time
• Similarity measure

10
Color Statistics

• Color Correlogram 5
• Encodes color-spatial information in
  co-occurrence matrix
• The autocorrelogram is often used to reduce the
  storage and the computation cost.
• This expresses the probability of having a pixel
  with the same color at a distance d
• The computation cost increases with d

• Color moments
• Average color
• Variance
• Skewness

11
Texture Properties

• Texture characterizes the way colors or gray
  levels are distributed in images
• Statistical models
• Tamura features (coarseness, contract,
  directionality)
• Markov random field
• Co-occurrence matrix
• Work well on regularly textured images
• Spectral models
• Gabor filter features
• Not invariant to rotation and scale
• Wavelet transform features
• Not invariant to image shift
• Extract spectral energy from images

12
Layered Color indexing (LCI) 6

• Add spatial component to a simple color histogram
  while minimizing the computation cost.
• Reduce the colors set to 64 (4x4x4).
• Segment the image in 4 frequency layers and built
  the image histogram of 256 bins from the 4 layers
  histograms
• First compute the Laplacian of the gray-level
  image.
• Its magnitude characterizes the sharpness of the
  image areas
• Apply threshold to obtain different layers

h Gaussian filter
Fig. layer classification method 6
13
Block-Based Methods

• Divides images into sub-blocks and built the main
  histogram from sub-blocks histograms.
• Efficient but sensitive to rotation, translation
  and scale changes

• Fig. The block-based method 7

14
Segmentation Methods

• Used to obtain middle level features such as
  shapes
• Two categories
• Boundary-based
• Region-based
• A shape descriptor is used to represent the shape
• Segmentation methods are difficult to apply in an
  uncontrolled scene.
• The difficulty increases with the complexity of
  the image
• These methods will not work for image comparison
  unless
• The objects are constrained to be entirely within
  images
• The shape descriptor is invariant to scale and
  rotation.

15
Methodology

• Requirements for the method
• Fast to compute
• Features distance should reflect images
  similarity
• Color histograms and Frequency Layer Indexing has
  been tested since theyre the most
  computationally efficient methods.
• Images resample to 512x512, (even up to 128x128)
  and quantized to 64 colors.
• A 64-bin histogram was created with the quantized
  pixels colors

16
Experiment setup

• 2 indoor sequences are used for the test

Sequence 1 (20 images) Sequence 2 (25
images)

• Because the images are taken from a lab there are
  a lot similarities between them.
• Sequence 2 is more challenging as many views are
  shared.

17
Experiment setup

• For the cube face matching we measured the
  average precision
• Pi 0 if the 6 faces doesnt match
• For sequence reordering performance we measured
  the average recall

,2
for
18
Cube Faces Mismatch Correction

• We assume the faces Up and Down are easy to
  match.

• Initialize the cube configuration 4x4 Matrix Mc
  to zeros
• Fill Mc with the 4 possible configurations
  (columns)
• From matched cube faces compute the support
  vector Vs.
• Compute sumS S Vsi
• - sumS 12 (3333) ? each face has 3
  supports , no correction needed
• - sumS 4 (1111) ? each face has 1
  support , 2-2 tied, hard to decide
• - sumS 6 (2220) ? 1 match is wrong,
  correction.
• - sumS 2 (2000) ? 2 faces support each
  other, correction suggested

19
Results Color histogram

• Panorama 1
• Up front Back Down
  Left Right

• Panorama 2
• Up front Back
  Down Left Right

20
Results Color histogram

• Cube face matching

Sequence 1
P 1
Sequence 2
Pc 0.95
P 0.70
Sequence reordering
Sequence 1
recall 0.9
Sequence 2
recall 0.58
21
Results LCI

• Image
• Layer 4 (High Freq.)

Layer 1 (Low Freq.) Layer 2
Layer 3
22
Results LCI (high Freq. layer)

• Cube face matching

Sequence 1
P 1
Sequence 2
Pc 1
P 0.29
Sequence reordering
Sequence 1
recall 0.55
Sequence 2
recall 0.44
23
Conclusion

• We evaluated some CRIR technique in order to
  accelerate multi-image matching.
• We found that very simple method such as Color
  Histogram is quite distinctive for a small set of
  images
• The computation cost required is very little
  compare to large size image matching.
• The LCI score was lower for cube face matching
  but generally only one face match was not correct
  and the precision was 1 after correction.
• The LCI using the highest frequency perform less
  than the simple color histogram because in the
  scene (lab), most objects (chair, computer,
  table, frames) contours are the same.
• Combining all layers should improve the LCI
  method.
• Both methods perform very fast (less than a
  second).

24
References

• D. Bradley, A. Brunton, M. Fiala, G. Roth,
  Image-based Navigation in Real Environments
  Using Panoramas, IEEE International Workshop on
  Haptic Audio Visual Environments and their
  Applications (HAVE 2005), Ottawa, Ontario.
• D. G. Lowe. Distinctive image features from
  scale-invariant keypoints, in International
  Journal of Computer Vision, vol. 60, no. 2, pp.
  91-110, 2004.
• Fuhui Long, Hongjiang Zhang, David D. Feng
  Fundamentals of Content-based Image retrieval,
  in Multimedia Information Retrieval and
  Management - Technological Fundamentals and
  Applications D. Feng, W.C. Siu, and H.J.Zhang.
  (ed.), Springer, 2002.
• Greg Pass, Ramin Zabih, and Justin Miller.
  Comparing images using color coherence vectors.
  In Proceedings of ACM Multimedia 96, pp 65.73,
  Boston MA USA, 1996.
• J. Huang, S. R. Kumar, M. Mitra, W. Zhu, and R.
  Zabih, Image indexing using color correlograms,"
  in Proc. Computer Vision and Pattern Recognition,
  pp. 762-768, 1997.
• G. Qiu and K.-M. Lam, Frequency layered color
  indexing for content-based image retrieval, IEEE
  Trans. Image Processing, vol. 12, no. 1, pp.
  102-113, Jan. 2003.
• Valtteri Takala, Timo Ahonen, Matti Pietikäinen
  Block-Based Methods for Image Retrieval Using
  Local Binary Patterns. SCIA 2005 882-891.