1
Saliency-Based Visual Attention
Computational Modeling of Visual Attention,
Itti, Koch, (Nature Reviews Neuroscience 2001
) A Model of Saliency-Based Visual Attention for
Rapid Scene Analysis , Itti, Koch and Nieburs
(IEEE PAMI 1998)
Zhengping Ji
2
Overview

• Background
• System architecture
• The saliency map
• Preprocessing
• Feature maps
• Feature integration
• Focus of attention
• Results
• Conclusion

3
Related Work

• Feature Integration Theory, Treisman Gelade,
  1980.
• Computational model of bottom-up attention, Koch
  and Ullman, 1985
• Saliency map is believed to be located in the
  posterior parietal cortex (Gotlieb, et al., 1998)
  and the pulvinar nuclei of the thalamus (Roinson
  Peterson, 1992)

4
Architecture
5
Gaussian Pyramids

• Repeated low-pass filtering
• 0, 1, 2, 3,,8
• I(0) is original input

6
Preprocessing

• Original image with red, green, blue channels
• Intensity as I (r g b)/3
• Broadly tuned color channels
• R r - (g b)/2
• G g - (r b)/2
• B b - (r g)/2
• Y (r g)/2 - r g/2 - b

7
Preprocessing
Intensity
B
R
G
Y
8
Center-surround Difference

• Achieve center-surround difference through
  across-scale difference
• Operated denoted by Q Interpolation to finer
  scale and point-to-point subtraction
• One pyramid for each channel I(s), R(s), G(s),
  B(s), Y(s)where s Î 0..8 is the scale

9
Intensity Feature Maps

• I(c, s) I(c) Q I(s)
• c Î 2, 3, 4
• s c d where d Î 3, 4
• So I(2, 5) I(2) Q I(5) I(2, 6)
  I(2) Q I(6) I(3, 6) I(3) Q I(6)
  
• ? 6 Feature Maps

10
Colour Feature Maps

• Similar to double-opponent cells (Prim. V. C)
• Red-Green and Yellow-Blue
• RG(c, s) (R(c) - G(c)) Q (G(s) - R(s))
• BY(c, s) (B(c) - Y(c)) Q (Y(s) - B(s))
• Same c and s as with intensity

R-G
G-R
B-Y
Y-B
R-G
G-R
B-Y
Y-B
11
Orientation Feature Maps

• Create Gabor pyramids for q 0º, 45º, 90º,
  135º
• c and s again similar to intensity

12
Normalization Operator

• Promotes maps with few strong peaks
• Surpresses maps with many comparable peaks
• Normalization of map to range 0M
• Compute average m of all local maxima
• Find the global maximum M
• Multiply the map by (M m)2

13
Normalization Operator
14
Conspicuity Maps
15
Saliency Map

• Average all conspicuity maps

16
Neural Layers
Stimulus

• Saliency Map (SM) modeled as layer of leaky
  integrate-and-fire neurons
• SM feeds into winner-take-all (WTA) neural
  network
• Inhibition of Return as transient inhibition of
  SM at FOA

SM
-
Inhibition of Return

WTA
FOA shifted to position of winner
17
Example of Operation
Inhibition of return
18
Results
Image Saliency Map High saliency
Locations (yellow circles)
19
Shifting Attention

• Using 2D winner-take-all neural network at
  scale 4
• FOA shifts every 30-70 ms

20
Summary

• Saliecy map can be broken down into main steps
• Create pyramids for 5 channels of original image
• Determine feature maps then conspicuity maps
• Combine into saliency map (after normalizing)
• The key idea of saliency map is to extract local
  spatial discontinuities in the modalities of
  color, intensity and orientation.
• Use two layers of neurons to model shifting
  attention.
• Model appears to work accurately and robustly
  (but difficult to evaluate)

21
Discussion

• No top-down attention modeling, e.g., top-down
  spacial control, obejct-based attention.
• Biologically plausible?
• Neuromorphic architecture?
• In which way the top-down and bottom-up processes
  are related?
• In which way the attention and recognition are
  integrated and interacted with each other?

22
References

• Itti, Koch, and Niebur A Model of
  Saliency-Based Visual Attention for Rapid Scene
  AnalysisIEEE PAMI Vol. 20, No. 11, November
  (1998)
• Itti, Koch Computational Modeling of Visual
  AttentionNature Reviews Neuroscience Vol. 2
  (2001)