The Selective Tuning Model of Visual Attention

1
A Neural Model for Detecting and Labeling Motion
Patterns in Image Sequences Marc Pomplun1 Julio
Martinez-Trujillo2 Yueju Liu2 Evgueni
Simine2 John Tsotsos2 1UMass Boston 2York
University, Toronto, Canada
2
Data Flow Diagram of Visual Areas in Macaque
Brain
Bluemotion perception pathway
Greenobject recognition pathway
3
Receptive Fields in Hierarchical Neural Networks
4
Receptive Fields in Hierarchical Neural Networks
neuron A
in top layer
5
Problems with Information Routing in Hierarchical
Networks
6
The Selective Tuning Concept (Tsotsos, 1988)
processing
pyramid
7
Hierarchical Winner-Take-All
top-down, coarse-to-fine WTA hierarchy for
selection and localization unselected
connections are inhibited
8
Selection Circuits
unit and connection
l
a
y
e
r

l

1
in the interpretive network
unit and connection
in the gating network
unit and connection
in the top-down bias network
l
a
y
e
r

l
l
a
y
e
r

l

-
1
I
9

• 3D Visualization of the Selective Tuning Network

Red WTA phase 1 active
Green WTA phase 2 active
Blue inhibition
Yellow WTA winner
10
The Motion Perception Pathway
MST
MT
V1
input
11
What do We Know about Area V1?

• cells have small receptive fields
• each cell has a preferred direction of motion

• there are three types of motion speed selectivity

12
What do We Know about Area MT?

• cells have larger receptive fields than in V1
• like in V1, each cell has a preferred combination
  of the direction and speed of motion
• MT cells also have a preferred orientation of the
  speed gradient

13
What do We Know about Area MST?

• cells respond to motion patterns such as
• translation (objects shifting positions)
• rotation (clockwise and counterclockwise)
• expansion (approaching objects)
• contraction (receding objects)
• spiral motion (combinations of rotation and
  expansion/contraction)
• the response of a cell is almost independent on
  the position of the motion pattern in the visual
  field

14
The Motion Hierarchy Model V1

• V1 receives image sequences as input and extracts
  the direction and speed of motion

counterclockwise rotation
clockwise rotation
contraction
expansion
15
The Motion Hierarchy Model V1

• V1 is simulated as 60x60 hypercolumns
• each column contains 36 cells one for each
  combination of direction (12) and speed tuning
  (3)
• direction and speed selectivity are achieved with
  spatiotemporal filters
• these filters process local information from the
  last seven images in the sequence
• example cells tuned towards upward motion

16
The Motion Hierarchy Model MT

• MT is simulated as 30x30 hypercolumns
• each column contains 432 cells one for each
  combination of direction (12) speed (3), and
  speed gradient tuning (12)
• problem how can gradient tuning be realized from
  activation patterns in V1?
• solution detect gradient differences across the
  three types of speed selective cells
• this solution leads to a simple network structure
  and remarkably good noise reduction
• the activation of an MT cell is the product of
  its activation by direction, speed, and gradient

17
The Motion Hierarchy Model MST

• how can MST cells detect motion patterns such as
  rotation, expansion, and contraction based on the
  activation of MT cells?

• idea the presence of these motion patterns is
  indicated by a consistent angle between the local
  movement and speed gradient

18
The Motion Hierarchy Model MST
direction of movement
orientation of speed gradient
19
The Motion Hierarchy Model MST

• MST cells integrate the activation of MT cells
  that respond to a particular angle between motion
  and speed gradient
• this integration is performed across a large part
  of the visual field and across all 12 directions
• therefore, MST can detect 12 different motion
  patterns
• we simulate 5x5 MST hypercolumns, each containing
  36 neurons (tuned for 12 different motion
  patterns, 3 different speeds)

20
MST
MT
V1
21
Simulation clockwise rotation
22
Simulation counter- clockwise rotation
23
Simulation receding object
24
Attention in the Motion Hierarchy
What happens if there are multiple motion
patterns in the visual input?

• Visual attention can be used to
• determine the type and location of the most
  salient motion pattern,
• focus on it by eliminating all interfering
  information,
• sequentially inspect all objects in the visual
  field.

25
(No Transcript)
26
(No Transcript)
27
(No Transcript)
28
Conclusions and Outlook

• the motion hierarchy model provides a plausible
  explanation for cell properties in areas V1, MT,
  and MST
• its use of distinct speed tuning functions in V1
  and speed gradient selectivity in MT leads to a
  relatively simple network structure combined with
  robust and precise detection of motion patterns
• visual attention is employed to segregate and
  sequentially inspect multiple motion patterns

29
Conclusions and Outlook

• the model predicts inhibition of visual functions
  around any attended motion pattern
• the model also predicts that different motion
  patterns induce different activation patterns in
  V1, MT, and MST
• linear motion activates V1, MT, and MST
• speed gradients increase MT and MST activation
• rotation, expansion, and contraction increase
  MST activation
• this is currently being tested by fMRI scanning
  experiments in Magdeburg, Germany

30
Conclusions and Outlook

• the model is well-suited for mobile robots to
  estimate parameters of ego-motion
• the area MST in the simulated hierarchy is very
  sensitive to any translational or rotational
  ego-motion
• in biological vision, MST is massively connected
  to the vestibular system
• in mobile robots, the simulated area MST could
  interact with position and orientation sensors to
  stabilize ego-motion estimation

31
Conclusions and Outlook

• Future work
• lateral interaction across neighboring sets of
  gating units for improved perceptual grouping
• simultaneous simulation of both the motion
  perception and object recognition pathways
• introduction of working memory for an adequate
  internal representation of the current visual
  scene