The Time Dimension for Scene Analysis

1
The Time Dimension for Scene Analysis

• DeLiang Wang
• Perception Neurodynamics Lab
• The Ohio State University, USA

2
Presentation outline

• Introduction
• Scene analysis and temporal correlation theory
• Oscillatory Correlation
• LEGION network
• Oscillatory Correlation Approach to Scene
  Analysis
• Image segmentation
• Object selection
• Cocktail party problem
• Concluding remarks

3
Scene analysis problem
4
Binding problem

• Feature binding (integration) is a fundamental
  problem in neuroscience and perception (and
  perceptrons)

Binding problem in Rosenblatts perceptrons
5
Temporal correlation theory

• Temporal correlation theory proposes a solution
  to the nervous integration problem (von der
  Malsburg81 also Milnor74)
• Application to cocktail party processing (von der
  Malsburg Schneider86)

6
Physiological evidence (Gray et al.89)
7
Oscillatory correlation theory

• Oscillators represent feature detectors
• Binding is encoded by synchrony within an
  oscillator assembly and desynchrony between
  different assemblies

8
Computational requirements

• Need to synchronize locally coupled oscillator
  population
• Need to desynchronize different populations, when
  facing multiple objects
• Synchrony and desynchrony
  must be achieved rapidly

9
LEGION architecture

• LEGION - Locally Excitatory Globally Inhibitory
  Oscillator Network (Terman Wang95)

10
Relaxation oscillator as building block

• With stimulus

Without stimulus
Typical x trace (membrane potential)
11
Analytical results

• Theorem 1. (Synchronization). The oscillators in
  a connected block synchronize at an exponential
  rate
• Theorem 2. (Multiple patterns) If at the
  beginning all the oscillators of the same block
  synchronize and different blocks desynchronize,
  then synchrony within each block and the ordering
  of activations among different blocks are
  maintained
• Theorem 3. (Desynchronization) If at the
  beginning all the oscillators of the system lie
  not too far away from each other, then the
  condition of Theorem 2 will be satisfied after
  some time. Moreover, the time it takes to
  satisfy the condition is no greater than N
  cycles, where N is the number of blocks

12
Connectedness problem

• Minsky-Papert connectedness problem is a
  long-standing problem in perceptron learning
• The problem exposes fundamental limitations of
  supervised learning, and illustrates the
  importance of proper representations

13
Connectedness problem LEGION solution

• Basic idea Synchronization within a connected
  pattern and desynchronization between different
  ones

14
Presentation outline

• Introduction
• Scene analysis and temporal correlation theory
• Oscillatory Correlation
• LEGION network
• Oscillatory Correlation Approach to Scene
  Analysis
• Image segmentation
• Object selection
• Cocktail party problem
• Concluding remarks

15
Oscillatory correlation approach to scene
segmentation

• Feature extraction first takes place
• An visual feature can be pixel intensity, depth,
  local image patch, texture element, optic flow,
  etc.
• An auditory feature can be a pure tone, amplitude
  and frequency modulation, onset, harmonicity,
  etc.
• Connection weights between neighboring
  oscillators are set to be proportional to feature
  similarity
• Global inhibitor controls granularity of
  segmentation
• Larger inhibition results in more and smaller
  regions
• Segments pop out from LEGION in time

16
Image segmentation example Demo
Input image
17
Image segmentation example
Input image
Segmentation result
18
Object selection

• The slow inhibitor keeps trace of each pattern,
  which can be overcome by only more salient
  (larger) patterns
• Unlike traditional winner-take-all dynamics,
  selection (competition) takes place at the object
  level
• Consistent with object-based attention theory
• Binding precedes attention, rather than attention
  precedes binding (Treisman Gelade80)

19
Results of object selection
LEGION output
Selection output
Input image
Input LEGION segmentation
Selection
20
Cocktail party problem

• In a natural environment, target speech is
  usually corrupted by acoustic interference,
  creating a speech segregation problem
• Popularly known as cocktail-party problem
  (Cherry53) also ball-room problem (Helmholtz,
  1863)
• Human listeners organize sound in a perceptual
  process called auditory scene analysis
  (Bregman90)
• Auditory scene analysis (ASA) takes place in two
  conceptual stages
• Segmentation. Decompose the acoustic signal into
  sensory elements (segments)
• Grouping. Combine segments into groups, so that
  segments in the same group likely originate from
  the same sound source

21
Oscillatory correlation for ASA (Wang Brown99)
Frequency
22
Auditory periphery Cochleagram

• Cochleagram representation of the utterance Why
  were you all weary? mixed with phone ringing

23
Grouping layer Example

• Two streams emerge from the group layer
• Foreground left (original mixture
  )
• Background right
• More recent results (Hu Wang04)

24
Back to physiology

• Chattering cells recorded by Gray McCormick96
• Burst oscillations are best modeled by relaxation
  oscillators

25
Versatility and time dimension

• The principle of universality Give me a
  concrete problem and I will devise a network that
  solves it. (von der Malsburg99)
• It characterizes artificial intelligence
• The principle of versatility Given the network,
  learn to cope with situations and problems as
  they arise. (von der Malsburg99)
• It characterizes natural intelligence
• Time dimension is necessary for versatility
• Flexible and infinitely extensible
• Irreplaceable by spatial organization

26
Conclusion

• Advances in dynamical analysis overcome
  computational obstacles of oscillatory
  correlation theory
• Major progress is made towards solving the scene
  analysis problem
• From Hebbs cell assemblies to von der Malsburgs
  correlation theory, time is an indispensable
  dimension for scene analysis