Information Processing with a Number of Neurons

1
Information Processing with a Number of Neurons
2
Neural Representation of moving direction

• Two pieces of information are important for
  representing the dynamical aspects of external
  objects, namely, the moving speed and the moving
  direction of objects.

• A. Georgopouloss group carried out a series
  elegant
• experiments to explore the neural representation
  of
• moving direction.

• It is one of the few examples in which the coding
  
• scheme is relatively well understood.

3
Experiment observations

• In the experiment, the monkey is guided to move
  the lever in the
• center of apparatus to one of eight peripheral
  locations.

• Neural activities in the motor area are recorded.

All pictures from M. Gazzaniga et al. Cognitive
Neuroscience, unless otherwise stated.
4
Preferred stimulus and tuning curve

• Preferred stimulus for each neuron, there is a
  stimulus value by
• which the neuron
  has the maximum response.
• Tuning curve the function mapping between the
  neural activity
• (measured by mean firing
  rate) and the stimulus value.

• Noises always exist

5
Why population code?

• How is the moving direction encoded by neural
  activities?
• By the most active neuron? This sounds reasonably
  if there is
• no noise, but it does not work in practice
  because of
• large fluctuations in neural activities.
• By a population of neurons?
• All active neurons contain a piece of
  information
• about the stimulus, why dont we consider all
  of them
• jointly encode the moving direction. It has at
  least one apparent
• advantage of averaging out noises in individual
  neurons,
• since they are (partially) independent.
• Georgopoulos et al. proposed an idea to
  reconstruct the moving
• direction from the observed neural activities.

6
Mathematical Modeling of Neuronal Response

• The smooth bell-shape tuning curve is often
  modeled by the
• Gaussian (or cosin) function

• For neural activity in one trial

7
Population Vector

• Georgopouloss idea the neural system reads out
  the moving
• direction by the average of preferred stimuli
  of all
• active neurons weighted by their activities.
• This sounds reasonable since more active
  neurons, whose preferred
• stimuli are more likely close to the true
  stimulus, and hence
• should contribute more on the final vote.

8
An illustration of Population Vector
9
The paradigm of population coding

• Population vector demonstrated that information
  can be accurately
• represented by the joint activities of a
  population of neurons in a noise
• environment. This coding strategy of using a
  population of neurons
• to represent stimulus is called population
  coding.

• The idea of population coding is also found in
  the representation of
• moving direction in other parts of cortex, and
  the representation of
• other stimuli, such as the orientation of
  object and the spatial location.

• Population coding seems to be a general framework
  for information
• processing in neural systems, and worth to be
  analyzed in more
• detail theoretically.

10
The mathematical model of population coding

• The encoding phase

• The decoding phase

Population vector is one of many inference
strategies.
11
Some interesting research issues

• How much information is encoded in a population
  code? Because of noise,
• finite number of neurons, and the decoding
  strategy used, the inferred result is in general
  different from the real stimulus value (the
  error). A theoretic question is what is the
  minimum error for any decoding strategy to
  achieve.

• What is the most efficient decoding strategy?
  i.e., the one that has the minimum decoding
  errormaximum likelihood inference.

• What is the effect of noise correlation? When
  noises at different neurons are independent, they
  can be perfectly clean out through population
  average by a generic decoding method. However, in
  real biologic system noise correlation is
  inevitable, then a question is how much does the
  correlation affect the decoding performance.

• What is the biologically plausible decoding
  strategy? For a strategy to be
• used in biological systems, we would expect that
  this method is reasonably
• efficient and can also be easily implemented by
  the neural networks
• architecture.

12
Template Matching

• In the noise-less case, the population activity
  profile (mean firing rates of all neurons) is
  given by the tuning function f(x). In the noisy
  case, we may use f(x) as a template to match the
  observed noisy population activity, and choose
  the solution as the one when the template has the
  maximum overlap with the noisy hill. The peak
  position of template then reports the estimated
  result.

• The noisy hill is the population activity
• when the stimulus x0.
• Among three positions, the red one ( )
• has the maximum overlap with the
• observed data.

• Mathematically, this is equivalent to

13
The performance of template-matching (1)

• Maximum likelihood inference (MLI)

MLI is asymptotically efficient in many cases.
This means that when the number of neurons is
sufficiently large, its decoding error reaches
the lower bound, i.e., the inverse of the Fisher
information, called the Cramer-Rao bound.
An illustration of asymptotical efficiency. The
decoding error is measured by
14
The performance of template-matching (2)

• MLI for independent Gaussian noise

Template-matching
15
The performance of template-matching (3)

• For independent Gaussian noise, template-matching
  is
• equivalent to MLI, and hence is asymptotically
  efficient.
• For correlated noise which is always the case in
  reality,
• template-matching can be regarded as a decoding
  method
• that ignore the neuronal correlation, but
  utilize
• the knowledge of tuning function whereas,
  population vector
• can thought as a method which even ignore the
  tuning
• function (only the knowledge of neurons
  preferred stimuli
• is used). We expect that template-matching
  outperforms
• population vector in general cases.

16
Comparing the performances of three methods
In terms of decoding error Population
vectorgtTemplate-matchinggtMLI
17
On the biological plausibility of a decoding
strategy

• For a decoding strategy to be used in neural
  systems, accuracy is
• only one aspect, simplicity, speed and whether
  it can be implemented
• in neural architecture are also important.
• MLI, though very accurate, is often too
  complicated to be
• implemented in neural architecture, especially,
  when noises are
• correlated.
• Population vector, may appears to be simple to
  computers (just
• some addition and times operations), is not
  guaranteed to be also
• simple in the view of neural systems (e.g., how
  to carry out these
• additions and times is not obvious). Moreover,
  in some noise
• correlation structures, population vector can be
  very inefficient.
• How about template-matching? In the below we
  will show that it
• can be naturally achieved in neural systems
  through the idea
• of continuous attractor.

18
Attractor Computation

• Attractor a steady state of a neural ensemble
  memorizes
• a stimulus value.

• Information retrieval (associative memory) a
  noisy input will
• be attracted to a steady state of the system
  that memorizing the
• prototype of the input.

• Discrete vs. Continuous attractor

Point attractor the steady state is only the
bottom of the bowl.
Line attractor the system is stable on the whole
valley.
19
The properties of continuous attractor

• The steady states of the system (the memorized
  stimulus values)
• forms a continuous space, on which the system
  is neutrally stable.

• This neutrally stability allows the system to
  change status smoothly,
• following a fixed path.
• This property (not shared by discrete
  attractors) is crucial for the
• system to seamlessly track the smooth change of
  stimulus.

• One drawback of continuous attractor may be that
  the neutrally
• stability implies the system is sensitive to
  fluctuations along the
• attractor space. This, however, can be overcome
  through on-line
• coding.

• Continuous attractor seems to be most suitable
  for representing
• continuous stimulus such as the moving
  direction, but may also
• works well for encoding discrete objects if
  there is a continuous
• underlying feature linking all these objects.

20
The monkey experiment revisit

• In the experiment, the monkey to required to
  compute the moving direction mentally (mental
  rotation).
• We observe that the representation of moving
  direction in the system can smoothly change
  states and follows a fixed path, indicating a
  structure of continuous attractor.

The monkey moved the lever from a
initial direction to a target direction guided by
signal. Data was recorded in the motor
area. Population vector was used to
reconstruct the intermediate (mental) directions
every 10 ms.
21
Neural implementation of template-matching
Three essential elements on the network
architecture

• One The steady states of the network must have
  the same shape of
• tuning function in order to generate
  the template.
• Two When no stimulus exists, the network should
  be neutrally stable
• on a line attractor (consider 1D
  stimulus), parameterized by all
• possible stimulus values. This enables
  the network to be ready
• to decode (match) any stimulus value
  that may arise.
• Three An external input that contains the
  stimulus information drives
• the template (the steady state of the
  system) to the position
• that has the maximum overlap with the
  noisy population
• activity (this is the final execution
  of template-matching).

22
A simple network model for line attractor

• The network structure (rate model)

The symmetry structure is the key. a the tuning
width.
23
The network estimation

• When no external input presents, the steady state
  has the form
• (the template)

• When a small or transient external input is
  presented,

the network will be stable at one point on the
line attractor, with the position determined by
(template-matching)
24
Matlab demo for continuous attractor
Demonstrating the smooth tracking capability of
continuous attractor
25
Matlab demo for template-matching
26
Project 1 Modelling Continuous Attractors

• Aim to study population coding and recurrent
  network models
• to investigate that continuous stimulus is best
  to be represented as
• continuous attractor in neural systems.

• Requirement to build up a  recurrent  network
  model with simplified
• rate coding units. Using this model to illustrate
  that when the recurrent
• interactions have proper symmetrical structure,
  the network is neutrally
•  stable on a continuous attractor with the steady
  state being of the
• bell-shape. Elucidate how continuous attractor
  can be used to
• seamlessly track stimulus change and perform
  template-matching
• for noisy input.

27
Main References
1. Ben-Yishai et al. Theory of orientation tuning
in visual cortex. Proceedings of the National
Academy of Sciences of the United States of
America, 923844-3848, 1995
2. Zhang K. Representation of spatial orientation
by the intrinsic dynamics of the head-direction
cell a theory. Journal of Neuroscience,
162112-2126.
3. Pouget et al. Statistically efficient
estimation using population coding. Neural
Computation, 10373-401, 1998.
4. Wu S. et al. Sequential Bayesian decoding with
a number of neurons. Neural Computation,
15993-1013, 2003. 5. Wu S. et al. Computation
with continuous attractors Stability and
on-line aspects. Neural Computation (in press).