NEURAL NETWORKS AND FUZZY SYSTEMS

1
NEURONAL DYNAMICS 2
ACTIVATION MODELS
? ??? ? ?0620110201 ? ?2006-10-09
2
3.1 NEURONAL DYNAMICAL SYSTEM
Neuronal activations change with time. The way
they change depends on the dynamical equations as
following
(3-1) (3-2)
3
3.2 ADDITIVE NEURONAL DYNAMICS

• first-order passive decay model

In the absence of external or neuronal stimuli,
the simplest activation dynamics model is
(3-3) (3-4)
4
3.2 ADDITIVE NEURONAL DYNAMICS
since for any finite initial condition
The membrane potential decays exponentially
quickly to its zero potential.
5
3.2 ADDITIVE NEURONAL DYNAMICS

• Passive Membrane Decay

• Passive-decay rate

scales the rate to
the membranes resting potential.

• solution

Passive-decay rate measures the cell
membranes resistance or friction to current
flow.
6
property

• Pay attention to property

The larger the passive-decay rate,the faster the
decay--the less the resistance to current flow.
7
3.2 ADDITIVE NEURONAL DYNAMICS

• Membrane Time Constants

• The membrane time constant scales the time
  variable of the activation dynamical system.

• The multiplicative constant model

(3-8)
8
Solution and property

• solution

• property

The smaller the capacitance ,the faster things
change
As the membrane capacitance increases toward
positive infinity,membrane fluctuation slows to
stop.
9
3.2 ADDITIVE NEURONAL DYNAMICS

• Membrane Resting Potentials

• Definition

Define resting Potential as the
activation value to which the membrane potential
equilibrates in the absence of external or
neuronal inputs
(3-11)

• Solutions

(3-12)
10
Note

• The time-scaling capacitance dose not affect
  the asymptotic or steady-state solution.

• The steady-state solution does not depend on the
  finite initial condition.

• In resting case,we can find the solution
  quickly.

11
3.2 ADDITIVE NEURONAL DYNAMICS

• Additive External Input

• Add input

Apply a relatively constant numeral input to a
neuron.
(3-13)

• solution

(3-14)
12
Meaning of the input

• Input can represent the magnitude of directly
• experiment sensory information or directly apply
  control information.

• The input changes slowly,and can be assumed
• constant value.

13
3.3 ADDITIVE NEURONAL FEEDBACK

• Neurons do not compute alone. Neuron modify
  their state activations with external input and
  with the feedback from one another.

• This feedback takes the form of path-weighted
  signals from synaptically connected neurons.

14
3.3 ADDITIVE NEURONAL FEEDBACK

• Synaptic Connection Matrices

• n neurons in field p neurons in field

The ith neuron axon in a synapse
jth neurons in
is constant,can be positive,negative or zero.
15
Meaning of connection matrix

• The synaptic matrix or connection matrix M is
  an
• n-by-p matrix of real number whose entries are
  the
• synaptic efficacies .the ijth synapse is
  excitatory
• if inhibitory if

• The matrix M describes the forward projections
  from
• neuron field to neuron field

• The matrix N describes the feedforward
  projections
• from neuron field to neuron field

• The neural network can be specified by the
  4-tuple
• (M, N, , )

16
3.3 ADDITIVE NEURONAL FEEDBACK

• Bidirectional and Unidirectional connection
  Topologies

• Bidirectional networks

M and N have the same or approximately the
same structure.

• Unidirectional network

A neuron field synaptically intraconnects to
itself.

• BAM Bidirectional associative memories

M is symmetric, the
unidirectional network is BAM
17
Augmented field and augmented matrix

• Augmented field

M connects to ,N connects to
then the augmented field intraconnects to
itself by the square block matrix B
18
Augmented field and augmented matrix

• In the BAM case,when then
  hence a BAM symmetries an arbitrary
  rectangular matrix M.

• In the general case,

P is n-by-n matrix. Q is p-by-p matrix.
If and only if,
the neurons in are symmetrically
intraconnected
19
3.4 ADDITIVE ACTIVATION MODELS

• Define additive activation model
• np coupled first-order differential equations
  defines the additive activation model

(3-15) (3-16)
20
additive activation model define

• The additive autoassociative model correspond to
  a system of n coupled first-order differential
  equations

(3-17)
21
additive activation model define

• A special case of the additive autoassociative
  model

(3-18) (3-19)
(3-20)
where
is
measures the cytoplasmic resistance
between neurons i and j.
22
Hopfield circuit and continuous additive BAM

• Hopfield circuit arises from if each neuron has
  a strictly increasing signal function and if the
  synaptic connection matrix is symmetric

(3-21)

• continuous additive bidirectional associative
  memories

(3-22)
(3-23)
23
3.5 ADDITIVE BIVALENT MODELS

• Discrete additive activation models correspond
  to neurons with threshold signal function

• The neurons can assume only two value ON and
  OFF.
• ON represents the signal value 1.
• OFF represents 0 or 1.

• Bivalent models can represent asynchronous and
  stochastic behavior.

24
Bivalent Additive BAM

• BAM-bidirectional associative memory

• Define a discrete additive BAM with threshold
  signal functions, arbitrary thresholds and
  inputs,an arbitrary but constant synaptic
  connection matrix M,and discrete time steps k.

(3-24)
(3-25)
25
Bivalent Additive BAM

• Threshold binary signal functions

(3-26)
(3-27)

• For arbitrary real-value thresholds
• for neurons
  for neurons

26
A example for BAM model

• Example
• A 4-by-3 matrix M represents the forward synaptic
  projections from to .
• A 3-by-4 matrix MT represents the backward
  synaptic projections from to .

27
A example for BAM model

• Suppose at initial time k all the neurons in
  are ON.
• So the signal state vector at time k
  corresponds to

• Input

• Suppose

28
A example for BAM model

• firstat time k1 through synchronous
  operation,the result is

• nextat time k1 ,these signals pass
  forward through the filter M to affect the
  activations of the neurons.

• The three neurons compute three dot products,or
  correlations.The signal state vector
  multiplies each of the three columns of M.

29
A example for BAM model

• the result is

• synchronously compute the new signal state
  vector

30
A example for BAM model

• the signal vector passes backward through the
  synaptic
• filter at time k2

• synchronously compute the new signal state
  vector

31
A example for BAM model
since
then

• conclusion

These same two signal state vectors will pass
back and forth in bidirectional equilibrium
forever-or until new inputs perturb the system
out of equilibrium.
32
A example for BAM model

• asynchronous state changes may lead to
  different bidirectional equilibrium

• keep the first neurons ON,only update
  the second and third neurons. At k,all
  neurons are ON.

• new signal state vector at time k1 equals

33
A example for BAM model

• new activation state vector equals

• synchronously thresholds

• passing this vector forward to gives

34
A example for BAM model

• Similarly,
• for any asynchronous state change policy we apply
  to the neurons

• The system has reached a new equilibrium,the
  binary pair
  represents a fixed point of the system.

35
Conclusion

• Different subset asynchronous state change
  policies applied to the same data need not
  product the same fixed-point equilibrium. They
  tend to produce the same equilibria.
• All BAM state changes lead to fixed-point
  stability.

36
Bidirectional Stability

• Definition
• A BAM system is
  Bidirectional stable if all inputs converge to
  fixed-point equilibria.
• A denotes a binary n-vector in
• B denotes a binary p-vector in

37
Bidirectional Stability

• Represent a BAM system equilibrates to
  bidirectional fixed
• point as

38
Lyapunov Functions

• Lyapunov Functions L maps system state
  variables to real
• numbers and decreases with time. In BAM case,L
  maps the
• Bivalent product space to real numbers.

• Suppose L is sufficiently differentiable to
  apply the chain
• rule

(3-28)
39
Lyapunov Functions

• The quadratic choice of L

(3-29)

• Suppose the dynamical system describes the
  passive decay system.

(3-30)

• The solution

(3-31)
40
Lyapunov Functions

• The partial derivative of the quadratic L

(3-32)
(3-33)
(3-34)
or
(3-35)
In either case
(3-36)
At equilibrium
This occurs if and only if all velocities equal
zero
41
Conclusion

• A dynamical system is stable if some Lyapunov
  Functions L decreases along trajectories.
• A dynamical system is asymptotically stable if
  it strictly decreases along trajectories
• Monotonicity of a Lyapunov Function provides a
  sufficient condition for stability and asymptotic
  stability.

42
Linear system stability
For symmetric matrix A and square matrix B,the
quadratic form behaves as a
strictly decreasing Lyapunov
function for any linear dynamical system
if and only if the matrix
is negative definite.
43
The relations between convergence rate and
eigenvalue sign

• A general theorem in dynamical system theory
  relates convergence rate and ergenvalue sign
• A nonlinear dynamical system converges
• exponetially quickly if its system Jacobian
  
• has eigenvalues with negative real parts.
• Locally such nonlinear system behave as linearly.

44
Bivalent BAM theorem

• The average signal energy L of the forward pass
  of the
• Signal state vector through M,and the
  backward pass
• Of the signal state vector through
  

(3-46)
since
(3-47)
(3-48)
45
Lower bound of Lyapunov function

• The signal is Lyapunov function clearly bounded
  below.
• For binary or bipolar,the matrix coefficients
  define the
• attainable bound

• The attainable upper bound is the negative of
  this expression.

46
Lyapunov function for the general BAM system

• The signal-energy Lyapunov function for the
  general BAM
• system takes the form

Inputs and
and constant vectors of
thresholds the attainable bound of this function
is.
47
Bivalent BAM theorem

• Bivalent BAM theorem.every matrix is
  bidrectionally stable
• for synchronous or asynchronous state changes.

• Proof consider the signal state changes that
  occur from time k to time k1,define the vectors
  of signal state changes as

48
Bivalent BAM theorem

• define the individual state changes as

• We assume at least one neuron changes state from
  k to time k1.
• Any subset of neurons in a field can change
  state,but in only one field at a time.
• For binary threshold signal functions if a state
  change is nonzero,

49
Bivalent BAM theorem
For bipolar threshold signal functions
The energychange
Differs from zero because of changes in field
or in field
50
Bivalent BAM theorem
51
Bivalent BAM theorem
Suppose
Then
This implies so the product is
positive

Another case suppose
52
Bivalent BAM theorem

• This implies so the product is
  positive
  

So for every state change.

• Since L is bounded,L behaves as a Lyapunov
  function for
• the additive BAM dynamical system defined by
  before.
• Since the matrix M was arbitrary,every matrix is
  bidirectionally stable. The bivalent Bam theorem
  is proved.

53
Property of globally stable dynamical system
54
Two insights about the rate of convergence

• First,the individual energies decrease
  nontrivially.the BAM system does not creep
  arbitrary slowly down the toward the nearest
  local minimum.the system takes definite hops into
  the basin of attraction of the fixed point.

• Second,a synchronous BAM tends to converge
  faster than an asynchronous BAM.In another word,
  asynchronous updating should take more iterations
  to converge.