Outline

1
Outline

• Competitive Learning continued
• Hamming network - review
• Self-organizing feature maps
• Learning vector quantization

2
Hamming Network
3
Hamming Network cont.

• Layer 1
• Consists of multiple instar neurons to recognize
  more than one pattern
• The output of a neuron is the inner production
  between the weight vector (prototype) and the
  input vector
• The output from the first layer indicates the
  correlation between the prototype pattern and the
  input vector
• It is feedforward

4
Layer 1 (Correlation)
We want the network to recognize the following
prototype vectors
The first layer weight matrix and bias vector are
given by
The response of the first layer is
The prototype closest to the input vector
produces the largest response.
5
Hamming Network cont.

• Layer 2
• It is a recurrent network, called a competitive
  network
• The neurons in this layer compete with each other
  to determine a winner
• After competition, only one neuron will have a
  nonzero output
• The winning neuron indicates which category of
  input was presented to the network

6
Layer 2 (Competition)
The second layer is initialized with the
output of the first layer.
The neuron with the largest initial condition
will win the competition.
7
Hamming Network cont.

• Lateral inhibition
• This competition is called a winner-take-all
  competition
• Because the one with the largest value decreases
  the slowest, it remains positive when all others
  become zero
• What will happen if there are ties?

8
Classification Example
9
Competitive Layer

• In a competitive layer, each neuron excites
  itself and inhibits all the other neurons
• A transfer function that does the job of a
  recurrent competitive layer
• It works by finding the neuron with the largest
  net input and setting its output to 1 (In case of
  ties, the neuron with lowest index). All other
  outputs are set to 0

10
Competitive Layer
11
Competitive Learning

• A learning rule to train the weights in a
  competitive network
• Instar rule
• In other words,
• For the competitive network, the winning neuron
  has an output of 1, and the other neurons have an
  output of 0.

12
Competitive Learning
Kohonen Rule
13
Graphical Representation
14
Example
15
Four Iterations
16
Problems with Competitive Layers

• Choice of learning rate
• A learning rate near zero results in slow
  learning but stable
• A learning rate near one results in fast learning
  but oscillate
• Stability problem when clusters are close to each
  other
• Dead neuron
• A neuron whose initial weight vector is so far
  from any input vectors that it never wins the
  competition
• The number of classes must be known
• These limitations can be overcome by the feature
  maps, LVQ networks, and ART networks

17
Choice of Learning Rates

• When learning rate is small, the learning is
  stable but slow
• When learning rate is close to 1, the learning is
  fast but slow
• Adaptive learning rate can be used
• Initial learning rate is large and gradually
  decrease the learning rate

18
Stability
If the input vectors dont fall into nice
clusters, then for large learning rates the
presentation of each input vector may modify the
configuration so that the system will undergo
continual evolution.
p3
p3
p1
p1
p5
p5
1w(0)
1w(8)
p8
p8
p7
p7
2w(8)
2w(0)
p6
p6
p2
p2
p4
p4
19
Another Stability Example
20
Typical Convergence (Clustering)
Weights
Input Vectors
Before Training
After Training
21
Dead Units
One problem with competitive learning is that
neurons with initial weights far from any input
vector may never win.
22
Dead Units cont.

• Solution
• Add a negative bias to each neuron, and increase
  the magnitude of the bias as the neuron wins
• This will make it harder to win if a neuron has
  won often
• This is called a conscience

23
Competitive Layers in Biology
On-Center/Off-Surround Connections for Competition
Weights in the competitive layer of the Hamming
network
Weights assigned based on distance
24
Mexican-Hat Function
25
Feature Maps
Update weight vectors in a neighborhood of the
winning neuron.
26
Example
27
Self-Organizing Feature Maps cont.
28
Self-Organizing Feature Maps cont.
29
Self-Organizing Feature Maps cont.
30
Self-Organizing Feature Maps cont.
31
Self-Organizing Feature Maps cont.
32
Improving SOFM

• Convergence speed-up of SOFM
• Variable neighborhood size
• Use a larger neighborhood size initially and
  gradually reduce it until it includes only the
  winning neuron
• Variable learning rate
• Use a larger learning rate initially (close to 1)
  and decrease it toward 0 asymptotically
• Let the winning neuron use a larger rate than the
  neighboring ones
• One can use distance as the net input instead of
  the inner product

33
Learning Vector Quantization
The net input is not computed by taking an inner
product of the prototype vectors with the input.
Instead, the net input is the negative of the
distance between the prototype vectors and the
input.
34
Subclass
For the LVQ network, the winning neuron in the
first layer indicates the subclass which the
input vector belongs to. There may be several
different neurons (subclasses) which make up each
class.
The second layer of the LVQ network combines
subclasses into a single class. The columns of W2
represent subclasses, and the rows represent
classes. W2 has a single 1 in each column,
with the other elements set to zero. The row in
which the 1 occurs indicates which class the
appropriate subclass belongs to.
35
Example
Subclasses 1, 3 and 4 belong to class
1. Subclass 2 belongs to class
2. Subclasses 5 and 6 belong to class 3.
A single-layer competitive network can create
convex classification regions. The second layer
of the LVQ network can combine the convex regions
to create more complex categories.
36
LVQ Design Example
37
LVQ Design Example
38
LVQ Design Example
39
LVQ Learning
LVQ learning combines competitive learning with
supervision. It requires a training set of
examples of proper network behavior.
If the input pattern is classified correctly,
then move the winning weight toward the input
vector according to the Kohonen rule.
If the input pattern is classified incorrectly,
then move the winning weight away from the input
vector.
40
Example
41
First Iteration
42
Second Layer
This is the correct class, therefore the weight
vector is moved toward the input vector.
43
Figure
44
Final Decision Regions
45
LVQ2
If the winning neuron in the hidden layer
incorrectly classifies the current input, we move
its weight vector away from the input vector, as
before. However, we also adjust the weights of
the closest neuron to the input vector that does
classify it properly. The weights for this second
neuron should be moved toward the input
vector. When the network correctly classifies an
input vector, the weights of only one neuron are
moved toward the input vector. However, if the
input vector is incorrectly classified, the
weights of two neurons are updated, one weight
vector is moved away from the input vector, and
the other one is moved toward the input vector.
The resulting algorithm is called LVQ2.
46
LVQ2 Example