Neural Networks Chapter 2

1
Neural NetworksChapter 2

• Joost N. Kok
• Universiteit Leiden

2
Hopfield Networks

• Network of McCulloch-Pitts neurons
• Output is 1 iff and is -1
  otherwise

3
Hopfield Networks
4
Hopfield Networks
5
Hopfield Networks
6
Hopfield Networks

• Associative Memory ProblemStore a set of
  patterns in such a way that when presented with a
  new pattern, the network responds by producing
  whichever of the stored patterns most closely
  resembles the new pattern.

7
Hopfield Networks

• Resembles Hamming distance
• Configuration space all possible states of the
  network
• Stored patterns should be attractors
• Basins of attractors

8
Hopfield Networks

• N neurons
• Two states -1 (silent) and 1 (firing)
• Fully connected
• Symmetric Weights
• Thresholds

9
Hopfield Networks

w13
w16
w57
-1
1
10
Hopfield Networks

• State
• Weights
• Dynamics

11
Hopfield Networks

• Hebbs learning rule
• Make connection stronger if neurons have the same
  state
• Make connection weaker if the neurons have a
  different state

12
Hopfield Networks
neuron 1
synapse
neuron 2
13
Hopfield Networks

• Weight between neuron i and neuron j is given by

14
Hopfield Networks

• Opposite patterns give the same weights
• This implies that they are also stable points of
  the network
• Capacity of Hopfield Networks is limited 0.14 N

15
Hopfield Networks

• Hopfield defines the energy of a network
• E - ½ ?ij SiSjwij ? i Siqi
• If we pick unit i and the firing rule does not
  change its Si, it will not change E.
• If we pick unit i and the firing rule does change
  its Si, it will decrease E.

16
(No Transcript)
17
Hopfield Networks

• Energy function
• Alternative Form
• Updates

18
Hopfield Networks
19
Hopfield Networks

• Extension use stochastic fire rule
• Si 1 with probability g(hi)
• Si -1 with probability 1-g(hi)

20
Hopfield Networks

• Nonlinear function

b ? ?
g(x)
b ? 0
x
21
Hopfield Networks

• Replace the binary threshold units by binary
  stochastic units.
• Define b 1/T
• Use temperature T to make it easier to cross
  energy barriers.
• Start at high temperature where its easy to cross
  energy barriers.
• Reduce slowly to low temperature where good
  states are much more probable than bad ones.

A B C
22
Hopfield Networks

• Kick the network our of spurious local minima
• Equilibrium becomes time independent