Neural%20Networks

1
Neural Networks

• Joost N. Kok
• Universiteit Leiden

2
Neural Networks

• Book Introduction to the theory of Neural
  Computation by Hertz, Krogh, Palmer
• Website www.liacs.nl/home/joost/nn08.htm
• Additional Material Powerpoint Sheets, Journal
  Articles, practical exercises
• Other recommended books Haykin, Bishop

3
The Von Neumann architecture
4
The Hungarian-born mathematician, John von
Neumann (1903-1957)
5
The biological architecture
6
Biological computers

• Five distinguishing properties
• Highly parallel
• Robust and fault tolerant
• Adaptive
• Deals with fuzzy, noisy information
• Small, compact

7
Graceful Degradation
performance
damage
8
Neurons
9
Brain consists of 100000000000 (1011) neurons
10
Neural activity
out
in
11
Artificial Neuron
12
Input-output function

• nonlinear function

a ? 0
f(e)
a ? ?
e
13
Artificial Connections (Synapses)

• wAB
• The weight of the connection from neuron A to
  neuron B

14
Supervised Networks
15
Example
16
Real Neurons

• Nonlinear Summation
• Sequences of pulses
• No fixed time-delay

17
History

• 1943 McCulloch and Pitts artificial neuron
• 1960 Rosenblatt perceptrons
• 1969 Minsky and Papert XOR problem
• 197? Associative content-addressable memory
• 198? Hopfield Networks, Boltzmann Networks
• 1985 Backpropagation learning rule
• 2000 Spiking networks, Support Vector Machines

18
Hopfield
McCulloch
Pitts
Papert
Minsky
Boltzmann
Hebb
19
(No Transcript)
20
Issues

• Neurocomputing vs. Neuroscience
• Types of Learning
• Supervised
• Unsupervised
• Reinforcement

21
Research Questions

• Design what is best architecture?
• Learning find good algorithms
• Analysis what is the power of networks?
• Implementation how should the network be
  implemented?