Introduction to Neural Networks

1
Introduction to Neural Networks

• Freek Stulp

2
Overview

• Biological Background
• Artificial Neuron
• Classes of Neural Networks
• Perceptrons
• Multi-Layered Feed-Forward Networks
• Recurrent Networks
• Conclusion

3
Biological Background

• Neuron consists of
• Cell body
• Dendrites
• Axon
• Synapses

• Neural activation
• Throught dendrites/axon
• Synapses have different strengths

4
Artificial Neuron
Input links (dendrites)
Unit (cell body)
Output links (axon)
aj
Wji
ai
5
Class I Perceptron
6
Learning in Perceptrons

• Perceptrons can learn mappings from inputs I to
  outputs O by changing weights W
• Training set D
• Inputs I0, I1 ... In
• Targets T0, T1 ...Tn

• Example boolean ORD
• Output O of network is not necessary equal to T!

7
Learning in Perceptrons

• Error often defined as E(W)
  1/2Sd?D(td-od)2
• Go towards the minimum error!
• Update rules
• wi wi Dwi
• Dwi -hdE/dwi
• dE/dwi d/dwi 1/2Sd?D(td-od)2
  Sd?D(td-od)iid
• This is called gradient descent

i
8
Class II Multi-layer Feed-forward Networks

• Feed-forward
• Output links only connected to input links in the
  next layer

• Complex non-linear functions can be represented

9
Learning in MLFF Networks

• For output layer, weight updating similar to
  perceptrons.
• Problem What are the errors in the hidden layer?
• Backpropagation Algorithm
• For each hidden layer (from output to input)
• For each unit in the layer determine how much it
  contributed to the errors in the previous layer.
• Adapt the weight according to this contribution
• This is also gradient descent

10
Class III Recurrent Networks

• No restrictions on connections
• Behaviour more difficult to predict/ understand

11
Conclusion

• Inspiration from biology, though artificial
  brains are still very far away.
• Perceptrons too simple for most problems.
• MLFF Networks good as function approximators.
• Many of your articles use these networks!
• Recurrent networks complex but useful too.

12
Literature

• Artificial Intelligence A Modern Approach
• Stuart Russel and Peter Norvig
• Machine Learning
• Tom M. Mitchell