Backpropagation algorithm for training multi layer perceptrons

1
Backpropagation algorithm for training multi
layer perceptrons

• By Christian Bayer and Stefan Farthofer

2
Multilayer perceptrons

• Several layers
• Adjusting weigths for hidden layers is crucial

3
Backpropagation algorithm

• Generalization of the Delta-rule
• Hidden layers weights are updated according to
  the derivation of the error function by the
  weight in question
• Can use partial result from previous layer to
  compute the next layers derivations
• First invented by Paul J. Werbos in 1974

4
Algorithm overview

• For each record of training data
• Compute net output
• Calculate weight changes beginning with the
  weights from the last hidden layer to the output
  layer
• Apply weight changes
• For batch update calculate all changes and
  accumulate for whole trainig data before applying

5
Maths behind it

• Error function sum of the squares of the
  differences between output and desired output
• The output is a function of the input values, the
  weights of all layers and the transfer function
• Transfer function must be continous

6
Maths behind it cont.

• Example for 3 layer network with 2 neurons per
  layer (On is the n-th ouput, Hn is the n-th
  hidden neurons output and In is the n-th Input
  value)

7
Math behind it cont.
8
Math behind it cont.

• The expression is
  denoted

9
DEMO

• We have implemented the algorithm to train a 2
  layer network to act as a XOR operator. This
  implementation is MS Excel based, so the formulas
  can be examined an all the intermediate result
  are always visible when changing any input.

10
Reference

• Werbos, Paul John, 1974, Beyond regression New
  tools for prediction and analysis in the
  behavioral sciences
• BPN Training algorithm, Daniel Franklin, 2003,
  http//ieee.uow.edu.au/daniel/software/libneural/
  BPN_tutorial/BPN_English/BPN_English/node8.html