Lecture%202:%20Basics%20and%20definitions

1
Introduction to Neural Networks Andy
Philippides Centre for Computational
Neuroscience and Robotics (CCNR) School of
Cognitive and Computing Sciences/School of
Biological Sciences andrewop_at_cogs.susx.ac.uk Spri
ng 2003
2
Lectures -- 2 per week Time
Day Place 1230 - 120
Mon Arun - 401 1130 - 1220
Wed Arun - 401 Seminar 1 per
week Group 1 3 3.50 Mon Pev1
2D4 Group 2 4 4.50 Mon Pev1
2D4 Group 3 2 2.50 Fri Arun
404B Group 4 3 3.50 Fri Arun
404B Office hour Friday12.30-1.30, BIOLS room
3D10 Lecture will be available online soon
3

• Todays Topics
• Course summary
• Components of an artificial neural network
• A little bit math
• Single artificial neuron

4
Course Summary
Course Summary
The course will introduce the theory of several
variants of artificial neural networks (ANNs)
discuss how they are used/trained in
practice Ideas will be illustrated using the
example of ANNs used for function
approximation Very common use of ANNs and also
shows the major concepts nicely. Idea
Data
Post-Processing
Pre-Processing
Neural Net model training method
Function approx
Will not specifically be using NNs as brain
models (Computational Neuroscience)
5
Topics covered 1. Introduction to neural
networks 2. Basic concepts for network
training 3. Single layer perceptron 4.
Probability density estimation 56. Multilayer
perceptron 78. Radial Basis Function
networks 910. Support Vector machines 1112.
Pre-processing Competitve Learning 1314.
Mixtures of Experts/Committee machines 1516.
Neural networks for robot control
6
Assessment

• 3rd years All coursework
• Masters students 50 coursework, 50 exam
  (start of next term)
• Coursework is 2 programming projects first is 20
  of coursework (details next week) due in week 6,
  second 80 due week 10.
• Coursework dealt with in seminars, some
  theoretical, some practical matlab sessions
  (programs can be in any language, but matlab is
  useful for in-built functions)
• This weeks seminar light maths revision

7
Course Texts

• 1. Haykin S (1999). Neural networks. Prentice
  Hall International. Excellent but quite heavily
  mathematical
• 2. Bishop C (1995). Neural networks for pattern
  recognition. Oxford Clarendon Press (good but a
  bit statistical, not enough dynamical theory)
• 3. Pattern Classification, John Wiley, 2001R.O.
  Duda and P.E. Hart and D.G. Stork
• 4. Hertz J., Krogh A., and Palmer R.G.
  Introduction to the theory of neural computation
  (nice, but somewhat out of date)

8

• 5. Pattern Recognition and Neural Networksby
  Brian D. Ripley. Cambridge University Press. Jan
  1996. ISBN 0 521 46086 7.
• 6. Neural Networks. An Introduction,
  Springer-Verlag Berlin, 1991 B. Mueller and J.
  Reinhardt
• As its quite a mathematical subject good to find
  the book that best suits your level
• Also for algorithms/mathematical detail see
  Numerical Recipes, Press et al.
• And appendices of Duda, Hart and Stork and Bishop

9
Uses of NNs Neural Networks
Are For Applications
Science Character recognition
Neuroscience Optimization
Physics, mathematics statistics Financial
prediction Computer science
Automatic driving
Psychology ..............................
...........................
10
What are biological NNs?

• UNITs nerve cells called neurons, many
  different types and are extremely complex
• around 1011 neurons in the brain (depending on
  counting technique) each with 103 connections
• INTERACTIONs signal is conveyed by action
  potentials, interactions could be chemical
  (release or receive neurotransmitters) or
  electrical at the synapse
• STRUCTUREs feedforward and feedback and
  self-activation recurrent

11
(No Transcript)
12
The nerve fibre is clearly a signalling
mechanism of limited scope. It can only transmit
a succession of brief explosive waves, and the
message can only be varied by changes in the
frequency and in the total number of these
waves. But this limitation is really a small
matter, for in the body the nervous units do not
act in isolation as they do in our experiments.
A sensory stimulus will usually affect a number
of receptor organs, and its result will depend on
the composite message in many nerve fibres.
Lord Adrian, Nobel Acceptance Speech, 1932.
13
We now know its not quite that simple

• Single neurons are highly complex electrochemical
  devices
• Synaptically connected networks are only part of
  the story
• Many forms of interneuron communication now known
  acting over many different spatial and temporal
  scales

14
The complexity of a neuronal system can be
partly seen from a picture in a book on
computational neuroscience edited by Jianfeng
that I am writing a chapter for
15
How do we go from real neurons to artificial
ones?
Hillock
input
output
16

• Single neuron activity
• Membrane potential is the voltage difference
  between a neuron and its surroundings (0
  mV)

Membrane potential
17

• Single
  neuron activity
• If you measure the membrane potential of a neuron
  and print it out
• on the screen, it looks like

spike
18

• Single
  neuron activity
• A spike is generated when the membrane potential
  is greater than
• its threshold

19

• 
  Abstraction
• So we can forget all sub-threshold activity and
  concentrate on spikes (action potentials), which
  are the signals sent to other neurons

Spikes
20

• Only spikes are important since other neurons
  receive them
• (signals)
• Neurons communicate with spikes
• Information is coded by spikes

• So if we can manage to measure the spiking
  time, we decipher how the brain works .

21

• Again its not quite that simple
• spiking time in the cortex is random

22
With identical input for the identical neuron
spike patterns are similar, but not identical
23
Recording from a real neuron membrane potential
24

• Single spiking time is meaningless
• To extract useful information, we have to average
  
• to obtain the firing rate r

• for a group of neurons in a local circuit where
  neuron
• codes the same information
• over a time window

25
Hence we have firing rate of a group of neurons
So we can have a network of these local groups
r1
w1 synaptic strength
wn
rn
26
ri is the firing rate of input local
circuit The neurons at output local circuits
receives signals in the form The output
firing rate of the output local circuit is then
given by R where f is the activation
function, generally a Sigmoidal function of some
sort
wi weight, (synaptic strength) measuring the
strength of the interaction between neurons.
27
Artificial Neural networks Local circuits
(average to get firing rates) Single neuron
(send out spikes)
28
Artificial Neural Networks (ANNs)

• A network with interactions, an attempt to mimic
  the brain
• UNITs artificial neuron (linear or nonlinear
  input-output unit), small numbers, typically less
  than a few hundred
• INTERACTIONs encoded by weights, how strong a
  neuron affects others
• STRUCTUREs can be feedforward, feedback or
  recurrent

It is still far too naïve as a brain model and an
information processing device and the
development of the field relies on all of us
29
Four-layer networks
x1
x2
Input (visual input)
Output (Motor
output)
xn
Hidden layers
30

• The general artificial neuron model has five
  components, shown in the following list. (The
  subscript i indicates the i-th input or weight.)
• A set of inputs, xi.
• A set of weights, wi.
• A bias, u.
• An activation function, f.
• Neuron output, y

31
Thus the key to understanding ANNs is to
understand/generate the local input-output
relationship