Neural Network Techniques

1
Neural Network Techniques

• Judah De Paula
• Neurodynamics - Fall 2002

2
Outline

• Motivation
• Survey of neural network models
• Examples of firing rate models
• Network training
• Genetic algorithms

3
Connectionism

• Local computations as biological metaphors
• Graceful degradation gives fault-tolerant network
  design
• Allows efficient use of parallel hardware
  architectures
• Does not replace traditional AI

4
Motivations

• Desire to model the brain
• Vision systems (Miikkulainen)
• Auditory systems (Eurich et. al)
• Desire to engineer a task
• Character recognition
• Control systems (autopilot, HCI)
• Computational Theory of Mind

5
McCulloch-Pitts Neuron
6
McCulloch-Pitts Neuron

• Nervous Activity (1943) was a decade before
  Hodgkin, Huxley.
• Created logical gates with simple threshold
  activations.
• Hand designed connections with locked weight
  assignments.

7
Perceptrons

• Rosenblatt (1958) wires McCulloch-Pitts neurons
  with a training procedure.
• Failure with linearly separable problems
• XOR (X1T X2F) or (X1F X2T)
• Weakness repaired with hidden layers

8
Hopfield Networks

• Memory
• Energy Functions
• Mathematical properties inherited from physics

9
Hebbian Learning

• Simultaneous activation causes increased synaptic
  strength
• Asynchronous activation causes weakened synaptic
  connection
• Pruning
• Hebbian, anti-Hebbian, and non-Hebbian connections

10
Common Rules

• Simplify the complexity
• Task specific

11
Biological Firing Rate

• Average Firing Rate depends on biological effects
  such as leakage, saturation, and noise

12
Training Methods

• Supervised training
• Unsupervised training
• Self-organization
• Back-Propagation
• Simulated annealing
• Credit assignment

13
GA Evaluation
14
Neural Encoding
new_value old_value Mut_Range rand
15
Genetic Algorithms

• How do you choose the connection representation?
• What type of neurons do you use?
• Good evaluation functions and proper subtasks are
  critical
• Competitive selection causes bias but you also
  need to maintain diversity
• Genetic crossovers

16
Sources

• http//psy.uq.oz.au/brainwav/Manual/ActivationFun
  ctions.gif
• Haykin, Simon. Neural Networks A Comprehensive
  Foundation
• http//psy.uq.oz.au/brainwav/Manual/HopfieldArch.
  gif
• Hopfield, John J. Neurons, Dynamics and
  Computation
• Ijspeert, Auke Jan et al. Evolving swimming
  controllers for a simulated lamprey with
  inspiration from Neurobiology