Nature requires Nurture

1
Nature requires Nurture

• Initial wiring is genetically controlled
• Sperry Experiment
• But environmental input critical in early
  development
• Occular dominance columns
• Hubel and Wiesel experiment

2
(No Transcript)
3
Cat Striate Cortex Layer IV
CLOSED EYE
OPEN EYE
Monkey Striate Cortex Area 17 (V1) Layer IV
4
Critical Periods in Development

• There are critical periods in development (pre
  and post-natal) where stimulation is essential
  for fine tuning of brain connections.
• Other examples of columns
• Orientation columns

5
Pre-Natal Tuning Internally generated tuning
signals

• But in the womb, what provides the feedback to
  establish which neural circuits are the right
  ones to strengthen?
• Not a problem for motor circuits - the infant
  moves its limbs to refine the feedback and
  control networks.
• But there is no vision in the womb.
• --Systematic moving patterns of activity are
  spontaneously generated pre-natally in the
  retina.
• A predictable pattern, changing over time,
  provides excellent training data for tuning the
  connections between visual maps.
• The pre-natal development of the auditory system
• Research indicates that infants, immediately
  after birth, preferentially recognize the sounds
  of their native language over others. The
  assumption is that similar activity-dependent
  tuning mechanisms work with speech signals
  perceived in the womb.

6
Post-natal environmental tuning

• The pre-natal tuning of neural connections using
  simulated activity can work quite well
• a newborn colt or calf is essentially functional
  at birth.
• This is necessary because the herd is always on
  the move.
• For many animals, including people, experience is
  absolutely necessary for normal development (as
  in the kitten experiment).
• For a similar reason, if a human child has one
  weak eye, the doctor will sometimes place a patch
  over the stronger one, forcing the weaker eye to
  gain experience.

7
Adult Plasticity and Regeneration

• The brain has an amazing ability to reorganize
  itself through new pathways and connections
  rapidly.
• Through Practice
• London cab drivers, motor regions for the
  skilled
• After damage or injury
• Undamaged neurons make new connections and take
  over functionality or establish new functions
• But requires stimulation
• Stimulation standard technique for stroke victim
  rehabilitation

8
When nerve stimulation changes, as with
amputation, the brain reorganizes. In one theory,
signals from a finger and thumb of an uninjured
person travel independantly to separate regions
in the brain's thalamus (left). After amputation,
however, neurons that formerly responded to
signals from the finger respond to signals from
the thumb (right).
9
Possible explanation for the recovery mechanism

• The initial pruning of connections leaves some
  redundant connections that are inhibited by the
  more active neural tissue.
• When there is damage to an area, the lateral
  inhibition is removed and the redundant
  connections become active
• The then can undergo activity based tuning based
  on stimulation.
• Great area for research.

10
Summary

• Both genetic factors and activity dependent
  factors play a role in developing the brain
  architecture and circuitry.
• There are critical developmental periods where
  nurture is essential, but there is also a great
  ability for the adult brain to regenerate.
• Next What computational models satisfy some of
  the biological constraints.
• Question What is the relevance of development
  and learning in language and thought?

11
Connectionist Models Basics

• Srini Narayanan
• CS182/CogSci110/Ling109
• Spring 2008

12
(No Transcript)
13
(Spike After Potential)
Excitatory PSP
Inhibitory PSP
14
Neural networks abstract from the details of real
neurons

• Conductivity delays are neglected
• An output signal is either discrete (e.g., 0 or
  1) or it is a real-valued number (e.g., between 0
  and 1)
• Net input is calculated as the weighted spatial
  sum of the input signals
• Net input is transformed into an output signal
  via a simple function (e.g., a threshold function)

15
The McCullough-Pitts Neuron
Threshold

• yj output from unit j
• Wij weight on connection from j to i
• xi weighted sum of input to unit i

16
Neural nets Mapping from neuron
Nervous System Computational Abstraction
Neuron Node
Dendrites Input link and propagation
Cell Body Combination function, threshold, activation function
Axon Output link
Spike rate Output
Synaptic strength Connection strength/weight
17
Simple Threshold Linear Unit
18
Simple Neuron Model
1
19
A Simple Example

• a x1w1x2w2x3w3... xnwn
• a 1x1 0.5x2 0.1x3
• x1 0, x2 1, x3 0
• Net(input) f 0.5
• Threshold bias 1
• Net(input) threshold biaslt 0
• Output 0

.
20
Simple Neuron Model
1
1
1
1
21
Simple Neuron Model
1
1
1
1
1
22
Simple Neuron Model
0
1
1
1
23
Simple Neuron Model
0
1
0
1
1
24
Abstract Neuron
25
Computing with Abstract Neurons

• McCollough-Pitts Neurons were initially used to
  model
• pattern classification
• size small AND shape round AND color green
  AND location on_tree gt unripe
• linking classified patterns to behavior
• size large OR motion approaching gt
  move_away
• size small AND direction above gt move_above
• McCollough-Pitts Neurons can compute logical
  functions.
• AND, NOT, OR

26
Computing logical functions the OR function
i1 i2 y0
0 0 0
0 1 1
1 0 1
1 1 1

• Assume a binary threshold activation function.
• What should you set w01, w02 and w0b to be so
  that you can get the right answers for y0?

27
Many answers would work

• y f (w01i1 w02i2 w0bb)
• recall the threshold function
• the separation happens when w01i1 w02i2 w0bb
  0
• move things around and you get
• i2 - (w01/w02)i1 - (w0bb/w02)

28
Decision Hyperplane

• The two classes are therefore separated by the
  decision' line which is defined by putting the
  activation equal to the threshold.
• It turns out that it is possible to generalise
  this result to TLUs with n inputs.
• In 3-D the two classes are separated by a
  decision-plane.
• In n-D this becomes a decision-hyperplane.

29
Linearly separable patterns
PERCEPTRON is an architecture which can solve
this type of decision boundary problem. An "on"
response in the output node represents one
class, and an "off" response represents the
other.
Linearly Separable Patterns
30
The XOR function
i1 i2 y
0 0 0
0 1 1
1 0 1
1 1 0
31
The Input Pattern Space
 
32
The Decision planes
 
33
Multiple Layers
y
0.5
-1
1
1.5
0.5
1
1
1
1
I1
I2
34
Multiple Layers
y
0.5
-1
1
1.5
0.5
1
1
1
1
I1
I2
0
1
35
Multiple Layers
y
0.5
-1
1
1.5
0.5
1
1
1
1
I1
I2
1
1
36
Types of abstract neuron parameters

• The form of the combination function - e.g.
  linear, sigma-pi, cubic.
• The activation-output relation - linear,
  hard-limiter, or sigmoidal.
• The nature of the signals used to communicate
  between nodes - analogue or boolean.
• The dynamics of the node - deterministic or
  stochastic.
• Spatio temporal information encoding
• Pulse coding and Spiking Neurons

37
Different Activation Functions
BIAS UNIT With X0 1

• Threshold Activation Function (step)
• Piecewise Linear Activation Function
• Sigmoid Activation Funtion
• Gaussian Activation Function
• Radial Basis Function

38
Types of Activation functions
39
The Sigmoid Function
ya
xneti
40
Nice Property of Sigmoids
41
The Sigmoid Function
Output1
ya
Output0
xneti
42
The Sigmoid Function
Output1
Sensitivity to input
ya
Output0
xneti
43
Changing the exponent k(neti)
K gt1
K lt 1
44
Nice Property of Sigmoids
45
Radial Basis Function
46
Stochastic units

• Replace the binary threshold units by binary
  stochastic units that make biased random
  decisions.
• The temperature controls the amount of noise

temperature
47
Spiking Neurons and Pulse coding

• Rate coding (ex. Sigmoid units)
• Spatial summation of input
• Output is the average number of spikes in some
  time window (normalized between 0 and 1).
• Pulse coding (More realistic)
• Look at each individual spike (the time it is
  generated)
• Can take into account refractory period
• EXAMPLE Integrate and fire neurons
• EXAMPLE Time to first spike (Thorpe 1996).
• Adds power to the basic neuron by adding temporal
  information

48
(No Transcript)
49
Triangle Nodes Encoding relational information
with abstract neurons

• The triangle node (aka 2/3 node) is a useful
  function that activates its outputs (3) if any
  (2) of its 3 inputs are active
• Such a node will be useful for lots of
  representations.

50
Triangle nodes and McCullough-Pitts Neurons?
Relation
Object
Value
A
B
C
51
Representing concepts using triangle nodes
triangle nodes when two of the neurons fire, the
third also fires
52
Networks of Triangle nodes example sentence
They all rose

• triangle nodes
• when two of the abstract neurons fire, the third
  also fires
• model of spreading activation

53
Link to Vision The Necker Cube
54
Basic Ideas behind connectionist models

• Parallel activation streams.
• Top down and bottom up activation combine to
  determine the best matching structure.
• Triangle nodes bind features of objects to values
• Mutual inhibition and competition between
  structures
• Mental connections are active neural connections

55
5 levels of Neural Theory of Language
Spatial Relation
Motor Control
Pyscholinguistic experiments
Metaphor
Grammar
Cognition and Language
Computation
Structured Connectionism
abstraction
Neural Net
SHRUTI
Computational Neurobiology
Triangle Nodes
Biology
Neural Development
Midterm
Quiz
Finals