Reactive Behavior Modeling Neural Networks (GATE-561)

1
Reactive Behavior ModelingNeural
Networks(GATE-561)
Dr.Çagatay ÜNDEGER Instructor Middle East
Technical University, GameTechnologies Bilkent
University, Computer Engineering General
Manager SimBT Inc. e-mail cagatay_at_undeger.com
Game Technologies Program Middle East Technical
University Fall 2009
2
Outline

• Neural Networks

3
Artificial Neural Networks (NNets)

• Inspired by human brain.
• Fundamental functional units of brain are
• Neurons / soma / nerve cells and
• Axons / dendrites connecting nerve cells.

4
Artificial Neural Networks (NNets)

• Signals can be thought as simple electrical
  impulses from axons to dendrites.
• Dendrites collect the signals.
• Soma performs a kind of summation.
• Axon fire and transmit the signal.

5
Artificial Neural Networks (NNets)

• Neural networks use a similar approach.
• Consist of
• Input arcs collect the signals,
• Neuron sums the signals and
• Output arcs transmit the summed signal based on a
  transmit function.
• Neural networks can be learned.

6
Neural Network Architecture
Arcs (storing weights)
Neurons
h
h
i
input 1
h
h
o
output 1
i
input 2
h
h
o
output 2
i
h
h
input 3
input layer
hidden layers
output layer
7
Usage of NNets

• Environmental Scanning and Classification
• Interpret vision and auditory information
• Memory
• Can learn through experience
• Behavior Control
• Outputs can be used as actions to be performed.
• Response Mapping
• Any input output mapping and association.

8
Usage of NNets

• The usage of NNets is a little bit fuzzy.
• No exact cases and rules.
• Assume it as a tool and use your creativity to
  get whatever you like.
• What NNets do is
• To seperate a solution space into partitions

9
A Neuron Architecture
B a constant bias node, which may represent
past history
w1
b
w2
y
to input of other neurons
Y
wi
Xi input i of neuron
wi weight of input i
wn
Y summing junction
y output of neuron
y is based on a transfer or activation function
10
Neuron Architecture
Summing is referred as the input activation Ya
w1
b
w2
y
Y
wi
Input activation is fed into the activation
function fa(x) and output y is acquired
wn
y fa(x)
Inputs and weights can be in range (-8, 8)
11
Some Activation Functions
y
y
1
1
0
0
x
x
?
Step function
Linear function
fs(x) 1, if x? 0, if xlt?
fs(x) x
12
Common Activation Function
y
? 1
1
0
x
Exponential / Sigmoid function
fs(x) 1 / ( 1 e-?x )
You can also scale and shift the output if
required
13
A Group Of Neurons

• A single neuron will not do a lot for us
• We need a group of them

A two layer NNet
A single layer NNet
14
Simple Logic Functions

• Created by McCulloch and Pitts in 1943.
• AND, OR, XOR functions could be done with simple
  neural network structures.

15
Simple Logic Function
AND
OR
XOR
16
Learning NNets

• Weights of NNets can be learned by various
  algorithms.
• The most common algorsthm is the Backpropagation.

17
Backpropagation

• Can learn weights of a multi-layer network
• With fixed set of units and interconnections
• Employs a gradient descent search
• To minimize the squared errors between target
  output and network output
• It is sub-optimal and may fail to reach true
  optimal point.

18
Backpropagation

• Can learn weights of a multi-layer network
• With fixed set of units and interconnections
• Employs a gradient descent search
• To minimize the squared errors between target
  output and network output

19
Backpropagation

• Error surface of the network may have multiple
  local minima.
• Therefore backprogation guarantees only to
  converge a local-optimum and may fail to reach
  the true-optimum.

20
Backpropagation
11
01
input 1
21
12
output 1
02
input 2
13
22
output 2
03
14
input 3
Initialize the network weights with small random
numbers (e.g. -0.5 ... 0.5) Train the network
with m training examples repeatedly for multiple
iterations. Stop training when a convergence
criteria is met.
21
Backpropagation
11
01
expected
actual
x01
21
12
o1
t1
02
x02
13
22
o2
t2
03
14
x03
For one training example, assume that Inputs
are x01, x02 and x03 and Expected outputs are t1
and t2 We propogate the inputs through the
network and Compute the actual outputs o1, o2
and o3
22
Backpropagation
11
expected
actual
01
x01
21
12
o1
t1
d21
02
x02
13
22
o2
t2
d22
03
14
x03
output errors
Errors will occur between actual and expected
outputs. We propagate errors backward through the
network. Compute error on each output unit k (
dk ) using dk ok (1 - ok) (tk - ok)
e.g. d21 o21 (1 - o21) (t21 - o21)
23
Backpropagation
d11
11
w11-21
expected
actual
01
x01
w11-22
d12
21
12
o1
t1
d21
02
d13
x02
13
22
o2
t2
d14
d22
03
14
x03
hidden unit errors
output errors
e.g. d11 o11 (1 - o11) (w11-21 d21 w11-22
d22 )
Compute error on each hidden unit h (dh)
using dh oh (1 - oh) S wh-k dk
24
Backpropagation
d11
x01
11
w01-11
w11-21
expected
actual
01
w01-12
x01
w11-22
d12
21
12
o1
t1
d21
02
d13
x02
13
22
o2
t2
d14
d22
03
14
x03
hidden unit errors
output errors
e.g. w01-11 w01-11 ? d11 x01-11
? learning rate (e.g. 0.05)
Update each weight w (wf-t) using wf-t wf-t
? dt xf-t
25
Backpropagation

• The main loop of the algorithm should repeatedly
  iterate over the training examples
• May iterate thousands of times using the same
  training examples.
• Until the network performs acceptably well
• Test with the entire training set or
• A better way test with a sparate test set.

26
An Action Selector Neural Network

• The inputs are state variables that equal to
  either 0 (no) or 1 (yes).
• Outputs are action variables that are between 0
  and 1.
• The action with the greatest output is selected.

27
A Number Recognizer Neural Network

• Problem To recognize normalized bitmaps of
  handwritten digits from 0 to 9.
• Data Set Collected from 43 people
• 30 people to training set
• 13 people to test set
• 32x32 bitmaps are divided into 4x4 blocks and
  black pixels are counted
• So we have 8x8 matrices and each member stores a
  number between 0 and 16

28
A Number Recognizer Neural Network

• 8x8 pixels 64 inputs
• No Missing Input Value
• Target Value class number
• from 0 to 9
• Training Set 3823 Instances
• Test Set 1797 Instances

29
A Number Recognizer Neural Network
Bias Control Unit
x012 1
x013 1
x011 1
x112
x113
0
x111
1
1
1
. . . . .
x212
x213
. . . .
. . . .
. . . .
x6412
Max
xn13
x0641 1
x1641
64
n
10
9
Result
64 Pixel values (0..16) of 8x8 image
30
A Number Recognizer Neural Network

• For each optic character (0..9), an output unit
  is used
• The class of the output unit having the highest
  value is chosen as result.
• Sample target values for training

0 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
1 0.1 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
2 0.1 0.1 0.9 0.1 0.1 0.1 0.1 0.1 0.1 0.1
31
A Number Recognizer Neural Network

• Random weights are given (0.5 ... 0.5)
• A fixed number of iterations are performed on
  Training Set
• Meanwhile weights of iterations are saved
• A test phase is performed using saved weights on
  Test Set to avoid overfitting
• The iteration weights that maximizes the test set
  accuracy is chosen

32
Test Platform
General Information
New Instance
Real Value
Result of NNet
Output layer of NNet
Loaded Instance Set
Current instance
33
First Attempt

• 4 hidden units
• 0.1 as learning rate
• 5000 iterations are performed (gt12 hours)
• Best on Test Set 86.9 on 450th iteration
• Meanwhile 92.2 on Training Set
• 82.5 accuracy on 5000th iteration
• Best on Training Set 93.0 on 1110th iteration
• Meanwhile 86.1 on Test Set

34
Graph of First Attempt
35
Second Attempt

• 4 hidden units to 8
• 0.1 as learning rate (same)
• 5000 iterations to 2500
• Best on Test Set 94.0 on 100th iteration
• Meanwhile 97.7 on Training Set
• 88.9 accuracy on 2500th iteration
• Best on Training Set 98.0 on 140th iteration
• Meanwhile 93.6 on Test Set

36
Graph of Second Attempt
37
Third Attempt

• 8 hidden units to 16
• 0.1 as learning rate (same)
• 2500 iterations to 200
• Best on Test Set 95.2 on 176th iteration
• Meanwhile 98.7 on Training Set
• 95.2 accuracy on 200th iteration
• Best on Training Set 98.7 on 176th iteration
• Meanwhile 95.2 on Test Set

38
Graph of Third Attempt
39
Fourth Attempt

• 16 hidden units (same)
• 0.1 as learning rate to 0.05
• 200 iterations to 400
• Best on Test Set 95.0 on 122th iteration
• Meanwhile 98.4 on Training Set
• 94.4 accuracy on 400th iteration
• Best on Training Set 98.7 on 394th iteration
• Meanwhile 94.4 on Test Set

40
Graph of Fourth Attempt
41
Last Attempt

• 16 hidden units to 32
• 0.05 as learning rate (same)
• 400 iterations to 1000 (4 hours and 38 minutes)
• Best on Test Set 96.4 on 315th iteration
• Meanwhile 99.5 on Training Set
• 96.0 accuracy on 1000th iteration
• Best on Training Set 99.9 on 945th iteration
• Meanwhile 96.1 on Test Set

42
Graph of Last Attempt
43
Advantages

• Handle errors well.
• Graceful degradation.
• Can learn novel solutions.

44
Disadvantages

• Cant understand how the learned network works,
  therefore they are the second best way to do
  something.
• Need as many examples as possible.
• Learning takes too much time / processing.
• Sometimes the network may not converge.