Neural Networks. R

1
Neural Networks. R G Chapter 8

• 8.1 Feed-Forward Neural Networks
• otherwise known as
• The Multi-layer Perceptron
• or
• The Back-Propagation Neural Network

2
A diagramatic representation of a Feed-Forward NN
x1
x2
y
x3
Inputs and outputs are numeric.
3
Inputs and outputs

• Must be numeric, but can have any range in
  general.
• However, R G prefer to consider constraining to
  (0-1) range inputs and outputs.

4
Neural Network Input FormatReal input data
values are standardized (scaled) so that they all
have ranges from 0 1.
5
Categorical input format

• We need a way to convert categores to numberical
  values.
• For hair-colour we might have values red,
  blond, brown, black, grey.
• 3 APPROACHES
• 1. Use of (5) Dummy variables (BEST)
• Let XR1 if hair-colour red, 0 otherwise, etc
• 2. Use a binary array 3 binary inputs can
  represent 8 numbers. Hence let red (0,0,0),
  blond, (0,0,1), etc
• However, this sets up a false associations.
• 3. VERY BAD red 0.0, blond 0.25, , grey
  1.0
• Converts nominal scale into false interval scale.

6
Calculating Neuron OutputThe neuron threshhold
function. The following sigmoid function, called
the standard logistic function, is often used to
model the effect of a neuron.
Consider node i, in the hidden layer. It has
inputs x1, x2, and x3, each with a
weight-parameter.
Then calculate the output from the following
function
7
Note the output values are in the range
(0,1). This is fine if we want to use our output
to predict a probability of an event happening.
.
8
Other output types

• If we have a categorical output with several
  values, then we can use dummy output notes for
  each value of the attribute.
• E.g. if we were predicting one of 5 hair-colour
  classes, we would have 5 output nodes, with 1
  being certain yes, and 0 being certain no..
• If we have a real output variable, with values
  outside the range (0-1), then another
  transformation would be needed to get realistic
  real outputs. Usually the inverse of the scaling
  transformation. i.e.

9
Training the Feed-forward net

• The performance parameters of the feed-forward
  neural network are the weights.
• The weights have to be varied so that the
  predicted output is close to the true output
  value corresponding to the inpute values.
• Training of the ANN (Artificial Neural Net) is
  effected by
• Starting with artibrary wieghts
• Presenting the data, instance by instance
• adapting the weights according the error for
  each instance.
• Repeating until convergence.

10
8.2 Neural Network Training A Conceptual View
11
Supervised Learning/Training with Feed-Forward
Networks

• Backpropagation Learning
• Calculated error of each instance is used to
  ammend weights.
• Least squares fitting.
• All the errors for all instances are squared and
  summed (ESS). All weights are then changed to
  lower the ESS.
• BOTH METHODS GIVE THE SAME RESULTS.
• IGNOR THE R G GENETIC ALGORITHM STUFF.

12
Unsupervised Clustering with Self-Organizing Maps
13
(No Transcript)
14
r
n
n
n n r(x-n)
x
Data Instance
15
8.3 Neural Network Explanation

• Sensitivity Analysis
• Average Member Technique

16
8.4 General Considerations

• What input attributes will be used to build the
  network?
• How will the network output be represented?
• How many hidden layers should the network
  contain?
• How many nodes should there be in each hidden
  layer?
• What condition will terminate network training?

17
Neural Network Strengths

• Work well with noisy data.
• Can process numeric and categorical data.
• Appropriate for applications requiring a time
  element.
• Have performed well in several domains.
• Appropriate for supervised learning and
  unsupervised clustering.

18
Weaknesses

• Lack explanation capabilities.
• May not provide optimal solutions to problems.
• Overtraining can be a problem.

19
Building Neural Networks with iDA

• Chapter 9

20
9.1 A Four-Step Approach for Backpropagation
Learning

1. Prepare the data to be mined.
2. Define the network architecture.
3. Watch the network train.
4. Read and interpret summary results.

21
Example 1 Modeling the Exclusive-OR Function
22
(No Transcript)
23
(No Transcript)
24
Step 1 Prepare The Data To Be Mined
25
(No Transcript)
26
Step 2 Define The Network Architecture
27
(No Transcript)
28
(No Transcript)
29
Step 3 Watch The Network Train
30
(No Transcript)
31
Step 4 Read and Interpret Summary Results
32
(No Transcript)
33
(No Transcript)
34
Example 2 The Satellite Image Dataset
35
Step 1 Prepare The Data To Be Mined
36
(No Transcript)
37
Step 2 Define The Network Architecture
38
(No Transcript)
39
Step 3 Watch The Network Train
40
Step 4 Read And Interpret Summary Results
41
(No Transcript)
42
(No Transcript)
43
9.2 A Four-Step Approach for Neural Network
Clustering
44
Step 1 Prepare The Data To Be Mined

• The Deer Hunter Dataset

45
Step 2 Define The Network Architecture
46
(No Transcript)
47
Step 3 Watch The Network Train
48
(No Transcript)
49
Step 4 Read And Interpret Summary Results
50
(No Transcript)
51
(No Transcript)
52
9.3 ESX for Neural Network Cluster Analysis