Artificial Intelligence Methods

1
Artificial Intelligence Methods

• Neural Networks
• Lecture 2
• Rakesh K. Bissoondeeal

2
ANNs Architectures

• Architecture
• - layers, connections, activation (transfer)
  functions
• Layers ANNs are usually thought as arranged in
  layers single and multilayer
• single layer

3
ANNs Architectures

• Multilayer ANNs
• -In addition to input and output
• layers
• - hidden layer(s)
• - allow(s) the ANN to
• learn nonlinear
• relationships

4
ANNs Architectures

• Single layer network
• - learning is simpler
• - limited in the tasks it can do
• Multilayer network
• -Learning is more difficult
• -can solve more complicated problems
• e.g network with hidden layer can approximate
  any continuous function

5
ANNs Architectures

• Optimal number of hidden layers
• -depends on the problem
• e.g. function approximation
• -network with 1 hidden layer can approximate any
  continuous function (well known and commonly
  used)
• -2 or more can be beneficial to certain problems
  but number of parameters (weights) increases
  problematic with small data samples

6
ANNs Architectures

• Optimal number of neurons (nodes)
• - input nodes input variables
• - depends on analysis, theory may help
• - hidden nodes nodes in hidden layer
• - too many nodes overfitting, i.e., networks
  perform well within sample, but poor
  out-of-sample performance
• - too little difficulty to learn pattern in
  the data
• - output nodes output
• - easiest to choose
• - depends on the problem

7
ANNs Architectures

• Connections
• -Feedforward and feedback
• Feedforward networks
• - no link backwards
• - each node is connected to nodes in the next
  layer
• - no links between nodes in same layer
• - no links skip a layer
• - simpler, proven most useful

8
ANNs Architectures

• Connections
• - Feedback networks
• - feedback from output to input units
• - Complex dynamics learning is more difficult
  in feedback network
• - E.g Hopfield network used for associative
  memory
• - Train network with a set of pictures, then
  present a piece of one pictures, network will
  produce the picture from which the piece of
  picture taken

9
ANNs Architectures

• Activation functions
• Usually same function used for all neurons in the
  same layer
• Linear and nonlinear activation functions
• - Linear functions, usually f(x)x

10
ANNs Architectures

• Activation functions
• - nonlinear activation functions
• - step function
• - sign function
• - logistic sigmoid function
• - hyperbolic tangent function

11
ANNs Architecture
Nonlinear functions
sign
step
12
ANNs Architectures
Nonlinear activation function Logistic Sigmoid
13
ANN Architectures
Nonlinear activation function hyperbolic tangent
14
Learning (Training)

• How do ANNs learn?
• -McCulloch-Pitt Neuron has no concept of
  learning
• - Learning is achieved by modifying the weights
  in the network
• - Weights are initially randomly selected
• Two types of Learning
• - Supervised
• - Unsupervised

15
Learning (Training)

• Supervised learning
• - Bulk of networks use supervised training
• - Both inputs and outputs provided
• (p1,t1), (p2,t2), . . ., (pn, tn)
• - Network processes inputs and compares
  resulting output with desired output
• - depending on the size the error (desired
  output-network output) the weights are adjusted
  using an appropriate learning algorithm

16
Learning (Training)

• Supervised learning
• -Process repeated until the error is considered
  to be small enough.
• Some cases error 0
• Some cases error cannot be O, avoid
  overtraining
• -Training set, validation set, test set
• -Examples, Backpropagation, Quasi-Newton,
  Levernberg-Marqualt

17
Learning (Training)

• Unsupervised learning
• - The network is provided with inputs but not
  the desired output
• - Often used for finding the patterns in the
  data rather than for modelling input/output
  mapping
• - The network must itself decide what features
  to use to group the input data
• -Not very well understood

18
Recommended Reading

• Fundamentals of neural networks Architectures,
  Algorithms and Applications, L. Fausett, 1994.
• Artificial Intelligence A Modern Approach, S.
  Russel and P. Norvig, 1995.
• An Introduction to Neural Networks. 2nd Edition,
  Morton, IM.