Artificial Intelligence

1
Artificial Intelligence

• Lecture 8
• Artificial Neural Network II

2
Overview

• Gradient Descent
• Multilayer Networks
• Sigmoid Unit
• Backpropagation Algorithm

3
The Delta Rule
4
(No Transcript)
5
Gradient Descent
6
Gradient Descent Summary

• Perceptron training rule guaranteed to succeed if
  
• Training examples are linearly separable
• Sufficiently small learning rate ?
• Linear unit training rule uses gradient descent
• Guaranteed to converge to hypothesis with minimum
  squared error
• Given sufficiently small learning rate ?
• Even when training data contains noise
• Even when training data not separable by H

7
Incremental (Stochastic) Gradient Descent
8
Multilayer Networks of Sigmoid Units
9
Sigmoid Unit
10
Error Gradient for a Sigmoid Unit
11
Backpropagation
12
Backpropagation Algorithm
Propagate input forward through the network
Propagate error backward through the network
13
More on Backpropagation
14
Summary

• Gradient Descent Summary
• Multilayer Networks
• Sigmoid Unit
• Backpropagation Algorithm

15
Possible Quiz Questions

• If there is a quiz next time, it might cover
• What is a sigmoid unit?
• Why should we use a sigmoid unit?
• What is the error gradient for a sigmoid unit?
• How does backpropagation work?
• What is the purpose of using stochastic gradient
  descent?
• How to avoid local minimum?

16
Summary

• Gradient Descent
• Multilayer Networks
• Sigmoid Unit
• Backpropagation Algorithm