The Perceptron

1
The Perceptron

• CS/CMPE 537 Neural Networks

2
The Perceptron Basics

• Simplest and one of the earliest neural network
  model proposed by Rosenblatt in 1958, 1962
• It is based on the McCulloch-Pitts model of a
  neuron
• Characteristics
• Single-layer feedforward network (A layer of
  input nodes and one layer of computation/output
  nodes)
• Threshold activation function (or hard limiter
  function)
• Performs classification of linearly separable
  patterns
• Trained using error-correcting learning

3
The Perceptron

• Consider a single neuron perceptron

b
1
4
Linear Separability (1)

• The function of the single neuron perceptron is
  to classify the input x into one of two classes,
  C1 and C2.
• In general, a q neuron perceptron can classify
  the input x into 2q classes
• For the two classes case, the decision boundary
  is defined by the hyperplane
• Si1 p wixi b 0

5
Linear Separability (2)

• When p 2 (i.e. two inputs), the decision
  boundary is a line

b 0
6
Error-Correction Learning and Pattern
Classification
1
b
7
Pattern Classification (1)

• Let n number of training samples (set X) X1
  set of training sample belonging to C1 X2 set
  of training sample belonging to C2
• For a given sample n
• x(n) 1, x1(n),, xp(n)T input vector
• w(n) b(n), w1(n),, wp(n)T weight vector
• Net activity level
• v(n) wT(n)x(n)
• Output
• y(n) 1 if v(n) gt 0 and y(n) -1 otherwise
• The decision hyperplane separates classes C1 and
  C2

8
Pattern Classification (2)

• If the two classes C1 and C2 are linearly
  separable, then there exists a weight vector w
  such that
• wTx 0 for all x belonging to class C1
• wTx lt 0 for all x belonging to class C2

9
Error-Correction Learning

• Update rule w(n 1) w(n) ?w(n)
• Learning process
• If x(n) is correctly classified by w(n), then
• w(n 1) w(n)
• Otherwise, the weight vector is updated as
  follows
• w(n 1) w(n) ?(n)x(n) if w(n)Tx(n) 0 and
  x(n) belongs to C2
• and
• w(n 1) w(n) ?(n)x(n) if w(n)Tx(n) lt 0 and
  x(n) belongs to C1

10
Perceptron Convergence Algorithm (1)

• Variables and parameters
• x(n) 1, x1(n),, xp(n) w(n) b(n),
  w1(n),,wp(n)
• y(n) actual response (output) d(n) desired
  response
• ? learning rate, a positive number less than 1
• Step 1 Initialization
• Set w(0) 0, then do the following for n 1, 2,
  3,
• Step 2 Activation
• Activate the perceptron by applying input vector
  x(n) and desired output d(n)
• Step 3 Computation of actual response
• y(n) sgnwT(n)x(n)
• Where sgn(.) is the signum function

11
Perceptron Convergence Algorithm (2)

• Step 4 Adaptation of weight vector
• w(n1) w(n) ?d(n) y(n)x(n)
• Where
• d(n) 1 if x(n) belongs to C1
• d(n) -1 if x(n) belongs to C2
• Step 5
• Increment n by 1, and go back to step 2

12
Performance Measure (1)

• A learning rule is designed to optimize a
  performance measure
• However, in the development of the perceptron
  convergence algorithm we did not mention a
  performance measure
• Intuitively, what would be an appropriate
  performance measure for a classification neural
  network?
• Define the performance measure
• J -Ee(n)v(n)
• Or, as an instantaneous estimate
• J(n) -e(n)v(n)
• e(n) error at iteration n d(n) y(n) v(n)
  linear combiner output at iteration n E
  expectation operator

13
Performance Measure (2)

• Can we derive our learning rule by minimizing
  this performance function
• Now v(n) wT(n)x(n), thus
• Learning rule

14
Concluding Remarks

• A single layer perceptron can perform pattern
  classification only on linearly separable
  patterns, regardless of the type of nonlinearity
  (hard limiter, signoidal)
• Papert and Minsky in 1969 elucidated limitations
  of Rosenblatts single layer perceptron (e.g.
  requirement of linear separability, inability to
  solve XOR problem) and cast doubt on the
  viability of neural networks
• However, multilayer perceptron and the
  back-propagation algorithm overcomes many of the
  shortcomings of the single layer perceptron