Pattern Recognition: Statistical and Neural

1
Nanjing University of Science Technology
Pattern RecognitionStatistical and Neural
Lonnie C. Ludeman Lecture 17 Oct 21, 2005
2
Lecture 17 Topics

• View of Perceptron Algorithm in Pattern Space
• Fractional Correction Perceptron Algorithm
• Simplified Perceptron Algorithm
• 4. Derivation of the perceptron algorithm
• 5. Extension of Perceptron Algorithm to M Class
  Case 3 Special Cases

3
Motivation
4
Question How do we find separating
Hyperplane??? a needle in the haystack
Answer The Perceptron Algorithm !!!!
Other ways exist like random selection
5
Motivation
Separating Hyperplane
6
Linear Discriminant Functions
where
Augmented Pattern Vector
Weight vector
C1
Decision Rule
if d(x) gt 0
lt
C2
7
Review Perceptron Algorithm
Finds a hyperplane that separates two sets of
patterns
Algorithm
8
Review Perceptron Algorithm
New Training Sample
9
Review Perceptron Algorithm
wT(k)x(k) lt 0
wT(k)x(k) gt 0
If c is too large we may not get convergence
10
View of Perceptron correction in pattern space
11
Fractional Correction Perceptron Algorithm
Weight Update Equation
Where
12
Weight update in original pattern space
13
Weight update in original pattern space
14
Weight update in original pattern space
15
Simplified Perceptron Algorithm
Given the following samples from two classes
We want a weight vector w such that
Negated patterns for class C2
16
We could define two classes as
Consider the set of samples x of C11 U C21 then
the weight vector update is as follows
That is all of the samples should be on the
positive side of the hyperplane boundary
17
This is a simplification of the perceptron
algorithm in that it only contains one if branch
Simplified algorithm
1. Augment all patterns in C1 and C2 2. Negate
the augmented patterns of C2 3. Combine the two
sets above to form one set of samples 4. Iterate
through this set using the weight update as
18
Derivation of the Perceptron Algorithm
Define the following performance measure
19
Minimizing J for each sample will satisfy the
conditions desired
To minimize J we can use the gradient algorithm
where the weight update is as follows
20
The partial derivatives with respect to each
weight are determined as
21
Substituting these partials into the weight
update equation yields the following
for
22
Which can be written in the following vector form
Which is the perceptron algorithm in augmented
and negated form. (end of proof)
23
Other Perceptron like Algorithms
If we use different performance measures in the
preceding proof we get perceptron like algorithms
to accomplish the separation of classes.
24
Other meaningful performance measures
Each leads to different weight update equations.
Each requires a convergence theorem to be useful.
25
So far only worked with separating two classes.
Most problems have more than two classes.
Question
Can we modify the perceptron algorithm to work
with more than two classes?
Answer
Yes for certain special cases.
26
Case 1. Pattern classes Group separable
27
Case 1. K Pattern classes - Group separable
Given
S1 is set of samples from class C1 S2 is set of
samples from class C2 SK is set of samples from
class CK S S1 U S2 U . . . U SK

Define
Assume Pattern classes are group separable
S1 linearly separable from S1/ S S1 S2
linearly separable from S2/ S S2 SK
linearly separable from SK/ S SK

28
Find K Linear Discriminant functions that
separate the Sk in the following manner
Discriminant Functions
for k 1, 2, , K
dk(x)wk1x1 wk2x2 wknxn wkn1


Decision Rule If dj(x) gt 0 then decide Cj
Solution
Find dk(x) by using the Perceptron algorithm on
the two classes Sk and Sk/
29
Solution Continued
(1) Find d1(x) by using the Perceptron algorithm
on the two sets S1 and S1/
(2) Find d2(x) by using the Perceptron algorithm
on the two sets S2 and S2/

(K) Find dK(x) by using the Perceptron algorithm
on the two sets SK and SK/
30
Solution Continued
This gives the decision boundaries as
for k 1, 2, , K
0 wk1x1 wk2x2 wknxn wkn1


31
Case 2. Pattern classes Pairwise separable
32
Case 2. Pattern classes Pairwise separable
Find Linear Discriminant functions that separate
all pairs of Sk in the following manner
dkj(x)wkj1x1 wkj2x2 wkjnxn wkjn1


Decision Rule
for k 1, 2, , K
On Boundaries decide randomly
33
Solution
for k 1, 2, , K
j 1, 2, , K , j k
Find dkj(x) by using the Perceptron algorithm on
the two classes Sk and Sj .
Notes
(1) dkj(x) - djk(x) .
(2) Requires determining K(K-1)/2 discriminant
functions
34
Case 3. K Pattern classes separable by K
discriminant functions
35
(No Transcript)
36
(No Transcript)
37
Flow Diagram for Case 3 Perceptron separability
38
Summary Lecture 17

• View of Perceptron Algorithm in Pattern Space
• Fractional Correction Perceptron Algorithm
• Simplified Perceptron Algorithm
• 4. Derivation of the perceptron algorithm
• 5. Extension of Perceptron Algorithm to M Class
  Case 3 Special Cases

39
End of Lecture 17