Classification

1
Classification

• Heejune Ahn
• SeoulTech
• Last updated 2015. May. 03

2
Outline

• Introduction
• Purpose, type, and an example
• Classification design
• Design flow
• Simple classifier
• Linear discriminant functions
• Mahalanobis distance
• Bayesian classification
• K-means clustering unsupervised learning

3
1.Pupose

• Purpose
• For decision making
• Topics of Pattern recognition (in artificial
  intelligence)
• Model
• Automation and Human intervention
• Task specification what classes, what features
• Algorithm to used
• Training tuning algorithm parameters

Classifier (classification rules)
Features (patterns, structures)
classes
Images
4
2. Supervised vs unsupervised

• Supervised (classification)
• trained by examples (by humans)
• Unsupervised (clustering)
• only by feature data
• using the mathematical properties (statistics) of
  data set

5
3. An example

• Classifying nuts

Pine-nuts Lentils Pumpkin seeds
Features (circularity, line-fit-error)
Classifier (classification rules)
pine nut
lentil
pumpkin seed
6

• Observations
• What if a single features used?
• What for the singular points?
• Classification
• draw boundaries

7

• Terminalogy

8
4. Design Flow
9
5. Prototypes min-distance classifier

• Prototypes
• mean of training samples in each class

 
 
10
(No Transcript)
11
6. Linear discriminant

• Linear discriminant function
• g(x1,x2) ax1 bx2 c 0
• Ex 11.1 Fig11.6

12
(No Transcript)
13
8. Mahalanobis distance

• Problems In min-dist.
• mean-value only, no distribution considered
• e.g. (right figure)
• std(class 1) ltlt std(class 2)
• Mahalanobis dist.

Variance considered. (larger variance, less
distance)
14
9. Bayesian classification

• Idea
• To assign each data to the most-probable class,
  based on apriori-known probability
• Assumption
• Priors (probability for class) are known.
• Bayes theorem

15
10. Bayes decision rule

• Classification rule

Intuitively
Bayes Theorem
Class-conditional probability density function
Prior probability
Total probability Not used in classification
decision
16

• Interpretation
• Need to know priors and class-conditional pdf
  often not available
• MVN (multivariate normal) distribution model
• Practically quite good approximation
• MVN
• N-D Normal distribution with

17
12. Bayesian classifier for M-varirates
taking log( ) It is monotonic increasing
function
18

• Case 1 identical independent

• Linear Machine the decision region is
  hyper-plane (linears)
• Note when same prob(w), then Minimum distance
  criterion

19

• Case 2 all covariance is same
• Matlab
• class, err  classify(test, training, group,
  type, prior)
• training and test
• Type DiagLinear for naïve Baysian

20

• Ex11.3

wrong priors
correct priors
21
13. Ensemble classifier

• Combining multiple classifiers
• Utilizing diversity, similar to ask multiple
  experts for decision.
• AdaBoost
• Weak classifier change (1/2) lt accuracy ltlt 1.0
• weighting mis-classified training data for next
  classifiers

22

• AdaBoost in details
• Given
• Initialize weight
• For t 1, . . ., T
• WeakLearn, which return the weak classifier
  with minimum
  error w.r.t. distribution Dt
• Choose
• Update
  
  
  
• Where Zt is a normalization factor chosen
  so that Dt1 is a distribution
• Output the strong classifier

23
14. K-means clustering

• K-means
• Unsupervised classification
• Group data to minimize
• Iterative algorithm
• (re-)assign Xis to class
• (re-)calculate ci
• Demo
• http//shabal.in/visuals/kmeans/3.html

24

• Issues
• Sensitive to initial centroid values.
• Multiple trials needed gt choose the best one
• K ( of clusters) should be given.
• Trade-off in K (bigger) and the objective
  function (smaller)
• No optimal algorithm to determine it.
• Nevertheless
• used in most of un-supervised clustering now.

25

• Ex11.4 F11.10
• kmeans function
• classIndexes, centers kmeans(data, k,
  options)
• k of clusters
• Options Replicates', Display