Solving Feature Subset Selection Problem by a Hybrid Metaheuristic

1
Solving Feature Subset Selection Problem by a
Hybrid Metaheuristic

• Miguel García Torres,
• Félix C. García López,
• Belén Melián Batista,
• José A. Moreno Pérez, (speaker)
• J. Marcos Moreno Vega
• Intelligent Computation Group
• University of La Laguna
• http//webpages.ull.es/users/gci/
• gci_at_ull.es

2
Index

• Introduction
• Feature Subset Selection Problem.
• VNSTS for the FSS problem.
• Computational Experiments.
• Conclusions.

3
1 INTRODUCTION

• Machine learning
• Instance-based learning
• Bayesian learning
• Decision Trees
• Metaheuristics
• Variable Neighbourhood Search
• Tabu Search

4
Machine Learning

• Classification
• To classify instances from their features
• Supervised Classification
• A set of instances with known classes is given
• Instance-based Learning uses nearest example(s)
• Bayesian Learning highest a posteriori
  probability
• Decision Trees a series of test based on the
  features

5
Feature Selection

• Select a subset of features to classify
• Filter (different methodology in training and
  testing)
• FOCUS (those classifying correctly the training
  set)
• RELIEF (assigns weights to the features)
• Wrapper (same methodology in training and
  testing)
• SFS (Sequential Forward Selection)
• SBE (Sequential Backward Elimination)

6
Metaheuristics

• Genetic Algorithms (GA)
• A population that evolves by crossover,
  mutation
• and selection
• Variable Neighbourhood Search (VNS)
• Systematic change in the neighbourhood
• Tabu Search (TS)
• Use of a selective history of the search
• Hybrids

7
2 FEATURE SELECTION PROBLEM
Formulation Objective accuracy k-fold cross
validation
8
Formulation of the FSS problem

• Let A be a set of instances characterized by d
  features
• X Xj j 1, , d (nominal or linear).
• To carry out the classification task, let T ? A
  the training set of instances and V A \ T the
  validation or test set.
• Objective to find the subset of features S ? X
  that provides the highest accuracy percentage in
  classification.
• Estimation of the accuracy k-fold cross
  validation.

9
k-Fold Cross Validation

• Estimating accuracy percentages of S ? X on B ?
  A.
• Divide B in k disjoint subsets Bi i 1, , k.
• Perform k executions each execution i
• Bi is the test set
• TB\Bi is the training set
• ca is the class of a and is that assigned
  by the classifier.

10
5x2 - Cross Validation

• 5 random arrangements and k 2
• The estimated accuracy is the average among the 5
  arrangements

A1
A
1
f(.)
A2
2
1
A2
ltf(.)gt
2
2
f(.)
A1
1
11
3 VNS/TS for the FSS problem

• FSS heuristics
• Usual heuristics SFS, SBE
• Usual MH GA
• New MH VNS, TS, hybrid

12
Standard Heuristics

• Sequential Forward Selection (SFS)
• Start with an empty set of features.
• Each step adds the feature with which the
  solution most improves the accuracy.
• If there is no improvement, stops.
• Sequential Backward Elimination (SBE)
• Start with the whole set of features.
• Each step removes the feature with which the
  solution most improves the accuracy.
• If there is no improvement, stops.

13
GA characteristics

• Is widely used in Machine Learning
• Population based metaheuristic.
• Solutions evolve by selection criteria
• Evolution based on mutation and crossover
  operators.
• It is the most used in AI.

14
GA Pseudocode

• t ? 0.
• Generate Pop(t)
• Evaluate Pop(t)
• Repeat
• t ? t 1.
• Select Pop(t)
• Cross Pop(t)
• Mutate Pop(t)
• Evaluate Pop(t)
• Until (StoppingCriterion)

15
VNS characteristics

• Variable Neighbourhood Search (VNS) is a recent
  metaheuristic based on systematic change of the
  neighbourhood in a search.
• It uses a series of neighborhoods Nk, k 1, ,
  kmax,
• They may be induced by a metric function on
  solutions.
• If d(.,.) is a distance and dk, k 1..kmax,
  are increasing values then Nk(s) s?S
  d(s,s) dk
• They may be induced by a single neighborhood if
  N(.) is a neighborhood take N1(s) N(s) and
  Nk1(s) N(Nk(s)).
• Thus a k-neighborhood move is performed by k
  1-moves.

16
VNS pseudocode

• Initialization
• Choose Nk, k 1, , kmax and the initial
  solution s.
• Iterations
• Repeat, until the stopping criterion is met
• (1) Set k ? 1.
• (2) Repeat, until k kmax
• (a) Shake Generate s ? Nk(s).
• (b) Local Search apply a local search to s to
  get s.
• (c) Move or not If f(s) lt f(s) then s ? s and
  k ? 1. Otherwise, k ? k 1.

17
Initialization

• Take the empty solution S
• Execute SFS and SBE one after other
• Until no improvement.
• SFS
• add the feature which most improves the
  accuracy, while there is an improvement.
• SBE
• remove the feature which most improves the
  accuracy, while there is an improvement.

18
Shaking

• Generates a solution s at random from the
  k-neighborhooh, S ? Nk(S)
• Choose k features at random from the solution S.
• Choose k features at random out the solution S.
• The the solution is generated on this way

19
Local Search

• Take the solution S
• Execute SFS and SBE one after other
• Until no improvement.
• SFS
• add the feature which most improves the
  accuracy, while there is an improvement.
• SBE
• remove the feature which most improves the
  accuracy, while there is an improvement.

20
Move or not

• If f(s) lt f(s) then s ? s and k ? 1.
• Otherwise, k ? k 1.

21
Hybridization

• Add the set of features involved in the shaking
  into a tabu list.
• Features included in tha tabu list can not be
  used for shaking in the next iteration.

22
4 COMPUTATIONAL RESULTS
COMPUTATIONAL EXPERIMENTS
23
Introduction

• Classifiers IB1, Naïve Bayes, C4.5.
• Data Bases UCI repository
• Evaluation 5x2 cross validation.
• Statistical comparisons
• VNSTS vs. VNS vs. GA

24
Data Bases
25
The Comparisons

• 5 x 2 F test Alpaydin, 1999
• The difference between the error rates of the two
  classifiers is pi(j).
• The average difference and the estimated variance
  are
• Under H0 pi(j) is the difference between two
  identical proportions.
• pi(j)/s N(0,1) Dietterich, 1998 and
  (pi(j)/s)2 ?21.
• Therefore, (N/10)/(M/5) F10,5

26
GA parameters

• Guided by Bäck,1996
• Crossover probability 1.0
• Mutation probability 1/d, (d is the number of
  features)
• Population size 1000.
• Fitness proportionate selection for crossover.
• New population the best from parents and
  offspring.
• Stopping criterion if best individual in current
  generation does not improve best individual.
• Used in FSS problem by Inza et al., 2001

27
NB Classifier Accuracy
28
C4.5 Classifier Accuracy
29
Naive Bayes Number of features
30
C4.5 Number of features
31
Naive Bayes CPU time and iterations
32
C4.5 CPU time and iterations
33
5 Conclusions

• Hybridization
• Performance
• Accuracy
• Number of Features
• CPU Time

34
The contributions

• We propose an hybrid VNS/TS based on VNS and TS
  metaheuristics to solve the FSS problem.
• This hybrid algorithm (VNS/TS) combines
  characteristics of both metaheuristics and a
  standard local search in ML.
• VNSTS is experimentally compared with VNS and GA
  using specific statistical tests for
  cross-validation.
• Data sets and classifiers are standard in ML
  experiments.

35
The results

• VNSTS and VNS get similar accuracy than GA
• VNSTS and VNS get better reduction than GA (with
  similar accuracy)
• VNSTS and VNS get much more efficient (CPU time
  of smaller level of magnitude) than GA (for
  better results)
• VNSTS gets better CPU and number of iterations
  than VNS (for similar accuracy and number of
  features)

36
Solving Feature Subset Selection Problem by a
Hybrid Metaheuristic
Thanks

• Miguel García Torres,
• Félix C. García López,
• Belén Melián Batista,
• José A. Moreno Pérez, (jamoreno_at_ull.es)
• J. Marcos Moreno Vega
• Intelligent Computation Group
• University of La Laguna.
• http//webpages.ull.es/users/gci/
• gci_at_ull.es