Apply Evolutionary Algorithm EA to Subset Selection Problem

1
Apply Evolutionary Algorithm (EA) to Subset
Selection Problem

• AbdulSalam Kalaji, Robert Hierons, Steven Swift
• Mark Harman, Zheng Li.
• Brunel University, Dept. of IS and Computing
• King's College London, Dept of Computer
  Science

2
Agenda

• What is subset selection problem
• Matrix representation
• One application (test suite reduction)
• How to employ evolutionary algorithms
• Experiments results
• Discussion
• References

3
Preliminaries

• Subset selection problem is the problem of
  finding a minimal set of test cases that
  satisfies the entire set of objectives 1.
• The problem is NP-Complete NP-Compete problems
  simply have no efficient algorithms to solve 2.
• Used in many applications i.e. signal coding for
  compression, chemical analysis of compounds and
  direction finding 3, and test suite reduction.
• Can be represented as a matrix where rows
  represent test cases and columns represent
  objectives.

4
Matrix representation

• Rows represent test cases, while columns
  represent objectives.

5
Test Suite Reduction as a Subset Selection Problem
A Statement Coverage adequate Suite T t1 (x
1) t2 (x 2) t3 (x 4) t4 (x -1) t5
(x -4) t6 (x -6) t7 (x 6) t8 (x 10)
1 0 0 0 0 0 0 1
6
EA and Subset Selection problem

• Given such a matrix, find the minimal set of test
  cases that achieve the complete set of
  Objectives.
• To Apply EA to the problem, we have to
• Define test adequacy criterion.
• Define the objective function.
• Test adequacy criterion is achieving all the
  objectives by the minimal set of test cases.

7
Objective function calculation

• fitness 0
• reward (TotalNumOfTestCases -1)
  (NumOfObjectives 4)
• For each t of T do
• If t 0 then fitness fitness 1
• Else fitness NumOfObjectives achieved by t
  4
• If T achieves all objectives then fitness
  fitness reward
• Objective Value (2 Reward ) fitness

8
Example (1)
1- Optimal case
T1 0 1 0 0
T2 0 1 1 0
Reward (total number of test cases -1) (total
number of objectives 4) Reward (3) (14)
19.

• T1 Fitness calculation
• There are three zeros (t1, t3, t4) ? fitness
  fitness 3
• There is one test case (t2) that achieves 4
  objectives ? fitness fitness 16 19
• T1 achieves all the objectives ? fitness
  fitness Reward 38
• Objective value (2 Reward) (fitness) ? (38)
  (38) 0 ? An optimal solution

• T2 Fitness calculation
• There are two zeros (t1, t4) ? fitness fitness
  2
• Two test case (t2, t3) achieve 4 objectives ?
  fitness fitness 16 18
• T2 achieves all the objectives ? fitness
  fitness Reward 37
• Objective value (2 Reward) (fitness) ? (38)
  (37) 1

9
Example (2)
1- Main diagonal case
T1 1 1 1 1
T2 0 1 1 0
Reward (total number of test cases -1) (total
number of objectives 4) Reward (3) (14)
19.

• T1 Fitness calculation
• There is no zero ? fitness fitness 0
• There are four test cases (t1, t2, t3, t4) that
  achieves 4 objectives ? fitness fitness 16
  16
• T1 achieves all the objectives ? fitness
  fitness Reward 35
• Objective value (2 Reward) (fitness) ? (38)
  (35) 3

• T2 Fitness calculation
• There are two zeros (t1, t4) ? fitness fitness
  2
• Two test case (t2, t3) achieve 2 objectives ?
  fitness fitness 8 10
• T2 does not achieve all the objectives ? fitness
  fitness 0 10
• Objective value (2 Reward) (fitness) ? (38)
  (10) 28

10
Experiments

• Subjects are Siemens programs test suites

• Tool is GeatBx EA tool box for Matlab 6

11
Results
12
Disagreement

• Previous research shows some disagreements.
• The empirical results show that a reduced test
  suite is equally efficient, or only slightly
  worse, in detecting program faults 4.
• However, a later research shows that a reduced
  test suite can potentially compromise the fault
  detection capability 5.
• This trade-off between saving in the size and
  fault detection capability requires further
  investigation and it is not in the scope of this
  paper.

13
Summary Conclusions

• Threats to external validity variety of
  different real world problems needed in order to
  generalise.
• The technique provides a stable method to apply
  EA to Subset Selection Problem.
• Real world examples in the area of test suite
  minimisation show significant saving in the size
  and therefore potentially reducing the testing
  cost.

14
References

• G.H. Golub and C.F.V. Loan, Matrix Computitions,
  2nd. ed. The Johns Hopkins University Press,
  Baltimore, MD, 1989.
• Garey, M. and Johnson, D. Computers and
  intractability a guide to the theory of
  NP-completeness. W.H. Freeman and Company, New
  York, 1979.
• Nafie, M., Tewfik, A.H. and Ali, M. Deterministic
  and iterative solutions to subset selection
  problems. Signal Processing, IEEE Transactions on
  see also Acoustics, Speech, and Signal
  Processing, IEEE Transactions on, 50 (7).
  1591-1601.
• Wong, W.E., Horgan, J.R., London, S. and Mathur,
  A.P. Effect of test set minimization on fault
  detection effectiveness. Softw. Pract. Exper, 28
  (4). 347-369.
• Rothermel, G., Harrold, M.J., Ostrin, J. and
  Hong, C., An empirical study of the effects of
  minimization on the fault detection capabilities
  of test suites. in, (1998), 34-43.
• Pohlheim, H. Genetic and evolutionary algorithms
  toolbox for use with matlab.

15

• Any Questions
• ??