Evolutionary Unit Testing of ObjectOriented Software

1

• Evolutionary Unit Testing of Object-Oriented
  Software
• Stefan Wappler, DaimlerChrysler AG

2
Agenda

• motivation
• object-oriented programming and testing
• evolutionary testing of object-oriented software
• case studies
• conclusion

3
Motivation

• more and more object-oriented software in nearly
  all application domains (even controler software
  for embedded systems)
• need to automate (development) testing to reduce
  costs
• application of evolutionary algorithms promising
  since they proved successful for the automatic
  generation of test data for procedural software

4
Object-Oriented Programming

• object set of functions working with
  instance-specific data
• data encapsulation (preventing direct access to
  certain variables)
• inheritance (reuse of implementations)
• abstract and generic classes, interfaces
• polymorphism

5
Testing Object-Oriented Software

• research traditional techniques based on code
  coverage effectively applicable (Kim et al.)
• especially data-flow-based and state-based
  techniques promising
• ? investigation to adapt the control-flow-coverage
  -based approach using evolutionary algorithms to
  the needs of object-oriented testing

6
Challenge of ET of Object-Oriented Software

• search for test programs (e.g. JUnit test class
  methods)
• target object needed to call a method for
• parameters of methods can be objects
• some objects participating in the test must be in
  a certain state in order to reach the test goal

public class CUT public void
methodUnderTest(Helper hlp) if(
hlp.getState() 0 ) // test goal
method to be tested
CUT cut1 new CUT() Helper h1 new
Helper() h1.setState( 0 ) cut1.methodUnderTest(
h1 )
example test program individual to be found by
an EA
7
Encoding of Information to Individuals
CUT cut1 new CUT() Helper h1 new
Helper() h1.setState( 0 ) cut1.mut( h1 )
test program
assignment and basic-type optimisation
call sequence optimisation
CUT.CUT() Helper.Helper() Helper.setState(int) CUT
.mut(Helper)
- - h1 CUT-gtcut1, Helper-gth1
- - int-gt0 -
object assignments
basic-typeparameters
method calls
8
Optimisation Strategy

• two-level optimisation
• 1st level optimisation of call sequences
• - generation of skeletons ( call sequences)
• - extension/manipulation of the skeletons
• 2nd level optimisation of object assignments
  and basic-type input data
• - analysis of skeleton for missing information
• - optimisation of missing information using
  standard evolutionary algorithms
• - termination criterion running best
  objective value

9
1st Optimisation Level Call Sequence Generation
(Initial)

• initial approach
• map all methods to IDs and construct ID sequences
• optimise the ID sequences numerically (standard
  EA)
• map the ID sequences back to method call
  sequences
• weakness of this approach
• evolutionary operators do not take into account
  the semantics of the IDs
• invalid ID sequences can be produced need for
  objective function to evaluate the correctness of
  the ID sequences complex evaluation model
  necessary

10
1st Optimisation Level Call Sequence Generation
(Improved)

• static analysis to compute all sequences
  describing the creation of objects which are
  necessary to call the method under test
• (partially ordered) object sets each set
  represents the objects which must be present for
  the call of mut
• ordered object sets produced by all possible
  sortings of the object sets
• skeleton sets in each object set the objects are
  replaced by one of their creation methods
  (constructor or object-returning method)
• each skeleton represents one most simplistic test
  program call sequence, the set of all skeletons
  makes up the 0. generation of the EA
• manipulation of the skeletons insertion/removal
  of method calls, crossover (similar/according to
  Tonella operators take into account the
  semantics of the individuals)

11
1st Optimisation Level Call Sequence Generation
Example
Test Cluster
object set C, B, A
class A public A(B b) public void mut(C
c)
ordered sets (C, B, A), (C, B, C, A)
skeletons ( C,
B, A)
C.C(int), B.B(C), A.A(B) C.C(int), B.B(C),
B.get(), B.B(C), A.A(B) (C, B,
C, A) C.C(int), B.B(C), C.C(int),
A.A(B) C.C(int), B.B(C), B.get(), A.A(B)
class B public B(C c) public C get()
class C public C(int i)
12
2nd Optimisation Level Object Assignments and
Basic-Type Input Data
call sequence optimisation
c1 C(?) // int? b1 B(c1) c2 C(?) //
int? a1 A(b1) a1.mut(?) // c1 or c2?
A.A(B), C.C(int), B.B(C), C.C(int)
skeleton
extended skeleton
skeleton analyis
optimisation of (int, int, 1,2)
basic-type input data
object assignment
13
Overall Optimisation Process
Skeleton Generation
Reinsertion
Mutation
execution, monitoring and objective value
calculation
Recombination
standard EA
Selection
Skeleton EA
14
Objective Functions

• Objective Function for Skeleton EA
• simply returns best achieved 2nd level
  objective value (termination criterion of 2nd
  level optimisation? running best obj.val as
  initial approach)
• Objective Function for Standard EA
• distance-based approach using approach level
  and local distance, but what about state
  behaviour of the objects (and flag problems)?

15
Condition Analysis

• classification of conditions
• boolean conditional operator working on boolean
  sets, e.g. flag true, obj null,
  isValid()
• indirect at least one operand delivered by a
  method, e.g. isValid(), obj.getX() 0
• pointer conditional operator working on
  addresses, e.g. obj null (subset of boolean)
• instanceof application of the instanceof
  operator, e.g. obj instanceof SpecialClass

16
Condition Analysis Intention of the classes

• boolean conditions hard to find a good
  objective function for
• indirect conditions probably states are
  involved, object management plays an important
  role
• pointer conditions very hard to find a good
  objective function for
• instanceof conditions almost impossible to
  find a good objective function for?

17
Condition Analysis Empirical Results
project name classes conditions booleans indirects
pointers instanceofs Jakarta REGEXP 13 122 23 40
11 0 Jakarta ORO 66 534 117 102 47 12 Apache
Xalan 407 3507 2347 1053 1579 422 Apache
Ant 410 3401 2527 1639 1371 87 summary 896 756
4 5014 2834 3008 521
project name classes conditions booleans indirects
pointers instanceofs Jakarta REGEXP 13 122 0,19
0,33 0,09 0,00 Jakarta ORO 66 534 0,22 0,19 0,09
0,02 Apache Xalan 407 3507 0,67 0,30 0,45 0,12 Ap
ache Ant 410 3401 0,74 0,48 0,40 0,03 summary 8
96 7564 0,66 0,37 0,40 0,07
? about 2/3 of all conditions are boolean
18
Condition Analysis -- Consequences

• objective functions for boolean conditions
  required? McMinns hybrid approach
  (EAchaining)?
• generation of the chaining tree considering the
  object context of the events (object context
  instance for which the event must occur)?
  derivation of call sequence skeletons?
  derivation of constraints for the object
  assignments of the 2nd optimisation level
  (according to the object context)

19
The State Problem?
StateClass3 obj new StateClass3() obj.countUp()
obj.countUp() obj.countUp() obj.countUp() obj
.countUp() obj.countUp() obj.test( 0 )

• public class StateClass3
• private int counter0
• public StateClass3()
• public void countUp()
• 1 counter
• public boolean test(int a)
• 2 if( a 403 )
• 3 countUp()
• 4 if( counter gt 5 )
• 5 return true
• 6 return false

definition of counter
private
implicit definition of counter
StateClass3 obj new StateClass3() obj.test(
403 ) obj.test( 403 ) obj.test( 403
) obj.test( 403 ) obj.test( 403 ) obj.test(
403 )
value of counter depends on call sequence and
parameter value
20
The State Problem again

• public class StateClass4
• private int counter0
• public StateClass4()
• public boolean isOk()
• 1 if( counterlt10 )
• 2 return true
• 3 else return false
• public boolean incr()
• 4 if( isOk() )
• 5 counter
• 6 return true
• else

event (node, constraint set, object context)
lt(4,0)gt lt(5,counter,a), (4,0,a)gt lt(5,0,a),(5,co
unter,a),(4,0,a)gt
event sequences
skeleton constraint
a.incr() a.incr() a.incr()
collect all skeletons from the generated set
which can potentially meet the skeleton
constraint derived from an event sequence
21
Case Study All statements coverage criterion
public class ClassA int counter 0
public ClassA() public void incr()
counter public void
decr() counter-- public
boolean mut(int a) if( counter gt 5 )
if( a 403 ) return
true return false
public class ClassB int counter 0
public ClassB() public void incr()
counter public void
decr() counter-- public
boolean mut(int a) if( a 403 ) if(
counter gt 5 ) return true return
false
solution after 13 1st level generations (average
after 4 runs)
solution after 14 1st level generations (average
after 4 runs)
22
Conclusion

• generate unit test cases automatically
• 2-level optimization first generate and optimize
  call sequences, on the 2nd level optimize object
  assignments and basic-type input data
• use chaining in difficult cases
• so far approach successful when no state
  behaviour
• to do implement/develop further the ideas of the
  chaining approach

23
References

• Kim et al. S Kim, JA Clark, JA McDermid.
  Investigating the applicability of traditional
  test adequacy criteria for object-oriented
  programs, In Proceedings of the ObjectDays 2000
• Tonella P Tonella. Evolutionary testing of
  classes, International Symposium on Software
  Testing and Analysis (ISSTA), pages 119-128,
  2004