Automatically Detecting Equivalent Mutants

1
Automatically Detecting Equivalent Mutants and
Infeasible Paths
By Jefferson Offutt and Jie Pan
Lectured by Oren Matza
2
Introduction
Mutation testing is a technique proposed by
DeMillo et al. (1978) and Hamlet (1977). The idea
is to take program that behave good on a test
case, change it (hence the term mutant) and
cause this faulty program to result a
failure. The goal is to make a lot of mutants,
from the original program, run it and get
different results from the original
program. Different behavior of the mutant
consider to be fault and then the mutant is
consider to be killed and dead so it will not
remain in the testing system. The test case that
was run is called efficient test case and it
save in the testing system.
3
5 mutants FUNCTION Min (I, J Integer) RETURN
Integer IS MinVal Integer Begin MinVal
I MinVal J ? 1
If (J lt I) THEN If (J gt I) THEN
? 2 If (J lt MinVal) THEN ? 3
MinVal j TRAP
? 4 MinVal I ? 5
END IF RETURN (MinVal)End Min
Example FUNCTION Min (I, J Integer) RETURN
Integer IS MinVal Integer Begin MinVal
I If (J lt I) THEN MinVal j
END IF RETURN (MinVal) End Min
1,3,5 changing operands 2 changing operator 4
sentence insert/change
4
The process of testing
? Automatic creation of mutants (types seen in
the example) ? Automatic / Manually creation of
test cases ? Run Original program on one of the
test cases. If failed, fix the bug and Run
again Else ? Run test case with each live
mutant If result different from the result
of the original program, it consider
incorrect and the mutant is killed. (the
test case is kept as effective one) At the end 2
types of mutant remains 1. Mutants which are
killable but the domain of the test cases is
not wide enough to kill them. 2. Mutants which
are equivalent in its behavior to the original
program i.e . Equivalent Mutants
The testing process cant finished until all the
mutants are killed. Detecting which of the
programs is EM was done manually. Doing this
is ? time consuming ? difficult to see in
glance which is EM and which is not, which lead
to incorrect marking of EM as non-EM and
vice versa. (Acree 1980) Because of this reason a
considerable effort should be done to find out
which are the EM in the mutants domain
5
Mutants Automatic Creation The Mothra mutation
system (DeMillo et al 1988) uses 22 mutation
operators to test Fortran 77 programs.
Mutation Operator
Description
AAR
array reference for array reference
replacement ABS
absolute value insertion ACR
array reference
for constant replacement AOR
arithmetic operator
replacement ASR
array reference for scalar variable
replacement CAR
constant for array reference
replacement CNR
comparable array name replacement CRP
constant
replacement CSR
constant for scalar variable
replacement DER
DO statement END replacement DSA
DATA
statement alterations GLR
GOTO label replacement LCR
logical
connector replacement ROR
relational operator
replacement RSR
RETURN statement replacement SAN
statement
analysis SAR
scalar variable for array reference
replacement SCR
scalar for constant replacement SDL
statement
deletion SRC
source constant replacement SVR
scalar variable
replacement UOI
unary operator insertion
6
Distribution of EM
In previous works and also in the research done
in this particular work, it was found that EM
are not evenly distribute among mutant types.
They tend to cluster around only few types. The
next table summarizes statistics from 11 programs
used in this work.
Mutant Type of
equivalent of all
mutants
ABS
47.19
4.30 ACR
14.10
1.28 SCR
7.05
0.64 UOI
6.04
0.55 SRC
4.89
0.45 SVR
4.46
0.41 ROR
3.60
0.33 SDL
2.16
0.20 CRP
1.58
0.14 AAR
1.44
0.13 RSR
1.44
0.13 LCR
1.15
0.10 ASR
1.15
0.10 CSR
1.01
0.09 SAR
1.01
0.09 All others
1.73
0.16 Total
100
9.10
7
Because ABS mutant as many more equivalent
mutants then any anther type. It can be divided
to 3 unary operators ABS -- compute
the absolute value of the expression NEGABS --
compute the negative of the absolute value
ZPUSH -- kills the mutant if the expression
is zero (then the mutant is really EM) (this
force the tester to cause the expression to be
zero - a common testing heuristic)
Do procedures for automatically detecting EM
exist ? Budd and Angluin (1982) showd that the
problem of determining if two programs are
equivalent is undecidable . Mutant and the
original program, are not just two arbitrary
programs, and they are very much syntactically
similar, but Budd and Angluin showed also that
this problem is undecidable . Still because EM
distribution and because of the fact that they
are different from the original program only in
one sentence, in many cases (statistically) it
can be determine that this mutant is EM.
8
Previous work
Buldwin and Sayward (1979) describe 6 types of
compiler optimization techniques that can be used
to identified EM. The motivation to use compiler
optimization techniques was ? programs after
optimization are mutants of the original program
? mutants can be optimization / de optimization
of the original program In 1994 Offutt and craft
designed algorithms for those 6 techniques built
a tool and succeeded to find 10 of the EM for
15 programs.
In his Ph.D (1988) Offutt presented a technique
called Constraint-Based-Testing (CBT) to
use mathematical constraints for
testing. Demillo and Offutt (1991) presented how
constraints can be used to generate test cases to
satisfy mutation testing but did not give details
how to do it. This work develop the idea, gives
strategies and algorithms for detecting EM and
show some results from implementing those
algorithms.
9
Feasible path problem
The original problem of FPA (feasible path
analysis) defined like this given a description
o a set of control flow paths through a
procedure, feasible test analysis determines if
there is input data that causes execution to flow
down some path. The generalized feasible path
problem (FTP) is given a requirement for a test
case, the feasible path problem determines if
there is inpt data that can satisfy the
requirement (constraints). This problem is
undecidable (Goldberg et al 1994, De Millo and
Offutt 1991). This work will focus in determine
feasibility with a heuristic-based set of
transformations, thus determine equivalency of
mutants.
Using Constraints to detect EM
In our context, constraint is a mathematical
expression that restrict the input space of the
program to be the portion of the space that
satisfy a certain property. For example the
constraint (x gt 0) restrict the input space only
to positive inputs. (complex constraints can be
used to restrict the input space only to inputs
that represent rectangle or sorted array) we
will use CBT to define constraints that represent
condition in which the mutant is killed. If
there is an input that satisfy the constraints
then it means that the mutant is not equivalent
and the mutant is killed. If there is not such an
input (infeasible input) then this is EM.
10
The CBT technique
We will mark P Program
M Mutant of P
S statement TC test
case. The state of the program is the values of
all data items and program counter. To kill M,
TC has to have those 3 characteristics Reachabil
ity execute the mutated statement. If will not
execute it, defiantly it will not kill M.
(Cr) Necessity it must be able to cause M
to have an incorrect state if it reaches the
mutated statement if S is
in a loop, the necessity condition must be hold
after each iteration. (the
necessity constraint requires that two
predicates/expressions will evaluate to
different results.


(Cn) Sufficiency the final state of M is
different from the final state of P.
(Cs) (Cn is necessary
but not sufficient. Cs is iff) Let D represent
the entire domain of all TC for P. D can be
divided in several ways, for each mutant ? D
Dr ? Dr ? D Dn ? Dn ? D Ds ? Ds
11
Some facts
Fact 1 - TC is an effective test case that will
kill M ? TC ? Ds for M (trivial) Fact 2 -
If TC is an effective test case that will kill, M
then TC ? Dr ? Dn (That means
that there are TC who satisfy Cr and also Cn but
not Cs - example latter) Fact 3 - Ds ? Dr ?
Dn Unfortunately finding TC such that TC ? Dr is
an undecidable problem. This is because the
determining whether TC executes S is reducible to
the halting problem. Thus a weaker condition is
defined. CR is defined such that if S is
executed, then CR 1is true. Since Cr ? CR
then the following fact is clear Fact 4 - Dr ?
DR
DR
Dn
Dr
Ds
CBT uses path expressions to describe
reachability condition (the weaker condition) ,
CR , for a statement. A path expression for a
statement S in a P is an algebraic expression
that describes a condition on test cases that
will be true when P reach S. Path expressions
usually describe multiple paths to S by using a
disjunctive formula, where each clause
represent a separate path. Path expressions are
automatically derived from the program by
extracting the predicate expressions on the
programs control flow graph.
12
Example
FUNCTION Mid (X, Y, Z Integer) RETURN Integer
IS MidVal integer BEGIN MidVal Z
IF (Y lt Z) THEN IF (X lt Y) THEN
MidVal Y ELSE IF (X lt Z) THEN
ELSE IF (XltZ) THEN ? 1 MidVal
X END IF ELSE IF (X gt Y) THEN
MidVal Y ELSE IF (X gt Z) THEN
MidVal X END IF RERURN
(MidVal) END
When x z Cn is true because (x lt z) !
(xltz) but Cs is not satisfied ( P return Z, M
return X which are equal) Mutant is not killed.
13
Formalization of what we saw until now
Constraints and detecting EM
P a program, M mutant of P. P(TC) and M(TC) are
the Outputs of P and M on TC Definition M is an
EM of P ? P(TC) M(TC) for every TC ? D This
says that if a mutant is functionally equivalent
to the original program, it is impossible to find
ant test case to kill the mutant. ?(? TCTC ? D
?P(TC) ? M(TC)) ? ?TCTC ? D ?P(TC) M(TC))
This leads to to the following
theorems THEOREM 1 Dr f (Cr is infeasible) ?
M is EM Proof (1) M is equivalent ?Ds f
--Definition, Fact 1 (2) Ds ? Dr ? Dn
--Fact 3 (3) Dr f ? Ds f
-- Rules of set, (2) (4) Dr f ? M
is equivalent --Substitutiof (1) in (3)
14
THEOREM 2 Dn f (Cn is infeasible) ? M is
EM Proof (1) M is equivalent ?Ds f
--Definition, Fact 1 (2) Ds ? Dr ? Dn
--Fact 3 (3) Dn f ? Ds f
-- Rules of set, (2) (4) Dn f ? M
is equivalent --Substitutiof (1) in
(3) THEOREM 3 Dr ? Dn f (Cr ? Cn is
infeasible) ? M is EM Proof (1) M is equivalent
?Ds f --Definition, Fact
1 (2) Ds ? Dr ? Dn
--Fact 3 (3) Dr ? Dn f ? M is equivalent
--Substitutiof (1) in (2) from the fact
that Cr ? CR the following claims could be
derived a) DR f ? M is EM b) DR ? Dn f ?
M is EM
15
All the above leads to the following
conclusions (a) If a path expression constraint
system (CR) for a mutated statement, of M , is
infeasible, then the set of test cases (DR)
that can kill M is empty - implying M is never
killed. So M is equivalent. (b) If a necessity
constraint system (Cn) for a mutant M , is
infeasible, then the set of test cases (Dn)
that can kill M is empty - implying M is never
killed. So M is equivalent. (c) If a constraint
system which is a conjunction of CR and Cn , is
infeasible, then the set of test cases (Dn
? Dn) that can kill M is empty - implying M is
never killed. So M is equivalent. This means
that to decide if a constraint system is
infeasible, there must be contradiction in the
constraint system itself. (for example the
constraint system (Xgt0) ? (Xlt 0) ) If M as a
constraint system like in the example then it is
EM. So far we saw how we can translate the
problem of detecting an EM to a problem of
finding contradictions in mathematical constraint
system. Also now test case generation uses
constraints and EM detection uses constraints.
Representation of the constraints
The expressions composed of variables and
operators from the programming language of P and
comes from the right hand side of assignments and
decision statements of P itself. It evaluate to
true or false. A clause is a list of constraints
connected with logical AND and OR. A conjunctive
clause uses only AND. All constraints kept in
disjunctive normal form (DNF) which is a list of
conjunctive clause connected only by ORs. DNF
formula referred to as a constraint system, in
which each conjunctive clause represent path
expression to a statement. During constraint
satisfaction only one clause need to be satisfied.
16
Note we said that constraint includes variables
from the program. Unfortunately this includes
internal variables. For test case generation a
symbolic evaluation (King 1976 Offutt 1991) is
used to rewrite variables to be in terms of input
variables.
Finally what we all been waiting for
The techniques to find the EM
Because it is undecidable problem, it cant be
solve algorithmically, but because EM are
currently detecting manually, even a partial
solution is valuable. There are some
off-the-shelf theories to the infeasible-constrain
t problem, but we will not use it because (1)
such a theorem gives much more then we need,
(2) it is difficult to integrate
this into already-existing software, for testing
this work. Because EM divided into more common an
d less common type, and because they are
different from the original program in a
well-defined way, we can use special techniques
to deal with this cases. 3 techniques will be
showed Negation, Constraint splitting, Constant
comparison.
17
Negation Definition 1 constraint C1 is the
negation of C2 ? the domains they describe

(a) non-overlap
(b) cover
entire domain of variables in C1 and C2
Definition 2 constraint C1 is a partial negation
of C2 ? the domains they describe

(a) non-overlap
(b)
do not cover entire domain Definition 3 two
constraints are semantically equal if they
describe the same domain Definition 4 two
constraints are syntactically equal if they
describe the same domain , and also have the same
string of symbols. (clearly two syntactically
equal constrains are also semantically equal
) Examples (1) A is the constraint x gt 1, B the
constraint x lt 1. A is negation of B and B is
negation of A (2) A is the constraint x gt 1, B
the constraint x lt 1. A is partial negation
of B and B is partial negation of A (3) A
is the constraint x gt 0, B the constraint x gt 0.
They are syntactically equal, so also
semantically equal (4) A is the constraint
x gt 0, B the constraint x gt1 (x integer). Then
they are semantically equal but not
syntactically equal
18
The negation technique is the basic technique to
recognize infeasible constraints. Just negate one
of the constraint and see if now they are
syntactically equal. If so, the constraint are
conflict (and the mutant is EM) For example A is
(xy) gt z, and B is (xy) lt z
The following table show how to negate / partial
negate a constraint

partial negation of C constraint C
Negation of C Partial negation 1
Partial negation 2 exp1 gt exp2
exp1ltexp2 exp1lt xp2
exp1exp2 exp1 gt exp2 exp1 lt
exp2 --
-- exp1 lt exp2
exp1gtexp2 exp1 gt exp2
exp1 exp2 exp1 lt exp2 exp1
gt exp2 --
-- exp1 exp2
exp1 ? exp2 exp1 gt exp2
exp1ltexp2 exp1 ? exp2 exp1
exp2 --
-- true
false
-- -- false
true
--
-- What about an algorithm ?
19
Algorithm Negation (A, B) Precondition
A and B are properly initialize
constraints. Postcondition Returns conflict if
A and B conflict, no-conflict otherwise begin neg
-A Negate (A) --use the table showed if
(neg-A syntactically equal B) return
conflict else if (the relatioon operator in
A is one of gt,lt, ) partial1-A
PartialNegate1 (A) --use the table showed
if (partial1-A syntactically equal B)
return conflict else
partial2-A PartialNegate2 (A) --use the table
showed if (partial2-A
syntactically equal B) return
conflict else
return no-conflict end-if
end-if end-if end-if end Negation
20
Constraint Splitting This technique is also
used to recognize infeasible constraints. If C
and D are two constraints and one of the
constrains is of the form (V1 AOP V2) ROP K),
then we can split this constraint. Suppose C has
this form. We will split C to two new constraints
A and B, such that C ? A ? B. It will be shown
that if A ? B conflict with D then also the
original C conflict with D. If so, the constraint
are conflict (and the mutant is EM) Note
usually A and B are weaker then C, but it easier
to decide if someone of them conflict with
D. Proof C ? A ? B ? ? C ? ( A ? B )
--implication ? (A ? B ) ? ? C
--commutativity ? ? ? ( A ? B ) ? ? C
--negation ? ? (? A ? ? B ) ? ? C --De
Morgan ? ? A ? ? B ? ? C
--commutativity now we assume that And B conflict
with D, so (1) ? A ? ? B ? D
--assumption (2) ? A ? ? B
-- (1), AND property (3) ? A ? ? B ? ? C
--assumption (4) ? C
--implication eliminationn, 2, 3 (5) D
-- (1), AND
property (6) ? C ? D --
(1),(4),(5) AND property
21
The following table show how to split constraint
C to to constraint A and B (such that C ? A ? B)
Original Constraint New
Constraint 1 New Constraint
2
(xy) gt 0 ?
x gt 0 ?
y gt 0 (xy) ? 0
? x ? 0
? y ? 0 (xy) lt 0
? x lt 0
? y lt
0 (xy) ? 0 ?
x ? 0 ?
y ? 0 (xy) 0
? x ? 0
? y ? 0 (xy) ? 0
? x ? -y (x
-y) gt 0 ?
x gt 0 ?
y lt 0 (x -y) ? 0
? x ? 0
? y ? 0 (x-y) lt 0
? x lt 0
? y gt 0
(x -y) ? 0 ?
x ? 0 ?
y ?0 (x-y) 0
? x ?0
? y ? 0 (x -y) ? 0
? x ? y (x y)
gt 0 ? x gt
0 ? y gt 0 ? x lt 0 ? y lt
0 (x y) ? 0 ?
x ? 0 ? y ? 0 ? x ? 0
? y ? 0 (x y) lt 0 ?
x gt 0 ? y lt 0 ? x
lt 0 ? y gt 0 (x y) ? 0
? x ? 0 ? y ? 0
? x ? 0 ? y ? 0 (x y) 0
? x 0
? y 0 (x
y) ? 0 ? x
? 0 ? y ? 0 (x / y) gt 0
? x gt 0 ? y gt 0
? x lt 0 ? y lt 0 (x / y) ?
0 ? x ? 0
? y gt 0 ? x ? 0 ? y lt 0
(x / y) ? 0 ?
x ? 0 ? y lt 0 ? x ? 0
? y gt 0 (x / y) lt 0 ?
x gt 0 ? y lt 0 ?
x lt 0 ? y gt 0 (x / y) 0
? x 0 (x / y) ? 0
? x ? 0
22
The Algorithm Algorithm Splitting
Constraints (NecConst, PEConst) Precondition
NecConst and PEConst are properly
initialized constraints Postcondition
Returns conflict if NecConst and PEConst
conflict, no-conflict otherwise. begin --V1 and
V2 are variables, K is a constant, aop is
aritmetic operator, rop is relation operator. if
(the format of NecConst is not ((V1 aop V2) rop
K)) return no-conflict else --use table to
split NecConst A NewConstraint1 (NecConst)
B NewConstraint2 (NecConst) endif if
(Negation (A, PEConst)conflict) AND (Negation
(B, PEConst)conflict) return conflict else
if (CompareConstraints (A, PEConst)) AND
CompareConstraints (B, PEConst))) return
conflict else return no-conflict
end if end if end
23
Constant Compression This technique is working
when both constraints have the form (v rop k). v
must be the same so the constraints are (v rop
k1) and (v rop k2). (This strategy also known as
grounding). If the constraint has the format (v
aop k1) rop k2 we can rewrite it as v rop (k2 aop
k1), so it will be in the format we want it to be
(aop is the inverse operation of aop). Then the
following table can help us determine if the two
constrains conflict.
Constraint A Constraint B
predicate (pred) Conclusion
(T for conflict, F for not) x gt k1
x gt k2 ---

F x gt k1 x ? k2
---
F x gt k1
x lt k2 K1 ? K2-1
if pred T, else F x gt
k1 x ? k2 K1 ?
K2 if
pred T, else F x gt k1 x
k2 K1 ? K2
if pred T, else F x gt k1
x ? k2 ---

F x ? k1 x gt k2
---
F x ? k1
x ? k2 ---

F x ? k1 x lt k2
K1 ? K2
if pred T, else F x ? k1
x ? k2 K1 gt K2
if pred T, else F x ? k1
x k2 K1 gt K2
if pred
T, else F x ? k1 x ? k2
---
F
24
x lt k1 x gt k2
K1 ? K21
if pred T, else F x lt k1 x
? k2 K1 ? K2
if pred T, else F x lt k1
x lt k2 ---

F x lt k1 x ? k2
---
F x lt k1
x k2 K1 ? K2
if pred T, else F
x lt k1 x ? k2
---
F x ? k1
x gt k2 K1 ? K2
if pred T, else F x ? k1
x ? k2 K1 lt K2
if pred
T, else F x ? k1 x lt k2
---
F x ? k1
x ? k2 ---

F x ? k1 x k2
K1 lt K2
if pred T, else F x ? k1
x ? k2 ---
F x
k1 x gt k2 K1 ?
K2 if
pred T, else F x k1 x ?
k2 K1 lt K2
if pred T, else F x k1
x lt k2 K1 ? K2
if pred T,
else F x k1 x ? k2
K1 gt K2
if pred T, else F x k1
x k2 K1 ? K2
if pred T, else F x
k1 x ? k2 K1
K2 if
pred T, else F x ? k1 x gt
k2 ---
F x ? k1
x ? k2 ---

F x ? k1 x lt k2
---
F x ? k1
x ? k2 ---

F x ? k1 x k2
K1 K2
if pred T, else F x ? k1
x ? k2 ---
F
25
Algorithm algorithm CompareConstants
(A, B) Precondition A and B are properly
initialize constraints. Postcondition Returns
conflict if A and B conflict, no-conflict
otherwise begin --V is a variable, k,k1,k2 are
constants, rop is relational operator, aop is
arithmetic operator if (the format of A is (V rop
K)) keep the format the same else if (the
format of A is (K rop V)) modify format to (V
rop K)) else if (the format of A is ((V aop K1)
rop K2)) modify format to (V rop (K2 aop
K1)) else if (the format of A is (K1 rop (V aop
K2))) modify format to (V rop (K1 aop
K2)) else return no-conflict end if if (the
format of B is (V rop K)) keep the format the
same else if (the format of B is (K rop V))
modify format to (V rop K)) else if (the format
of B is ((V aop K1) rop K2)) modify format to
(V rop (K2 aop K1)) else if (the format of B is
(K1 rop (V aop K2))) modify format to (V rop
(K1 aop K2)) else return no-conflict end if
26
if (the vs in A and B are not the same)
return no-conflict end if if (ConstantCompression
( A, B) true) -- see the table before
return conflict else return no-conflict end
if end CompareConstants
A proof of concept tool To test the techniques
developed above, a tool call Equivalencer was
created. Equivalencer is ? integrated with
Godzilla - a test data generator. ? inserted to
the Mothra mutation tool set. ? implemented in
C ? work on Sun Sparc workstation running SunOs
4.1.3 ? contain more then 2000 executable lines
of code ? uses some of the Mothra and Godzilla
libraries. ? It implemented inside the 3
strategies for detecting EM. First it apply
Negation, if no conflict Constant comparison, if
no conflict Constraint Splitting. Assertions In
Equivalencer, assertions are constraints that the
user insert into the test program , to restrict
the input domain of some variables
manually. Assertions on Parameter variables are
precondition and derived by hand from the
specifications. Assertions on Internal variables
can derived automatically - slicing (Weiser
1984) and control flow analysis (Fischer and
Leblanc 1988)
27
Godzilla generate constraints related to array it
does not take into account the array index
expression. This means that A(i) gt0 and A(j) lt 0
will generate A() gt 0 and A() lt0 which leads to
conflict. To avoid this Equivalencer will check
array constraints , only if it is assertion
constraint which is related to all the array.
This will be called array-extension. Equivalencer
Design 1. Initialization - open all files that
it needs and load data to memory. 2. Consult
failure information of the Godzilla about simple
EM cases. exit with conflict if found. 3. Gets
from Godzilla the path expressions and combined
it with assertions. Check with all 3
techniques. exit with conflict if found. 4.
Take necessity constraints and check with all 3
techniques. exit with conflict if found. 5.
Combine path expressions and necessity
constraints and check with all 3 techniques.
exit with conflict if found. 6. Combine
necessity constraints and array-extension and
check with all 3 techniques. exit with
conflict if found. 7. Combine path expressions
and array-extension and check with all 3
techniques. exit with conflict if
found. 8. Exit with no-conflict (no EM).
Note about efficiency as a proof -of-concept
tool it was not meant to be efficient, so every
path-expression checked against every necessity
constraint. This increase time of execution.
28
Empirical results
Program Statements Mutants
Equivalents Equi. Detected
Percent. Detected Bsearch 20
299 27
19
70.37 Bub 11
338 35
24 68.57 Cal
29 3010
236 37
15.67 Euclid 11
196 24
18
75.00 Find 28 1022
75 63
84.00 Insert
14 460
46 32
69.57 Mid 16
183 13
3
23.08 Pat 17
513 61
29 47.54 Quad
10 359
31 4
12.90 Trityp 28
951 109
80
73.39 Warshall 11 305
35 22
62.86 TotalAvg
185 7636 695
331
47.63
Discussion about the results Although the total
is less then 50, the percentage are dramatically
change between the programs (84 and 12.9) This
is because some factors, but maybe most of all it
is the limitation of the tools the Equivalencer
is based upon. First is the problem with array
reference discussed before. If constraint such as
A(4) 0 could be taken into account, then the
mutant could have been killed if A(4) was not
0. Also Godzilla associate every variable in each
statement with statement number, which propagate,
if the value of the variable is not change, to
the next statement. This make checking the
constraint complicated, specially because
Godzilla has a bug in it.
29
Feasible Path results Although this research was
focused on EM, the same technique can be used on
the problem of infeasible paths. Specifically
result of Theorem 1 is that if the reachability
condition for a sentence is infeasible then the
sentence is unreachable. To do something based
on this, some programs were artificially created,
such that some of the sentences were unreachable,
and the Equivalencer have been tried on it. (A
mutation operator was defined such as S is
unreachable ? M is EM) The results are shown in
the table below
Program unreachable
detected percentage
detected prog 1 2
1
50.00 prog 2
1 0
0.00 prog 3
1 1
100.00 prog 4
1 1
100.00 prog 5
1
1
100.00 prog 6 1
1
100.00 prog 7 2
2
100.00 prog 8
3 3
100.00 prog 9
2 0
0.00 Total Avg.
14 10
71.43
30
Improving the software ? The Equivalencer is
very heavily rely on Godzilla. Godzilla implement
symbolic evaluation as separate step from
infeasible constraint and throw away considerable
information that the Equivalencer needs,
for example reference to internal variables. ?
Godzilla generate array constraints without
indexes. If the indexes could also be take,
analysis of the CAL program found that
Equivalencer could find 69 more EM thus
increasing detection from 15.67 to
44.92. ? Give the tester opportunity to help in
recognizing difficult constraints. It will be
more easier for him then to find EM, and
will help the tool to find EM ? This software
analyze program only until the mutated statement,
but the analysis could be done further
on, until the end of the program.
Conclusions ? A partial desolation to the EM
problem was shown here. ? An algorithms were
given, and a proof-of-concept tool was built. ?
It was shown that this was effective partial
solution. ? The technique is general and could be
used to every feasible problem. ? Compare to
finding EM by hand, even this non optimized tool
was fast. ? This type of system allow the
programmer to submit a software module, and after
few minutes of commutations come up with a
set of test cases as inputs and a set of outputs
to be examined to find failure in the
software - inputs and outputs that can be latter
use for debug when a failure is found.
31
The End