Title: Model based Software Testing Test Assessment and Enhancement
1Model based Software Testing Test Assessment
and Enhancement
- Aditya P. Mathur
- Purdue University
- Fall 2005
Last update August 18, 2005
2Learning Objectives
- To understand the relevance and importance of
test assessment.
- To learn the fundamental principle underlying
test assessment.
- To learn various methods and tools for test
assessment.
- To understand the relative strengths/weaknesses
of test assessment methods.
- To learn how to improve tests based on a test
assessment procedure.
3What is Test Assessment?
- Once a test set T, a collection of test inputs,
has been developed, we ask - How good is T?
- It is the measurement of the goodness of T which
is known as test assessment.
- Test assessment is carried out based on one or
more criteria.
4Test Assessment (contd.)
- These criteria are known as test adequacy
criteria.
- Test assessment is also known as test adequacy
assessment.
5Test assessment (contd.)
- Test assessment provides the following
information
- A metric, also known as the adequacy score or
coverage, usually between 0 and 1.
- A list of all the weaknesses found in T, which
when removed, will raise the score to 1.
- The weaknesses depend on the criteria used for
assessment.
6Test assessment (contd.)
- Once the coverage has been computed, and the
weaknesses identified, one can improve T.
- Improvement of T is done by examining one or more
weaknesses and constructing new test requirements
designed to overcome the weakness(es).
- The new test requirements lead to new test
specifications and to further testing of the
program.
7Test Assessment (contd.)
- This is continued until all weaknesses are
overcome, i.e. the adequacy criterion is
satisfied (coverage1).
- In some instances it may not be possible to
satisfy the adequacy criteria for one or more of
the following reasons - Lack of sufficient manpower
- Weaknesses that cannot be removed because they
are infeasible.
8Test Assessment (contd.)
- The cost of removing the weaknesses is not
justified.
- While improving T by removing its weaknesses, one
usually tests the program more thoroughly than it
has been tested so far.
- This additional testing is likely to result in
the discovery of remaining errors.
9Test Assessment (contd.)
- Hence we say that test assessment and improvement
helps in the improvement of software reliability.
- Test assessment and improvement is applicable
throughout the testing process and during all
stages of software development.
10Test Assessment Procedure
11Principle Underlying Test Assessment
- There is a uniform principle that underlies test
assessment throughout the testing process.
- This principle is referred to as the coverage
principle.
- It has come about as a result of intensive
research at Purdue and other research groups in
software testing.
12The Coverage Principle
- To formulate and understand the coverage
principle, we need to understand - coverage domains
- coverage elements
- A coverage domain is a finite domain, related to
the program under test, that we want to cover.
Coverage elements are the individual elements of
this domain
13The Coverage Principle (contd.)
Coverage Domains
Coverage Elements
14The Coverage Principle (contd.)
- Measuring test adequacy and improving a test set
against a sequence of well defined, increasingly
strong, coverage domains leads to improved
confidence in the reliability of the system under
test.
15The Coverage Principle (contd.)
- Note the following properties of a coverage
domain
- It is related to the program under test.
- It may come from program requirements, related to
the inputs and outputs.
16The Coverage Principle (contd.)
- It may come from program code. Can you think of a
coverage domain that comes from the program code?
- It aids in measuring test adequacy as well as the
progress made in testing. How?
17The Coverage Principle (contd.)
- Example
- It is required to write a program that takes in
the name of a person as a string and searches for
the name in a file of names. The program must
output the record ID which matches the given
name. In case of no match a -1 is returned.
What coverage domains can be identified from this
requirement?
18The Coverage Principle (contd.)
- As we learned earlier, improving coverage
improves our confidence in the correct
functioning of the program under test.
- Given a program P and a test T suppose that T is
adequate w.r.t. a coverage criterion C.
- Does this mean that P is error free?
Obviously???
19Test Effort
- There are several measures of test effort.
- One measure is the size of T. By this measure a
test set with a larger number of test cases
corresponds to higher effort than one with a
lesser number of test cases.
20Error Detection Effectiveness
- Each coverage criterion has its error detection
ability. This is also known as the error
detection effectiveness or simply effectiveness
of the criterion.
- One measure of the effectiveness of criterion C
is the fraction of faults guaranteed to be
revealed by a test T that satisfies C.
21Effectiveness (contd.)
- Another measure is the probability that at least
fraction f of the faults in P will be revealed
by test T that satisfies C.
- Unfortunately there is no absolute measure of the
effectiveness of any given coverage criterion for
a general class of programs and for arbitrary
test sets.
22Effectiveness (contd.)
- One coverage criterion results in an exception to
this rule What is it?
- Empirical studies conducted by researchers give
us an idea of the relative goodness of various
coverage criteria.
- Thus, for a variety of criteria we can make a
statement like Criterion C1 is definitely better
than criterion C2.
23Effectiveness-continued
- In some cases we may be able to say Criterion C1
is probably better than criterion C2.
- Such information allows us to construct a
hierarchy of coverage criteria.
- This hierarchy is helpful in organizing and
managing testing. How?
24Strength of a coverage criterion
- The effectiveness of a coverage criterion is also
referred to as its strength.
- Strength is a measure of the criterions ability
to reveal faults in a program.
- Criterion C1 is considered stronger than
criterion C2 if C1 is is capable of revealing
more faults than C2.
25The Saturation Effect
- The rate at which new faults are discovered
reduces as test adequacy with respect to a finite
coverage domain increases it reduces to zero
when the coverage domain has been exhausted.
26Saturation Effect Fault View
27 Saturation Effect Reliability View
Functional, Decision, Dataflow, and
Mutation tsting provide various test assessment
criteria.
28Coverage principle-discussion
- Discuss
- How will you use the knowledge of coverage
principle and the saturation effect in organizing
and managing testing?
Can you think of any other uses of the coverage
principle and the saturation effect?
29Control flow graph
- Control flow graph (CFG) of a program is a
representation of the flow of execution within
the program.
- More formally, a CFG G is
- G(N,A)
- where N set of nodes and A set of arcs
- There is a unique entry node en in N.
- There is a unique exit node ex in N. A node
represents a single statement or a block.
- A block is a single-entry-single-exit sequence of
instructions that are always executed in a
sequence without any diversion of path except at
the end of the block.
30Control flow graph (contd.)
- Every statement in a block, except possibly the
first one, has exactly one predecessor.
- Similarly, every statement in the block, except
possibly the last one, has exactly one successor.
- An arc a in A is a pair (n,m) of nodes from N
which represent transfer of control from node n
to node m.
- A path of length k in G is an ordered sequence of
arcs, from A such that
31Control flow graph (contd.)
- For any two adjacent arcs ai (n,m) and aj
(p,q), mp.
- A path is considered executable or feasible if
there exists a test case which causes this path
to be traversed during program execution,
otherwise the path is unexecutable or infeasible.
32Control flow graph-example
- Exercise
- Draw a CFG for the following program and identify
all paths.
1. scanf (x,y) if (ylt0) 2. pow0-y 3. else
powy 4. z1.0 5. while (pow !0) 6. zzx
powpow-1 7. if (ylt0) 8. z1.0/z 9. printf(z)
What does the above program compute?
33 Control-flow Graph
34Structure-based Test Adequacy
- Based on the CFG of a program several test
adequacy criteria can be defined.
- statement coverage criterion
- branch coverage criterion
- condition coverage criterion
- path coverage criterion
35Statement Coverage
- The coverage domain consists of all statements in
the program. Restated, in terms of the control
flow graph, it is the set of all nodes in G.
- A test T satisfies the statement coverage
criterion if upon execution of P on each element
of T, each statement of P has been executed at
least once.
36Statement coverage (contd.)
- Restated in terms of G, T is adequate w.r.t. the
statement coverage criterion if each node in N is
on at least one of the paths traversed when P is
executed on each element of T.
37Statement Coverage (contd.)
- Class exercise
- For the program for which you have drawn the
control flow graph, develop a test set that
satisfies the statement coverage criterion.
- Follow the procedure for test assessment and
improvement suggested earlier.
38Statement Coverage-Weakness
- Consider the following program
- int abs (x)
- int x
-
- if (xgt0) x0-x
- return x
-
39Statement coverage-weakness
- Clearly, T satisfies the statement coverage
criterion.
- But is the program correct and is the error
revealed by T which is adequate w.r.t. the
statement coverage criterion?
What do you suggest we do to improve T?
40Branch (or edge) coverage
- In G there may be nodes which correspond to
conditions in P. Such nodes, also called
condition nodes, contain branches in P.
- Each such node is considered covered if during
some execution of P, the condition evaluates to
true and false these executions of P need not
be the same.
41Branch coverage
- The coverage domain consists of all branches in
G. Restated, in terms of the control flow graph,
it is the set of all arcs exiting the condition
nodes.
- A test T satisfies the branch coverage criterion
if upon execution of P on each element of T, each
branch of P has been executed at least once.
42Branch coverage
- Class exercise
- Identify all condition nodes in the flow graph
you have drawn earlier. - Does T (x0) satisfy the branch coverage
criterion? - If not, then improve it so that it does.
43Branch Coverage-Weakness
- Consider the following program that is supposed
to check if the input data item is in the range 0
to 100, inclusive
int check(x) int x if ((xgt0 ) (xlt200))
checktrue else checkfalse
44Branch Coverage-Weakness
- Class exercise
- Do you notice the error in this program?
- Find a test set T which is adequate w.r.t.
statement coverage and does not reveal the error. - Improve T so that it is adequate w.r.t. branch
coverage and does not reveal the error. - What do you conclude about the weakness of the
branch coverage criterion?
45Condition Coverage
- Condition nodes in G might have compound
conditions.
- For example, in the check program the condition
node contains the condition
((xgt0 ) (xlt200))
- This is a compound condition which consists of
the elementary conditions xgt0 and xlt200.
46Condition coverage (contd.)
- A compound condition is considered covered if all
of its constituent elementary conditions evaluate
to true and false, respectively, during some
execution of P.
- A test set T is adequate w.r.t. condition
coverage if all conditions in P are covered when
P is executed on elements of T.
47Condition coverage (contd.)
- Class exercise
- Improve T from the previous exercise so that it
is adequate w.r.t. the condition coverage
criterion for the check function and does not
reveal the error. - Do you find the above possible?
48Branch coverage-weakness (contd.)
- Consider the following program
0. int set_z(x,y) 1. int x,y 2. if
(x!0) 3. y5 4. else zz-x 5. if
(zgt1) 6. zz/x 7. else 8. zy
49Branch Coverage-Weakness
- Class exercise
- Construct T for set_z such that (a) T is adequate
w.r.t. the branch coverage criterion and (b) does
not reveal the error. - What do you conclude about the effectiveness of
the branch and condition coverage criteria?
50Path coverage
- As mentioned before, a path through a program is
a sequence of statements such that the entry node
of the program CFG is the first node on the path
and the exit node is the last one on the path.
Is this definition equivalent to the one given
earlier?
51Path coverage (contd.)
- A test set T is considered adequate w.r.t. the
path coverage criterion if all paths in P are
executed at least once upon execution on each
element of T.
- Class exercise
- Construct T for set_z such that T is adequate
w.r.t. the path coverage criterion and does not
reveal the error. - Is the above possible?
52Path Coverage-Weakness
- The number of paths in a program is usually very
large.
- How many paths in set_z ?
- How many paths in check ?
53Path Coverage-Weaknesses
- It is the infinite or a prohibitively large
number of paths that prevent the use of this
criterion in practice.
- Suppose that a test set T covers all paths. Will
it guarantee that all errors in P are revealed ?
- Is obtaining 100 path coverage equivalent to
exhaustive testing?
54Variants of Path Coverage
- As path coverage is usually impossible to attain,
other heuristics have been proposed.
- Make sure that each loop is executed 0, 1, and 2
times.
- Try several combinations of if and switch
statements. The combinations must come from
requirements.
55Hierarchy in Control flow criteria
Path coverage
56Exercise
- Develop a test set T that is adequate w.r.t. the
statement, condition, and the loop coverage
criteria for the exponentiation program.
57Test strategy
- One can develop a test strategy based on any of
the criteria discussed.
- Example
- A test strategy based on the statement coverage
criterion will begin by evaluating a test set T
against this criterion. Then new tests will be
added to T until all the statements are covered,
i.e. T satisfies the criterion.
58Definitions
- Error-sensitive path a path whose execution
might lead to eventual detection of an error.
- Error revealing path a path whose execution will
always cause the program to fail and the error to
be detected.
59Definitions Reliable Technique
- Reliable A test technique is reliable for an
error if it guarantees that the error will always
be detected.
- This implies that a reliable testing technique
must lead to the exercising of at least one
error-revealing path.
60Definitions Weakly Reliable
- Weakly reliable A test technique is weakly
reliable if it forces the execution of at least
one error sensitive path.
61Example Error Detection 1
- Let us go over the example in Korel and Laskis
paper.
- It is a sorting program which uses the bubble
sort algorithm.
- It sorts an array a0N in descending order.
- There are two, nested, loops in the program.
- The inner loop from i6-i10 finds the largest
element of aR1N.
62Example Error Detection (contd.)
- The largest element is saved in R0 and R3 points
to the location of R0 in a.
- The outer loop swaps a(R1) with a(R3).
- The completion of one iteration of the outer loop
ensures that the sub-array a0R1-1 has been
sorted and that aR1-1 is greater than or equal
to any element of aR1N.
63Example Error Detection (contd.)
- There is a missing re-initialization of R3 to R1
at the beginning of the inner loop.
- In some cases this will cause the program to
fail. - What are these cases?
- We will get back to this error later!
64Data flow graph
- It represents the flow of data in a program.
- The graph is constructed from the control flow
graph (CFG) of the program.
- A statement that occurs within a node of the CFG
might contain variables occurrences.
- Each variable occurrence is classified as a def
or a use.
65defs and uses
- A def represents the definition of a variable.
Here are some sample defs of variable x - xyx
- scanf(x,y)
- int x
- xi-1yx
All defs of x are italicized.
- A use represents the use of a variable in a
statement. Here a few examples of use of variable
x
66def-use (contd.)
All uses of x are italicized.
- xx1
- printf (x is d, y is d, x,y)
- cout ltlt x ltlt endl ltlt y
- zxi1
- if (xlty)
- Uses of a variable in input and assignments are
classified as c-uses. Those in conditions are
classified as p-uses.
67def-use (contd.)
- c-use stands for computational use and p-use for
predicate-use.
- Both c- and p-uses affect the flow of control
p-uses directly as their values are used in
evaluating conditions and c-uses indirectly as
their values are used to compute other variables
which in turn affect the outcome of condition
evaluation.
68def-use (contd.)
- A path from node i to node j is said to be
def-clear w.r.t. a variable x if there is no def
of x in the nodes along the path from node i to
node j. Nodes i and j may have a def of x.
- A def-clear path from node i to edge (j,k) is one
in which no node on the path has a def of x.
69global-def
- A def of a variable x is considered global to its
block if it is the last def of x within that
block.
- A c-use of x in a block is considered global
c-use if there is no def of x preceding this
c-use within this block.
70def-use graph definitions
- def(i) set of all variables for which there is a
global definition at node i.
- c-use(i) set of all variables that have a
global c-use at node i.
- p-use(i,j) set of all variables for which there
is a p-use for the edge (i,j).
- dcu(x,i) set of all nodes such that each node
has x in its c-use and x is in def(i).
71def-use graph definitions
- dpu(x,i) set of all edges such that each edge
has x in its p-use , x is in def(i).
- The def-use graph of program P is constructed by
associating defs, c-use, and p-use sets with
nodes of a flow graph. -
72def-use graph (contd.)
Sample program
1. scanf (x,y) if (ylt0) 2. pow0-y 3. else
powy 4. z1.0 5. while (pow !0) 6. zzx
powpow-1 7. if (ylt0) 8. z1.0/z 9. printf(z)
73def-use graph (contd.)
Unlabeled edges imply empty p-use set.
defx,y c-use?
1
y
y
defpow c-usey
defpow c-usey
2
3
4
defz c-use?
def? c-use?
5
def? c-use?
pow
pow
defz,pow c-usez,x,pow
7
6
y
y
def? c-usez
defz c-usez
8
9
74def-use graph exercise
Draw a def-use graph for the following program.
0. int set_z(x,y) 1. int x,y 2. if
(x!0) 3. y5 4. else zz-x 5. if
(zgt1) 6. zz/x 7. else 8. zy
75def-use graph (contd.)
- Traverse the graph to determine dcu and dpu sets.
76Test generation
- Exercises
- For the above graph generate a test set that
satisfies - the branch coverage criterion
- the all-defs criterion - for definitions of all
variables at least one use (c- or p- use) must be
exercised. - the all-uses criterion- all p-uses and all c-uses
of all variable definitions be covered.
- Develop the tests incrementally, i.e. by
modifying the previous test set!
77?SUDS processing Phase I
P, Program under test
78?ATAC processing phase II
coverage analyzer
79Mutation Testing
- What is mutation testing?
- Mutation testing is a code-based test assessment
and improvement technique.
- It relies on the competent programmer hypothesis
which is the following assumption
- Given a specification a programmer develops a
program that is either correct or differs from
the correct program by a combination of simple
errors.
80Mutation testing (contd.)
- The process of program development is considered
as iterative whereby an initial version of the
program is refined by making simple, or a
combination of simple changes, towards the final
version.
81Mutant
- Given a program P, a mutant of P is obtained by
making a simple change in P.
What is zpush?
82Another mutant
83Mutant
- A mutant M is considered distinguished by a test
case t ?T iff - P(t)?M(t)
- where P(t) and M(t) denote, respectively, the
observed behavior of P and M when executed on
test input t.
- A mutant M is considered equivalent to P iff
- P(t)?M(t) ?t ? T.
84Mutation score
- During testing a mutant is considered live if it
has not been distinguished or proven equivalent.
- Suppose that a total of M mutants are generated
for program P.
- The mutation score of a test set T, designed to
test P, is computed as - number of live mutants/(M-number of equivalent
mutants)
85Test adequacy criterion
- A test T is considered adequate w.r.t. the
mutation criterion if its mutation score is 1.
- The number of mutants generated depends on P and
the mutant operators applied on P.
- A mutant operator is a rule that when applied to
the program under test generates zero or more
mutants.
86Mutant Operators
- Consider the following program
- int abs (x)
- int x
-
- if (xgt0) x0-x
- return x
-
87Mutation operator
- Consider the following rule
- Replace each relational operator in P by all
possible relational operators excluding the one
that is being replaced.
- Assuming the set of relational operators to be
lt, gt, lt, gt, , !, the above mutant
operator will generate a total of 5 mutants of P.
88Mutation Operators
- Mutation operators are language dependent.
- For Fortran a total of 22 operators were proposed.
- For C a total of 77 operators were proposed. None
have been proposed for C though most of the
operators for C are applicable to C programs.
89Equivalent mutant
- Consider the following program P
- int x,y,z
- scanf(x,y)
- if (xgt0)
- xx1 zx(y-1)
- else
- xx-1 zx(y-1)
- Here z is considered the output of P.
90Equivalent mutant (contd.)
- Now suppose that a mutant of P is obtained by
changing xx1 to xabs(x)1.
- This mutant is equivalent to P as no test case
can distinguish it from P.
91Mutation Testing Procedure
Given P and a test set T
1. Generate mutants
2. Compile P and the mutants
3. Execute P and the mutants on each test case.
4. Determine equivalent mutants..
5. Determine mutation score.
6. If mutation score is not 1 then improve the
test set and repeat from step 3.
92Mutation Testing Procedure (contd.)
- In practice the above procedure is implemented
incrementally.
- One applies a few selected mutant operators to P
and computes the mutation score w.r.t. to the
mutants generated.
- Once these mutants have been distinguished or
proven equivalent, another set of mutant
operators is applied.
93Mutation Testing Procedure
- This procedure is repeated until either all the
mutants have been exhausted or some external
condition forces testing to stop.
- We will not discuss the details of practical
application of mutation testing.
94Tools for Mutation Testing
- Mothra for Fortran, developed at Purdue, 1990
- Proteum for C, developed at the University of
Saõ Paulo at Saõ Carlos in Brazil.
95Uses of Mutation Testing
- Mutation testing is useful during integration
testing to check for integration errors.
- Only the variables that are in the interfaces of
the components being integrated are mutated. This
reduces the complexity of mutation testing.
96Summary
- Data flow criteria
- def, use, p-use, c-use, all-uses
97Summary (contd.)
- xSUDS, data flow testing tool.
- Mutation testing
- mutant, distinguishing a mutant, live mutant,
mutant score, competent programmer hypothesis.