Software Testing

1
Software Testing

• Sudipto Ghosh
• CS 406 Fall 99
• November 16, 1999

2
Learning Objectives

• Data flow-based testing
• Test adequacy assessment

3
Problems with path testing

• With each predicate, there is a combinatorial
  explosion in the number of paths.
• Potentially infinite number of paths because of
  loops.
• Not all paths a feasible, i.e., there may not be
  inputs for which the path is executed.
• Suppose that somehow, we satisfied this
  criterion. Would we find all the faults?

4
More problems

• A path is tested only if it is present!!
• Consider the example
• double logBase19 (double x)
• return ln(x)/ln(19)
• Missing condition is
• if(x gt 0)

5
More problems

• It is possible to test every path without
  detecting the fault in the product.
• Function to test the equality of 3 integers
• (Fallacious) assumption
• If the average of the 3 numbers is equal to the
  first, then they are equal.
• boolean areEqual(int x,int y,int z)
• if ((xyz)/3 x) return TRUE
• else return FALSE
• Exercise Think of test cases.

6
Reducing the number of paths to be covered

• Find something between branch and path coverage.
• Select a set of paths that ensure branch coverage
  and try some other paths that help reveal errors.
• Consider two paths same if they differ only in
  their sub-paths that are caused due to loops.
• Restrict test cases to linear code sequences
• Identify set of points L, from which control flow
  may jump includes entry, exit, branch statements
• Paths begin and end at elements in L.

7
Data Flow-Based Testing

• Select paths to be executed based on data flow
  analysis
• Information about where
• variables are defined
• variables are used
• Idea is to make sure that the definitions of
  variables and their subsequent use is tested

8
Variables in statements

• Variables occur in statements as
• Definition def
• Variables on the left side of a statement ( c )
• c 10
• Variable is used in a computation c-use
• Variables on the right-hand side of a statement (
  c )
• temp c 5
• Variables used in a write statement
• Variable is used in a predicate p-use
• if( c! MAXLENGTH )

9
An example (xy)

• scanf(x, y) if(y lt 0)
• pow 0 y
• else pow y
• z 1.0
• while(pow ! 0)
• z z x pow pow 1
• if ( y lt 0 )
• z 1.0/z
• printf(z)

10
Def/use graph of example
def x, y c-use
1
y
y
3
def pow c-use y
2
def pow c-use y
4
def z c-use
5
def c-use
pow
pow
def z, pow c-use x, z, pow
6
7
def c-use
y
y
9
def z c-use z
def c-use z
8
11
More on def/use graphs

• With each node, we associate the c-uses of a
  variable
• With each edge, we associate the p-uses of a
  variable

12
Def-clear paths

• Def-clear path with respect to a variable x
• Path from node i to node j, such that there is no
  def of x in the nodes in the path from i to j.
  Nodes i and j may have a def.
• Find out examples from the previous graph.
• Path from node i to edge (j, k), such that there
  is no def of x in the nodes in the path.
• Find out examples from the previous graph.

13
Global c-use

• A c-use of a variable is considered global if
• there is not def of x within the block preceding
  the c-use
• Example?

14
Global def

• A def is global if it can be used outside the
  block in which it is defined.
• A def of a variable x, in a node i, is a global
  def, if
• it is the last def of x in the block (node) i
• a def-clear path exists from i to another node
  which has a global c-use of x
• Example?

15
Set def(i)

• Set of variables for which there is a global def
  in the node i.
• Examples

16
Set c-use(i)

• Set of variables for which there is a global def
  in the node i.
• Examples

17
Set p-use(i, j)

• For an edge (i, j), the set p-use(i, j), is the
  set of variables for which there is a p-use for
  the edge (i, j).
• Examples

18
Set dcu(x, i)

• dcu(x, i) represents all those nodes in which the
  global c-use of x uses the value assigned by the
  def of x in i.
• Suppose a variable x is in def(i) of node i.
• Set of nodes, such that
• Each node has x in its c-use
• There is a def-clear path from i to j
• Examples

19
Set dpu(x, i)

• dpu(x, i) represents all those edges in which the
  p-use of x uses the value assigned by the def of
  x in i.
• Suppose a variable x is in def(i) of node i.
• Set of edges, such that
• each edge has x in its p-use
• There is a def-clear path from i to (j, k)

20
Test case selection criteria

• Let G be the def-use graph for a program
• Let P be a set of all the complete paths of G
  that were executed during the test.
• Examples

21
All-defs criterion

• Make sure that the definitions of all variables
  are tested.
• For every def of every variable, one of its uses
  (either p-use or c-use) must be included in a
  path.
• For every node i in G and every x in def(i), P
  includes a def-clear path w.r.t. x to some member
  of dcu(x, i)

22
All-p-uses criterion

• All p-uses of all definitions should be tested.
• For every x ? def (i), P includes a def-clear
  path w.r.t. x from i to all members of dpu(x, i).
• Note that a c-use may not be tested
• Can require
• all p-uses, some c-uses criterion

23
All c-uses criterion

• All c-uses of all definitions should be tested.
• For every x ? def (i), P includes a def-clear
  path w.r.t. x from i to all members of dcu(x, i).
• Note that a p-use may not be tested
• Can require
• all c-uses, some p-uses criterion

24
All-uses criterion

• All p-uses and all c-uses of a definition must be
  exercised.
• The set P must include,
• for every node i and every x ? def(i)
• a def-clear path w.r.t. x from i to all elements
  of dcu(x, i) and to all elements of dpu(x, i).

25
Number of test cases

• Can be shown theoretically that the limit of the
  size of test set is up to quadratic in the number
  of two-way decision statements in the program.
• Empirical studies have shown that the actual
  number of test cases that satisfy a criterion is
  quite small and increases linearly with the
  number of two-way decisions.

26
Inclusion relationship between criteria
all-paths (path coverage)
all-uses
all-c-uses/ some-p-uses
all-p-uses/ some-c-uses
all-defs
all-p-uses
all-edges (branch coverage)
all-nodes (statement coverage)
27
Implication of inclusion

• Suppose C1 includes C2 (C1 ? C2), I.e test cases
  satisfying C1 also satisfy C2
• Does it mean that C1 is always better than C2?
• Statistically, the set of test cases satisfying
  C1 will be better than those satisfying C2.
• Testing done using these criteria performs better
  than randomly selected test cases.

28
Test assessment

• Develop
• a test set T
• a collection of test inputs
• Question
• How good is T?
• Test assessment is the measurement of the
  goodness of T.
• Test assessment is carried out based on one or
  more criteria.

29
Test adequacy assessment

• These criteria are known as test adequacy
  criteria.
• Test assessment is also known as test adequacy
  assessment.

30
Test adequacy assessment (contd)

• Test adequacy assessment provides the following
  information
• A metric, also known as adequacy score, or
  coverage, usually between 0 and 1.
• A list of all the weaknesses found in T, which
  when removed, will raise the score to 1.
• The weaknesses depend on the criteria used for
  assessment.

31
Test adequacy assessment (contd)

• Once the coverage has been computed, and the
  weaknesses identified, one can improve T.
• Improvement of T is done by examining one or more
  weaknesses and constructing new test requirements
  designed to overcome the weaknesses.

32
Test adequacy assessment (contd)

• This is continued until all weaknesses are
  overcome, i.e., the adequacy criterion is
  satisfied (coverage 1)
• In some instances, it may not be possible to
  satisfy the adequacy criteria for one or more of
  the following reasons
• Lack of sufficient resources (people, time)
• Weaknesses that cannot be removed because they
  are infeasible
• The cost of removing the weakness is not justified

33
Test adequacy assessment (contd)

• While improving T by removing its weaknesses, one
  usually tests the program more thoroughly than it
  has been tested so far.
• This additional testing is likely to result in
  the discovery of remaining errors.
• Hence, we say that test assessment and
  improvement helps in the improvement of software
  reliability.

34
Test adequacy assessment - summary procedure
0. Develop T
1. Select an
adequacy criterion C
2. Measure adequacy
criterion of T w.r.t C
3. Is T adequate?
Yes
No
Yes
4. Improve T
5. More testing is
warranted?
No
6. DONE!!