Test Coverage

1
Test Coverage

• CS-300 Fall 2005
• Supreeth Venkataraman

2
Test Coverage

• Coverage is a measure of how much testing can be
  done given a set of test cases.
• Given a set of test cases, how much of the code
  in the program being tested is covered?
• Or.. Given a set of test cases how many
  functionalities as listed in the requirements can
  be tested?
• Coverage in most cases is a term associated with
  white-box testing or structural testing.

3
Black Box Testing

• Black-box testing involves deriving test cases
  based on the functional requirements of the
  program being tested.
• Does the software exhibit the intended behavior
  for each of its functionalities?
• Black-box testing can be used to detect improper
  functionality, errors of interface, errors in
  data structures, and initialization and
  termination errors with little regard for the
  logical structure of the program. Pressman 97

4
Black Box Testing

• We have already seen many types of blackbox
  testing
• Fault-Based testing, boundary-value testing, etc.
• Functional testing is thought of as being
  superior to structural testing in most cases
• However, it is acknowledged that code based
  testing does uncover a lot of defects.

5
Equivalence Partitioning

• A good test case is one that has a good
  likelihood of uncovering a fault.
• Exhaustive testing is impossible due to the
  infinite domain of inputs.
• We must choose a suitable subset of inputs so
  that it covers a large part of other possible
  test cases Myers, p.45

6
Equivalence Partitioning

• Partition the input domain into equivalence
  classes
• The goal here is to partition the inputs in such
  a way that if one test case in an equivalence
  class is capable of detecting some fault, then
  all test cases in that class are capable of
  detecting the same fault.

7
Equivalence Partitioning

• An example of an equivalence class for E-LIB.
• A set of strings that obey the rules of structure
  of the SSN.
• While checking for the structure of the SSN, if
  one test case from our class does not uncover a
  fault, it is reasonable to assume that none of
  the strings in the class will reveal a fault.

8
White Box Testing

• White-box testing is also known as structural
  testing or code-based testing.
• The test cases to be used to test the program are
  derived from the program structure, and not from
  functional specifications.
• Many many variations of white-box testing exist.

9
White Box Testing and criticisms

• A lot of research has been undertaken to discover
  the best kind of code coverage.
• All testing that does not derive from functions
  is fundamentally misguided
• People do not use software in order to execute
  its statements rather they invoke its
  functionality

10
White box or Black box?

• Achieving code coverage does not imply that good
  functional coverage is achieved as well.
• A good mix of functional and structural testing
  is desirable as both functional as well as code
  coverage are important.
• Structural test cases must wait to be derived
  until the code is written, but functional tests
  can be derived from requirements.
• Weyuker proposed that structural testing be used
  as a way of evaluating functional test quality.

11
White box or Black Box?

• In many cases, structural coverage is done after
  adequate functional coverage is achieved.
• Structural testing is used to back up functional
  testing
• If structural testing is inadequate, create
  additional functional test cases.

12
Test Adequacy

• The term adequacy can be thought of as asking
  oneself the question How many test cases do I
  need in order to completely cover the entire
  program based on the technique that I have
  chosen for testing?
• Or.When do I stop testing?

13
Control Flow Testing

• Control-flow testing is based on the flow of
  control in a program.
• There are many variations of control-flow
  testing. Statement coverage, branch coverage,
  path coverage etc.
• We will examine statement coverage and branch
  coverage in some detail.

14
The Program Flowgraph

• A program flowgraph graphically represents the
  potential control flow of a program.
• Program flowgraphs are made up of nodes and
  directed edges
• Each node represents a basic block, which can be
  defined as a collection of statements that are
  always executed together.
• Each directed edge between two nodes A and B in a
  control flow graph represents a potential control
  path from A to B

15
Example Program

• void myflow(int x,int y)
• int a
• if (y 0)
• a 1
• else
• a y
• y y 1
• if (a gt 0)
• a x
• else
• a x 1
• printf(d\n, a)

16

• void myflow(int x, int y)
• 2. int a
• if (y 0)
• a 1
• else
• a y
• y y 1
• if (a gt 0)
• a x
• else
• a x 1
• printf(d\n, a)

17
Statement Coverage

• Achieving statement coverage means that all
  statements in the program have to be executed at
  least once by a set of test cases
• Having complete statement coverage does not
  necessarily mean that the program cannot fail
• Think about the input space....
• Statement coverage is also called all-nodes
  testing.

18
Statement Coverage Test Cases

• Test case1 input x 7, y 0
• expected output a 7
• Test case2 input x 7, y -1
• expected output a 8

19
Branch Coverage

• Achieving branch coverage means that there must
  be enough test cases so that every path for a
  conditional transfer of control is exercised
• Or... Every true and false branch have to be
  exercised by some test case
• Complete coverage means that the test cases
  should be sufficient to traverse all the edges in
  the program graph.
• This form of testing is also known as all-edges
  testing .

20
Branch Coverage

• The fact that there is complete branch coverage
  does not mean that all errors will be found.
  Branch testing does not necessarily check all
  combinations of control transfers.

21
Branch Coverage Test Cases

• Test case1 input x 7, y 0
• expected output a 7
• Test case2 input x 7, y -1
• expected output a 8

22
Path Coverage

• Path coverage means that all possible execution
  paths in the program must be executed.
• This is a very strong coverage criterion, but
  impractical.
• Programs with loops have an infinite number of
  execution paths, and thus would need an infinite
  number of test cases

23
Data Flow Testing

• Data flow testing is based on the definition and
  use of variables in a program.
• Deals with flow of data rather than flow of
  control.
• There are many data-flow testing methods
• All-definitions coverage, all-uses coverage,
  all-du-paths coverage etc.
• We will look at all-uses testing in some detail

24
Terminology (Beizer)

• Definition a variable is said to be defined
  when it is initialized with a value as part of a
  declaration, or when it appears on the left hand
  side of an assignment statement .
• eg x 4
• int y 0

25
Terminology

• Use The value of a variable is said to be used
  either when the variable appears on the right
  hand side of an assignment statement or when it
  appears in a predicate statement like if (x gt y)
  (Beizer)
• Eg y x

26
Terminology

• Definition-clear path - This is a set of
  connected nodes in the flowgraph where some
  variable x is defined in the first node, and is
  not subsequently redefined in any other node in
  the path.
• For example, the path 2-3-4-7-8-9-12 is a
  definition-clear path with respect to variable
  x in figure 2.2.

27

• void myflow(int x, int y)
• 2. int a
• if (y 0)
• a 1
• else
• a y
• y y 1
• if (a gt 0)
• a x
• else
• a x 1
• printf(d\n, a)

28
All-Uses Testing

• The all-uses treatment states that there must be
  at least one definition-clear path exercised by
  some test case from every definition of a
  variable x to all of its uses.
• All-uses testing is a powerful test coverage
  criterion.

29
All-Uses test cases

• Test case1 input x 7, y 0
• expected output a 7
  
• Path 2-3-4-7-8-9
• Test case2 input x 7, y -1
• expected output a 8
  
• Path 2-3-6-7-8-9
• Test case3 input x 7, y 4
• expected output a 7
  
• Path 2-3-6-7-8-11

30
The Subsumes Relationship

• The subsumption relationship means that
  satisfying one test coverage criterion may
  implicitly force another test coverage criterion
  to be satisfied as well
• Eg Branch coverage forces statement coverage to
  be attained as well (This is strict subsumption)
• Subsumes does not necessarily mean better

31
Subsumption Hierarchy (Rapps and Weyuker)
32
Mutation Testing

• Mutation testing involves changing (mutating) the
  program source statement(s)
• If the actual output from a test case is
  different from the expected output (hopefully it
  will be), the test case is said to kill the
  mutant.
• Mutation coverage requires that all mutants be
  killed

33
Some Mutation Examples

• Operator mutation change the operator in the
  mutated version.
• Eg change x y to x y or x y
• Change x lt y to x lty or x gty
• If all the mutants are killed we are satisfied
  that no wrong operator has been mistakenly used.

34
Choosing a Testing Method

• The process
• If the development process is not well defined
  without a proper test plan one does not really
  know what type of testing to apply.
• If the project starts getting delayed, then
  testing time is extremely limited.
• Resources available
• testing time, testing experience, budget ...

35
Choosing a Testing Method

• Feasibility Issues
• Structural testing may not achieve complete code
  coverage because of dead code
• Dead code is code that never gets executed.
• There could be dead statements, dead paths etc.
• The tester must devote time to identifying
  infeasible code.

36
Conclusions

• In practice, there is no agreement as to what the
  payoffs are for each form of structural testing
• Functional testing has advantages over structural
  testing because functional tests capture the
  expected actions of users.