Unit Testing

1
Unit Testing

• Colin Potts
• CS 6300. Fall, 2001

2
Agenda

• Testing Definitions and objectives
• Unit Testing
• Black-box test case generation for units of code
• Clear-box/White-box/Glass-box testing guidelines
• Planning testing
• System Building and Integration Testing
• When can/should you stop testing?

3
Importance of Testing

• Testing is most effort-intensive activity in
  software projects
• But is still often stopped too early or curtailed
  in a late project
• Scope of testing
• Unit testing testing of individual modules
• Integration testing testing of assemblies of
  modules
• System testing is the limiting case

4
Questions on Testing

• What is testing?
• Seeing whether a program compiles?
• Reading a program carefully?
• Proving that a program works?
• What is the difference between testing and
  debugging?
• What gets tested?
• Entire system?
• Modules, one-by-one?

5
What you need for testing

• Something to test
• code stubs for code that calls
• Test cases
• concrete inputs and expected outputs
• Some way of providing test case inputs
• drivers (including CLI main program)
• An oracle for judging whether test cases pass
• usually the specification and your judgment

6
Terminology

• Failures faults
• Failure Undesirable difference between observed
  expected behavior
• Fault/defect Cause of failure
• Defect rate Number of defects per 1000 lines of
  code
• Testing vs. debugging
• Testing Systematically finding failures by
  running code
• Debugging Finding fixing defects in the code
• Black-box vs. white-box testing
• Do you have access to the code in producing test
  cases?
• Scaffolding software to support testing

7
Testing

• Purpose
• "Program testing can be used to show the presence
  of errors, but never to show their absence"
  (Dijkstra)
• Gain confidence in the system by failing to find
  defects
• Who tests?
• Programmers should only release code with which
  they are confident
• But programmers are predisposed to believe their
  programs are correct
• Test each others code or use independent
  organization

8
Agenda

• Testing Definitions and objectives
• Unit Testing
• Black-box test case generation for units of code
• Clear-box/White-box/Glass-box testing guidelines
• Planning testing
• System Building and Integration Testing
• When can/should you stop testing?

9
Test Cases
Format is a set of input conditions (may
include state information) and expected outputs.
10
Black box testing example

• You have to test a function that allegedly
  correctly categorizes triangles
• Function accepts triple of integers
• No need to test the type of the inputs
• Prepare a list of test cases
• Example input triples
• Expected return value

Equilateral
Isosceles
Scalene
Unfeasible
11
Requirement Without Necessary Errors (Myers)
A function that tells whether three numbers
produce an equilateral triangle (whose sides are
all equal), an isosceles triangle (containing
exactly two equal sides) or a scalene triangle (a
triangle which is neither equilateral nor
isosceles).
12
More Complete

• A function that tells whether a triplet of
  numbers produces
• (1) an equilateral triangle (whose sides are all
  greater than zero and equal), in which case it
  outputs E at the prompt, or
• (2) an isosceles triangle (whose sides are
  greater than zero, exactly two of which are
  equal, and which form a triangle), in which case
  it outputs I at the system, or
• (3) a scalene triangle (whose sides are all
  greater than zero, which form a triangle, and
  which is neither equilateral nor isosceles), in
  which case it outputs S at the prompt, or
• (4) no triangle, in which case it outputs N at
  the prompt.

13
Equivalence classes and boundary values

• Testing is a form of sampling
• Effectively unbounded state space to be tested by
  small number of test cases
• Equivalence classes define (we hope!) effectively
  equivalent test cases
• But they are always subject to judgment.

• Errors tend to occur at the boundaries of
  equivalence classes
• Supposedly equivalent test cases may be close to
  a misprogrammed equivalence class boundary
• Classic cases Divide by zero, off by one errors
• Error guessing is sampling of potential space
  of test cases by using knowledge of common errors

14
Test Subject and Scaffold

• Need code to test (subject) and code to support
  the testing activity (scaffolding)
• Two kinds of scaffolding
• Stubs
• What should a stub do for the subject to be
  tested adequately?
• Just return with a message? Always return the
  same value?
• Drivers
• Test data files are special case of drivers

15
Agenda

• Testing Definitions and objectives
• Unit Testing
• Black-box test case generation for units of code
• Clear-box/White-box/Glass-box testing guidelines
• Planning testing
• System Building and Integration Testing
• When can/should you stop testing?

16
Clear/white-box testing

• Clear box testing takes account of the structure
  of the program
• Not just i/o correspondences, as in black box
  testing
• Also called white box and glass box testing
• Coverage of clear box tests is proportion of
• Statements executed
• Branches followed
• Branch exit branch from a decision point
• Paths followed through the flow graph
• Data flow from definition to use
• At least one vs. all uses vs. types of use

17
Covering Every Statement is Not Sufficient (Myers)
Required program
ugt1 and v0
Yes
x x/u
No
u2 or xgt0
Yes
x
No
18
Covering Every Statement is Not Sufficient (Myers)

• Code attempt to implement flowchart
• if( (ugt1) (v0) ) (1)
• x x/u (2)
• if( (u2) (xgt3) ) (3)
• x (4)
• u2, v0 and x3
• executes every line (1) - (4)
• gives the correct output x 2.5
• However, line (3) is wrong

Required program
ugt1 and v0
Yes
x x/u
No
u2 or xgt0
Yes
x
No
19
Paths to be Checked
Parameter settings make sense?
N
Y
Set _name to defaultName"
Parameter name too long?
Y
N
Set _name to parameter
Truncate name
20
Paths to be Checked
Parameter settings make sense?
N
Y
Set _name to defaultName"
Parameter name too long?
Y
N
Decision Coverage
Set _name to parameter
Truncate name
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
21
Perform Method Testing (Humphrey) 1/2
One way to ...
1. Verify operation at normal parameter values
(a black box test based on the units
requirements) 2. Verify operation at limit
parameter values (black box) 3. Verify
operation outside parameter values (black box)
4. Ensure that all instructions execute
(statement coverage) 5. Check all paths,
including both sides of all branches (decision
coverage) 6. Check the use of all called
objects 7. Verify the handling of all data
structures 8. Verify the handling of all files
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
22
Perform Method Testing (Humphrey) 2/2
One way to ...
9. Check normal termination of all loops (part
of a correctness proof) 10. Check abnormal
termination of all loops 11. Check normal
termination of all recursions 12. Check abnormal
termination of all recursions 13. Verify the
handling of all error conditions 14. Check timing
and synchronization 15. Verify all hardware
dependencies
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
23
Unit Testing Summary

• Testing cannot show the absence of defects, just
  increase your confidence in their absence
• Unit testing ? pieces
• Other testing ? assemblies
• Black box input / output only
• White/glass box verifies processing
• Several ways
• Ensure completeness
• Test planning earlier / better
• helps clarify requirements
• Decide when to stop testing

Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
24
Agenda

• Testing Definitions and objectives
• Unit Testing
• Black-box test case generation for units of code
• Clear-box/White-box/Glass-box testing guidelines
• Planning testing
• System Building and Integration Testing
• When can/should you stop testing?

25
V-model of development process

• Q How when can you assess a design?
• Designs dont run
• LHS -gt RHS
• LHS synthesis (1st)
• RHS testing (last)
• LHS-gtRHS is test planning (as early as possible)
• Design (modules and interfaces) -gt Integration
  test plan
• Detailed design / module spec. -gt Unit test plan
  cases

Requirements analysis
Acceptance/ reqts validn.
Design
System test/ design review
Coding
Unit testing/ review
26
Terminology

• Integration Testing
• Testing of subsystems as a whole
• Units assumed to have been tested and passed
• Review
• Systematic examination of a design, usually by a
  team
• Inspection
• Special kind of review

27
Integration testing
Ctrl

• Principle
• Individual program units may work in isolation
• But may not work correctly when integrated
• Localization of defects
• Defect may be manifested not at source, but in a
  different program unit

O'
I'
P'
R
Q
J
failure here
M
defect here
28
Interface errors

• Interface misuse
• A calling component calls another component and
  makes an error in its use of its interface e.g.
  parameters in the wrong order
• Interface misunderstanding
• A calling component embeds assumptions about the
  behaviour of the called component which are
  incorrect
• Timing errors
• The called and the calling component operate at
  different speeds and out-of-date information is
  accessed

29
Testing strategies

• Testing strategies are ways of approaching the
  testing process
• Incremental testing of builds
• Usually combined in different parts of the system
• Examples
• Top-down testing
• Bottom-up testing
• Thread testing

30
"Big bang" testing
I'
Test sequence
P'
O'
J
Q
Design

• Errors are difficult to locate

31
Top-down testing

• Start with the high-levels of a system and work
  your way downwards
• Advantages
• Works well with top-down development
• Finds architectural errors early
• Disadvantage
• System relies on whats underneath
• May need too much infrastructure before testing
  is possible (tends toward big-bang integration)
• May be difficult to develop program stubs that
  youre confident simulate the infrastructure

32
Unit-Oriented Build 1
Module 1
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
33
Unit-Oriented Build 2
Module 1
2
3
4
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
34
Unit-Oriented Build 3
Module 1
2
3
4
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
35
Last Build
Module 1
2
3
4
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
36
Meeting Scheduler example
37
Meeting scheduler example
Dialog Mgr
Dialog Mgr
Notifier
Time Chooser
Notifier
Time Chooser
Calendar Mgr
Locate Person
Locate Person
Calendar Mgr
System Services
System Services
Dialog Mgr
Dialog Mgr
Notifier
Time Chooser
Notifier
Time Chooser
Locate Person
Calendar Mgr
Locate Person
Calendar Mgr
System Services
System Services
38
Bottom-up testing

• Start with the lower levels of the system and
  work upward
• Advantages
• Appropriate for object-oriented systems
• Or where infrastructure components are critical
• Disadvantages
• Needs test drivers to be implemented
• May not simulate eventual calling envt.
• Does not find major design problems until late in
  the process

39
Meeting scheduler example
Dialog Mgr
Dialog Mgr
Notifier
Time Chooser
Notifier
Time Chooser
Calendar Mgr
Locate Person
Locate Person
Calendar Mgr
System Services
System Services
Dialog Mgr
Dialog Mgr
Time Chooser
Notifier
Notifier
Time Chooser
Calendar Mgr
Locate Person
Calendar Mgr
Locate Person
System Services
System Services
40
Thread Testing

• Testing tests behavior, so why base it on
  structure?
• In top-down bottom-up integration, boxes
  modules arrows interfaces
• A thread is an execution trace that defines a set
  of interrelated modules
• boxes modules arrows interaction instances
• Build systems around major use cases

41
Thread testing

• Based on testing an operation which involves a
  sequence of processing steps which thread their
  way through the system
• Start with single event threads then go on to
  multiple event threads
• Requires traceability between design models
• Advantages
• Suitable for real-time and object-oriented
  systems
• Disadvantage
• Difficult to assess test adequacy, because of
  large number of event combinations

42
Relating Tests to Requirements Design
(2) Design GameCharacter Requirements An abstract
class with attribute name ...
(1) D-Requirements 3.2.EC.1.2 Qualities of
Encounter characters Every game character has the
same set of qualities. Each quality shall be a
non-negative floating point number with at least
one decimal of precision. . . .
. . . .
43
Relating Tests to Requirements Design
(2) Design GameCharacter Requirements An abstract
class with attribute name ...
(1) D-Requirements 3.2.EC.1.2 Qualities of
Encounter characters Every game character has the
same set of qualities. Each quality shall be a
non-negative floating point number with at least
one decimal of precision. . . .
Test this class ...
... against this requirement
Characters
GameCharacter
Test this method ...
Encounter Characters
... against this requirement
EncounterCharacter adjustQuality()
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
44
Partitioning of Range for Unit Testing 1 of 2
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
45
Partitioning of Range for Unit Testing 2 of 2
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
46
Meeting scheduler example
Dialog Mgr
Dialog Mgr
1
Notifier
Time Chooser
Notifier
Time Chooser
Locate Person
Calendar Mgr
Calendar Mgr
Locate Person
System Services
System Services
Dialog Mgr
Dialog Mgr
Time Chooser
Notifier
2
Notifier
Time Chooser
Calendar Mgr
Locate Person
Calendar Mgr
Locate Person
System Services
System Services
47
Typical Build 1
Module 1
2
3
4
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
48
Typical Build 2
Module 1
2
3
4
Adapted from Software Engineering An
Object-Oriented Perspective by Eric J. Braude
(Wiley 2001), with permission.
49
Agenda

• Testing Definitions and objectives
• Unit Testing
• Black-box test case generation for units of code
• Clear-box/White-box/Glass-box testing guidelines
• Planning testing
• System Building and Integration Testing
• When can/should you stop testing?

50
Quality of test cases

• Error density is not usually constant
• Pareto (80/20) rule
• 80 defects originate in 20 of units/places in a
  unit
• Most errors are local
• Reqts. design errors are disproportiately
  expensive because of delocalized effects in code
• But most programming errors require changes to
  one unit
• Typical defect rates (Microsoft applications)
• 10-20 defects/KLOC during unit testing
• 0.5 defects/KLOC after release

51
Mutation testing

• How does a manager measure the adequacy of the
  test cases?
• E.g. to decide when to release?
• (1) Track error detection and release when below
  a threshold
• (2) Mutation testing
• Deliberately introduce bugs
• Measure how many of those bugs are found
• Use statistical reasoning to predict how many
  other bugs remain in the code
• Assumptions
• Injected bugs and residual bugs are similar
• Bugs are equivalent in effect

52
Stopping Testing
Assuming that defects detected are
representative S/N s/n
(total)
(detected)
A
B
size(A) a size (B) b size (A B) q size (A
v B) n effectiveness(A) a/N
q/b effectiveness (B) b/N q/a gt N
q/effectiveness(A)effectiveness(B)
defects in 1000 LOC
Example a 25 b 30 q 15 N ? (N-n)/N
? Would you release?