White-Box Testing

1
White-Box Testing
( FÅP First-year Project Course, ITU, Denmark )

• Claus Brabrand
• brabrand_at_itu.dk

2
Outline

• Motivation
• Why bother with testing?
• What is Testing?
• Relation to other programming-related tasks?
• White-box Testing
• Methodology coverage testing
• Automation
• How to automate testing
• Exercises
• Training for the exam

3
Learning Exam Goals

• Product
• Oral Exam

Today, well train for the exam -)
4
Software Errors

• Mobile Phones
• 00-
• Freeze and odd behaviors (really annoying)!
• Cruise Control System Model
• 86 (Grady Booch)
• Accellerated after car ignition ? car crashes
• Baggage Handling System
• 94-95 (at Denver Intl Airport)
• 360,000,000 USD

5
Software Errors (contd)

• Train Control System
• 98 (Berlin)
• Train cancellations
• Mars Pathfinder
• July 97
• Periodic resets
• Win95/98 w/ 3rd-Party Device Drivers
• late 90es
• Dysfunction (blue screen of death)!

...on mars!
6
Software Errors (contd2)

• Therac-25 Radiation Therapy
• 85-87
• Massive overdoses (6 deaths / amputations)!
• Patriot Missile Guidance System
• 91 (Gulf War 1.0)
• Accumulating rounding errors ? deaths
• Ariane V
• 96 (one of the most expensive bugs, ever)
• Conversion from 64-bit float to 16-bit signed int

7
and what about?!

• Surgical Laser Control System
• Oops!
• Air Plane Control System
• Dysfunction (plane crash)!
• Nuclear Powerplant Control System
• Core melt-down (China-syndrome)!

8
Outline

• Motivation
• Why bother with testing?
• What is Testing?
• Relation to other programming-related tasks?
• White-box Testing
• Methodology coverage testing
• Automation
• How to automate testing
• Exercises
• Training for the exam

9
Errors! (different kinds)

• Syntactic errors
• Mal-formed program
• Semantic errors
• Symbol errors
• Type errors
• Other semantic errors(e.g. uninitialized vars)
• Logical errors
• Compiler no errors

int square(int x) return xx
syntax error at line 2 expected
int square(int x) return nn
symbol error at line 2 undefined variable
n
int square(float x) return xx
type error at line 2 function returns
float, not int
int square(int x) return xx
no errors found!!!
10
Intl standard for evaluation of Software Quality
ISO 9126
Does the SW do what its supposed to? Does it
work as intended?
How much time/memory/space/- bandwidth/ does
the SW consume?
Functionality
Efficiency
Usability
Reliability
How robust is the SW wrt. incorrect inputs,
C, external netwk failures, ...?
How easy is the SW to understand? and use?
Portability
Maintainability
How easy is it to transfer and adapt SW to new
environment / platform?
How easy is it to modify the SW? And fix errors?
11
Testing vs. Debugging?

• Testing vs. Debugging

Functionality
Efficiency
Regarding
Purpose
Quality Assurance
(Functionality) Testing
(Performance) Testing
Diagnosis
Profiling
Debugging
12
Testing vs. Debugging (contd)
Program
01101021
SYSTEMATIC
(confidence?!?)
Re-
Fix problem (reprogram)
Evaluate test results
01101011
?
?
01101011
Document test results
(greater confidence!)
13
Performance Testing vs. Profiling
Program
01101021
SYSTEMATIC
(confidence?!?)
Re-
Improve program (reprogram)
Evaluate test results
01101011
?
?
01101011
Document test results
(greater confidence!)
14
Testing

Testing is easy (e.g., random
experimentation)

• Testing well is not easy
• Requires SYSTEMATIC approach test-case
• production
• evaluation
• documentation

Testing can never prove error absence (i.e.,
testing is an incomplete process)
15
Appropriate Test Sampling?

• Representative?
• Comprehensive?
• Quality?
• Quantity?
• ?

?
?
?
?
16
White-box vs. Black-box Test

• White-box Testing
• (aka., structural testing)
• (aka., internal testing)
• Test focus
• source code

• Black-box Testing
• (aka., behavioral testing)
• (aka., external testing)
• Test focus
• specification (manual)

Complementary Approaches!!!
17
Software Testing (R. Patton)

• Background reading
• Software Testing, Ron Patton, Sams Publishing,
  2006
• Part II (pp. 53 123) Testing Fundamentals

-testing
-testing
Time
Type
Examining the Code (chapter 6)
Examining the Spec. (chapter 4)
Static (before runtime)
Testing w/Blinders On (chapter 5)
Testing w/X-ray Glasses (chapter 7)
Dynamic (at runtime)
18
Outline

• Motivation
• Why bother with testing?
• What is Testing?
• Relation to other programming-related tasks?
• White-box Testing
• Methodology coverage testing
• Automation
• How to automate testing
• Exercises
• Training for the exam

19
Test Coverage?

• Method coverage
• Does every method run (at least once)?
• Statement coverage
• Does every statement run (at least once)?
• Branch coverage
• Does every branch run (at least once)?
• Path coverage
• Does every path run (at least once)?

20
Statement coverage

• Branch coverage
• Does every branch run (at least once)?
• -Box Branch Coverage Testing is
• Efficient (fast) !
• Effective (thorough) !
• Good for complicated program logic(esp.
  initialization errors)

21
Control Structures

• Control Structures
• Statements (or Exprs) that affect flow of
  control
• if-else
• if

if ( Exp ) Stm1 else Stm2
syntax
semantics
The expression must be of type boolean if it
evaluates to true, Statement-1 is executed,
otherwise Statement-2 is executed.
if ( Exp ) Stm
syntax
semantics
The expression must be of type boolean if it
evaluates to true, the given statement is
executed, otherwise not.
22
Control Structures (contd)

• while
• for

while ( Exp ) Stm
syntax
semantics
The expression must be of type boolean if it
evaluates to false, the given statement is
skipped, otherwise it is executed and afterwards
the expression is evaluated again. If it is still
true, the statement is executed again. This is
continued until the expression evaluates to false.
for (Exp1 Exp2 Exp3) Stm
syntax
semantics
Equivalent to
Exp1 while ( Exp2 ) Stm Exp3
23
Stm/Branch Coverage Testing

• if
• TEST condition true and false
• if-else
• TEST condition true and false
• while
• TEST zero, one, more-than-one iterations in loop
• for
• TEST zero, one, more-than-one iterations in loop

24
Example 1
Choice points?
public static void main ( String args )
int mi, ma if (args.length 0)
System.out.println("No numbers") else
mi ma Integer.parseInt(args0)
for (int i1 i lt args.length i)
int obs Integer.parseInt(argsi)
if (obs gt ma) ma obs
else if (mi lt obs) mi obs
System.out.println(min" mi
"," "max"
ma)
/ 1if-else /
if
else
/ 2for /
for
/ 3if-else /
if
else
/ 4if /
if
25
Control-Flow Graph
int mi, ma

• CFG

1
args.length 0
true
false
System.out.println("No numbers")
mi ma Integer.parseInt(args0)
int i1
2
i lt args.length
true
false
int obs Integer.parseInt(argsi)
3
obs gt ma
true
false
4
ma obs
mi lt obs
true
false
mi obs
i
System.out.println(min" mi "," "max"
ma)
26
Coverage Table

• Coverage Table

Data set A B B C E C D E (3rd num) E (2nd num)

• Input property
• No numbers
• At least one number
• Exactly one number
• Exactly two numbers
• At least three numbers
• N gt current max
• N ? current max
• N ? cur max N gt cur min
• N ? cur max N ? cur min

Choice 1ife true false 2for zero-times
once more-than-once 3ife true false 4if
true false
27
Expectancy Table

• Expectancy Table

Data set A B C D E
Input ? 17 27,29 39,37 49,47,48
Expected output no numbers min17,max17 min
27,max29 min37,max39 min47,max49
Actual output no numbers min17,max17 min27
,max29 min39,max39 min49,max49

?
?
?
?
?
Advice Avoid expected 0s (i.e.,
zeroes) (Default value in many languages.)
Advice Avoid reusing same numbers in tests (Data
layout sometimes reuse old memory.)
28
Debugging D then reveals
public static void main ( String args )
int mi, ma if (args.length 0)
System.out.println("No numbers") else
mi ma Integer.parseInt(args0)
for (int i1 i lt args.length i)
int obs Integer.parseInt(argsi)
if (obs gt ma) ma obs
else if (mi lt obs) mi obs
System.out.println(min" mi
"," "max"
ma)
/ 1if-else /
if
else
/ 2for /
for
/ 3if-else /
if
else
/ 4if /
if
Should have been
(obs lt mi)
29
Re-Test !

• as debugging oftenintroduces new errors !

Fixed Program
Coverage Table
Expectancy Table
?
Recall no guarantee!
30
Example 2
public static void main ( String args )
int mi1 0, mi2 0 if (args.length 0)
/ 1if-else /
System.out.println("No numbers") else
mi1 Integer.parseInt(args0) if
(args.length 1) /
2if-else / System.out.println("Smalle
st " mi1) else int obs
Integer.parseInt(args1) if (obs
lt mi1) / 3if
/ mi2 mi1 mi1 obs
for (int i 2 i lt args.length i)
/ 4for / obs
Integer.parseInt(argsi) if
(obs lt mi1) / 5if-else
/ mi2 mi1 mi1 obs
else if (obs lt
mi2) / 6if /
mi2 obs
System.out.println("The two smallest are "
mi1 " and "
mi2)
31
Coverage Table (Ex. 2)
Choice Input property
Data set 1ife true No numbers 1ife
false At least one number 2ife true
Exactly one number 2ife false At
least two numbers 3if true 2nd number
1st number 3if false 2nd number lt
1st number 4for zero-times Exactly two
numbers 4for once Exactly three
numbers 4for more-than-once At least four
numbers 5ife true 3rd number lt current
min 5ife false 3rd number current
min 6if true 3rd cur min 3rd lt
2nd least 6if false 3rd cur min
3rd 2nd least
A B B C C D D E H E F F G
32
Expectancy Table (Ex. 2)
Data set Input Expected output
Actual output A ? no
numbers no numbers B 17
17 17 C 27,29
27 and 29 27 and 0 D 39,37
37 and 39 37 and 39 E
49,48,47 47 and 48 47 and 48
F 59,57,58 57 and 58 57 and
58 G 67,68,69 67 and 68
67 and 0 H 77,78,79,76 76 and 77
76 and 77
?
?
?
?
?
?
?
?

• Debugging reveals that variablemi2 erroneously
  retains initialization (0).

33
Debugging (Ex. 2)
public static void main ( String args )
int mi1 0, mi2 0 if (args.length 0)
/ 1if-else /
System.out.println("No numbers") else
mi1 Integer.parseInt(args0) if
(args.length 1) /
2if-else / System.out.println("Smalle
st " mi1) else int obs
Integer.parseInt(args1) if (obs
lt mi1) / 3if
/ mi2 mi1 mi1 obs
for (int i 2 i lt args.length i)
/ 4for / obs
Integer.parseInt(argsi) if
(obs lt mi1) / 5if-else
/ mi2 mi1 mi1 obs
else if (obs lt
mi2) / 6if /
mi2 obs
System.out.println("The two smallest are "
mi1 " and "
mi2)
mi2 obs
Re-Test ?
34
Control Structures (contd2)
Swb

• switch
• do-while
• ? conditional expression
• lazy conjunction (aka., short-cut ?)
• lazy disjunction (aka., short-cut ?)

case Exp Stm break
switch ( Exp ) Swb
default Stm break
Choice points?
do Stm while ( Exp )
Exp1 ? Exp2 Exp3
Exp1 Exp2
Exp1 Exp2
35
Control Structures (contd3)

• try-catch-finally (exceptions)
• return / break / continue
• method invocation
• e.g.
• recursive method invocation
• e.g.
• virtual dispatching
• e.g.

try Stm1 catch ( Exp ) Stm2 finally Stm3
return
return Exp
break
continue
f(x)
f(x)
f(x)
36
Outline

• Motivation
• Why bother with testing?
• What is Testing?
• Relation to other programming-related tasks?
• White-box Testing
• Methodology coverage testing
• Automation
• How to automate testing
• Exercises
• Training for the exam

37
Test Automation

• (Re-)Running tests is boooring ( error prone)
• Thus, automate them once-and-for-all
• JUnit
• Can be run from Eclipse/JUnit
• (if appropriately subclassing TestCase)

public class MyTestCase extends TestCase
/ ...other tests... /
_at_Test // Testing if 326 public void
testMultiplication() assertEquals("Multi
plication", 6, 32)
38
Outline

• Motivation
• Why bother with testing?
• What is Testing?
• Relation to other programming-related tasks?
• White-box Testing
• Methodology coverage testing
• Automation
• How to automate testing
• Exercises
• Training for the exam

39
Exercise
Warm up exercise Draw a control-flow diagram
for the do-while construction

• Part I
• Part II

• Program merge (in Java)
• Test your merge method
• Label choice points
• Build coverage table make data set (test
  suite)
• Build expectancy table
• Run test suite (upon failure fix and retest
  program)
• Introduce subtle bug
• Run test to document presence of bug
• Submit erroneous program to class program pool

ListltIntegergt merge(ListltIntegergt list1,
ListltIntegergt list2)
(produce)

• Pick erroneous program from class program pool
• Test merge program (and debug to find bug)
• Re-Test fixed merge program
• Write report (and send it to the teaching
  assistant)

(consume)
40
Specification (merge)

• Interface (for merge)
• I/O assumptions
• Input both lists are sorted
• and, numbers occur maximum once (in each list)
• Output list must be sorted
• and, numbers occur maximum once
• Programming constraints
• no recursion !
• no java.util.Iterators !

ListltIntegergt merge(ListltIntegergt list1,
ListltIntegergt list2)
41
The Testing Report

• Report (ca. 3 pages)
• Must explain how you tested, (debugged),
  re-testedthe erroneous merge (it must include,
  at least)
• i) The erroneous program
• ii) a Control-Flow Graph for the program
  hand-drawn ok(incl. labelled choice points)
• iii) Coverage Tables
• iv) Expectancy Tables
• Submit the testing report to the T.A. (Anders)
• (deadline Tuesday March, 11 at 0900 CET)
• Note the report is only on Part II
• (i.e., the program you didnt write)

42
Example 1
public static void main ( String args )
int mi, ma if (args.length 0)
System.out.println("No numbers") else
mi ma Integer.parseInt(args0)
for (int i1 i lt args.length i)
int obs Integer.parseInt(argsi)
if (obs gt ma) ma obs
else if (mi lt obs) mi obs
System.out.println(min" mi
"," "max"
ma)