Testing in the Small (aka Unit Testing, Class Testing)

1
Testing in the Small (aka Unit Testing, Class
Testing)
1209
2
Unit Testing

• Focus on the smallest units of code
• Functions
• Methods
• Subroutines
• Frequently done (at least partially) during code
  development by code developers
• Often the target of testing frameworks such as
  JUnit

3
Each component is

• Tested in isolation from the rest of the system
• Tested in a controlled environment
• Uses appropriately chosen input data
• Uses component-level design description as guide

4
During Unit Tests

• Data transformations across the module are tested
• Data structures are tested to ensure data
  integrity

5
Unit Test Procedures
Driver
Module to be tested
Results
Test cases
Stub
Stub
6
Two fundamental approaches

• Black box
• Based on specification
• Inner structure of test object is not considered
• White box
• Based on specification
• Inner structure of test object is the basis of
  test case selection

7
Usefulness

• Effectiveness of black box is similar to white
  box, but the mistakes found are different.
  (Hetzel 1976, Myers 1978)
• Examples of errors not detected by white box
  testing
• Lock up PC if keyboard used during disk IO
  (neglected to turn off interrupts in pre 84
  versions of DOS)
• Monitor destroyed if both color and monochrome
  monitors used to play PC games

8
Black Box Unit Testing
1209
9
What is black-box testing?

• Unit (code, module) seen as a black box
• No access to the internal or logical structure
• Determine if given input produces expected
  output.

Output
Input
10
Black Box Testing

• Test set is derived from requirements
• Goal is to cover the input space
• Lots of approaches to describing input space
• Equivalence Classes
• Boundary Value Analysis
• Decision Tables
• State Transitions
• Use Cases
• . . .

11
Advantage and Disadvantage

• Advantage it does not require access to the
  internal logic of a component
• Disadvantage in most real-world applications,
  impossible to test all possible inputs
• Need to define an efficient strategy to limit
  number of test cases

12
General Black Box Process

• Analyze requirements
• Select valid and invalid inputs
• Determine expected outputs
• Construct tests
• Run tests
• Compare actual outputs to expected outputs

13
Equivalence Relations

• Reflexive
• Symmetric
• Transitive

14
Equivalence Relations

• Reflexive
• Symmetric
• Transitive

What are these?
G
15
Equivalence Relations

• Reflexive
• ?x R(x,x)
• Symmetric
• ?x,y R(x,y) ? R(y,x)
• Transitive
• ?x,y,z R(x,y) ? R(y,z) ? R(x,z)

16
Equivalence classes Basic Strategy

• Partition the input into equivalence classes
• This is the tricky part
• Its an equivalence class if
• Every test using one element of the class tests
  the same thing that any other element of the
  class tests
• If an error is found with one element, it should
  be found with any element
• If an error is not found with some element, it is
  not found by any element
• Test a subset from each class

17
Basic strategy

• Example fact(n)
• if nlt0, or ngt200 error
• if 0ltnlt20, exact value
• if 20ltnlt200, approximate value within .1
• What classes can you see?

G
18
Basic strategy

• Example fact(n)
• if nlt0, or ngt200 error
• if 0ltnlt20, exact value
• if 20ltnlt200, approximate value within .1
• Obvious classes are nlt0, 0ltnlt20, 20ltnlt200, and
  200ltn.
• Might be some subclasses here, also, but this is
  a good start.

19
Simple Example

• Suppose you are building an airline reservation
  system. A traveler can be a child, an adult, or a
  senior.
• The price depends on the type of traveler.
• The seat reservation does not depend on the type
  of traveler.
• How many test cases can you identify for the
  reservation component and the billing component?

G
20
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification

21
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)

22
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes (one invalid, two
  invalid)

23
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes
• If a set of values is specified where each may be
  treated differently, create a class for each
  element of the set and one more for elements
  outside the set

24
Finding equivalence classes

• Identify restrictions for inputs and outputs in
  the specification
• If there is a continuous numerical domain, create
  one valid and two or three invalid classes
  (above, below, and NaN)
• If a number of values is required, create one
  valid and two invalid classes
• If a set of values is specified where each may be
  treated differently, create a class for each
  element of the set and one more for elements
  outside the set
• If there is a condition, create two classes, one
  satisfying and one not satisfying the condition

25
Boundary Values

• Programs that fail at interior elements of a
  class usually fail at the boundaries too.
• Test the boundaries.
• if it should work for 1-99, test 0, 1, 99, 100.
• if it works for A-Z, try _at_, A, Z, , a, and z
• The hard part is identifying boundaries

26
Hints

• If a domain is a restricted set, check the
  boundaries. e.g., D1,10, test 0, 1, 10, 11
• It may be possible to test the boundaries of
  outputs, also.
• For ordered sets, check the first and last
  elements
• For complex data structures, the empty list, full
  lists, the zero array, and the null pointer
  should be tested
• Extremely large data sets should be tested
• Check for off-by-one errors

27
More Hints

• Some boundaries are not obvious and may depend on
  the implementation (use gray box testing if
  needed)
• Numeric limits (e.g., test 255 and 256 for 8-bit
  values)
• Implementation limits (e.g., max array size)

28
Boundary Value Exercise

• Determine the boundary values for US Postal
  Service ZIP codes
• Determine the boundary values for a 15- character
  last name entry.

G
29
Decision Tables

• Construct a table (to help organize the testing)
• Identify each rule or condition in the system
  that depends on some input
• For each input to one of these rules, list the
  combinations of inputs and the expected results

30
Decision Table Example
Theater ticket prices are discounted for senior
citizens and students.
Test Case C1 Student C2 Senior Result Discount?
1 T T T
2 T F T
3 F T T
4 F F F
31
Pairwise Testing
32
Problem 1From Lee Copeland, A Practitioners
Guide to Software Test Design, Artech House
Publishers, 2004.

• A web site must operate correctly with different
  browsers IE 5, IE 6, and IE 7 Mozilla 1.1
  Opera 7 and FireFox 2, 3, and 4.
• It must work using RealPlayer, MediaPlayer, or no
  plugin.
• It needs to run under Windows ME, NT, 2000, XP,
  and Vista, and 7.0
• It needs to accept pages from IIS, Apache and
  WebLogic running on Windows NT, 2000, and Linux
  servers
• How many different configurations are there?

G
33
Problem 1

• A web site must operate correctly with different
  browsers IE 5, IE 6, and IE 7 Mozilla 1.1
  Opera 7 and FireFox 2, 3, and 4.
• It must work using RealPlayer, MediaPlayer, or no
  plugin.
• It needs to run under Windows ME, NT, 2000, XP,
  and Vista, and 7.0
• It needs to accept pages from IIS, Apache and
  WebLogic running on Windows NT, 2000, and Linux
  servers
• How many different configurations are there?
• 8 browsers x 3 plugins x 6 OS x 3 web servers x 3
  server OSs 1296 combinations

34
Problem 2
From Lee Copeland, A Practitioners Guide to
Software Test Design, Artech House Publishers,
2004.

• A bank is ready to test a data processing system
• Customer types
• Gold (i.e., normal)
• Platinum (i.e., important)
• Business
• Non profits
• Account types
• Checking
• Savings
• Mortgages
• Consumer loans
• Commercial loans
• States (with different rules) CA, NV, UT, ID,
  AZ, NM
• How many different configurations are there?

G
35
When given a large number of combinations
(options improve )

• Give up and dont test

36
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business

37
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases

38
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran

39
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create

40
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few

41
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few
• List all combinations and randomly choose some

42
When given a large number of combinations
(options improve )

• Give up and dont test
• Test all combinations miss targets, delay
  product launch, and go out of business
• Choose one or two cases
• Choose a few that the programmers already ran
• Choose the tests that are easy to create
• List all combinations and choose first few
• List all combinations and randomly choose some
• Magically choose a small subset with high
  probability of revealing defects

43
Orthogonal Arrays

• A 2-D array with the property
• All pairwise combinations occur in every pair of
  columns
• Example consider 3 variables (columns) with
  a,b, 1,2, and (a,ß)

1 2 3
1 1 a a
2 1 b ß
3 2 a ß
4 2 b a
44
Orthogonal Array Example
1 2 3
1 1 a a
2 1 b ß
3 2 a ß
4 2 b a
Look at each pair of columns (1 and 2),( 1 and
3), and (2 and 3) Does each of the 4 pairings
appear in each? (Yes, of course!)
45
Pairwise Testing Algorithm

1. Identify variables
2. Determine choices for each variable
3. Locate an orthogonal array
4. Map test cases to the orthogonal array
5. Construct tests

46
Example using Bank

• Identify variables
• Customers (4)
• Account types (5)
• States (6)
• Determine choices for each variable
• Customer types (Gold, Platinum, Business, Non
  Profit)
• Account types (Checking, Savings, Mortgages,
  Consumer loans, Commercial loans
• States (CA, NV, UT, ID, AZ, NM)
• Locate an orthogonal array
• Map test cases to the orthogonal array
• Construct tests

47
Locate an orthogonal array (look up on web)
1 1 1
1 2 2
1 3 3
1 4 4
2 1 2
2 2 1
2 3 4
2 4 3
3 1 3
3 2 4
3 3 1
3 4 2
4 1 4
4 2 3
4 3 2
4 4 1
5 5 1
5 1 2
5 2 3
5 3 4
6 5 2
6 1 1
6 2 4
6 3 3
1 5 3
2 5 4
3 5 1
4 5 2
5 4 1
6 4 2
48
https//controls.engin.umich.edu/wiki/index.php/De
sign_of_experiments_via_taguchi_methods_orthogona
l_arrays
49
Map Test Cases (30)
CA Check Gold
CA Save Plat
CA Mort Busi
CA Cons NonP
NV Check Plat
NV Save Gold
NV Mort NonP
NV Cons Busi
UT Check Busi
UT Save NonP
UT Mort Gold
UT Cons Plat
ID Check NonP
ID Save Busi
ID Mort Plat
ID Cons Gold
AZ Comm Gold
AZ Check Plat
AZ Save Busi
AZ Mort NonP
NM Comm Plat
NM Check Gold
NM Save NonP
NM Mort Busi
CA Comm Busi
NV Comm NonP
UT Comm Gold
ID Comm Plat
AZ Cons Gold
NM Cons Plat
50
Pairwise Summary

• When the combination is large, test all pairs,
  not all combinations
• Studies indicate that most defects are single or
  double-mode effects
• Can be found in pairing
• Few defects require more than two modes
• Orthogonal arrays have the pairwise property and
  can be found or generated

51
State Transition Testing
52
State Transition Testing

• Build STD of system or component
• Cover the STD
• Visit each state
• Trigger each event
• Exercises each transition
• Exercise each path
• Might not be possible

53
Example
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
Customer makes reservation and has limited time
to pay. May cancel at any time.
Groups How many tests to visit each state?
54
Example Each State
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed From start to final From
start to cancelled non-pay From start to Res
Made to Cancelled Cust
55
Example Each Event
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
3 tests needed From start to final From
start to cancelled non-pay From start to Res
Made to Cancelled Cust
56
Example Each Transition
Payment made
Ticket Printed
Paid
Res Made
Ticketed
Cancel
Cancel/refund
Cust Order/start timer
Time Expires
Cancelled Cust
Cancelled Non Pay
Ticket Delivered
Cancel/refund
Used
5 tests needed
57
Use Case Testing
58
Use Case Testing

• Use the use cases to specify test cases
• Use case specifies both normal, alternate, and
  exceptional operations
• Use cases may not have sufficient detail
• Beizer estimates that 30-40 of effort in testing
  transactions is generating the test data
• Budget for this!

59
White-Box Testing

• Pfleeger, S. Software Engineering Theory and
  Practice 2nd Edition. Prentice Hall, 2001.
• Ghezzi, C. et al., Fundamentals of Software
  Engineering. Prentice Hall, 2002.
• Pressman, R., Software Engineering A
  Practitioners Approach, Mc Graw Hill, 2005.
• Hutchinson, Marnie, Software Testing
  Fundamentals, Wiley, 2003.

0310
60
White Box Testing

• Also known as
• Glass box testing
• Structural testing
• Test set is derived from structure of code
• Code structure represented as a Control Flow
  Graph
• Goal is to cover the CFG

61
Control Flow Graphs

• Programs are made of three kinds of statements
• Sequence (i.e., series of statements)
• Condition (i.e., if statement)
• Iteration (i.e., while, for, repeat statements)

62
Control Flow Graphs

• Programs are made of three kinds of statements
• Sequence (i.e., series of statements)
• Condition (i.e., if statement)
• Iteration (i.e., while, for, repeat statements)
• CFG visual representation of flow of control.
• Node represents a sequence of statements with
  single entry and single exit
• Edge represents transfer of control from one node
  to another

63
Control Flow Graph (CFG)
n1
n1
n1
n3
Join
Join
Sequence
If-then-else
Iterative
If-then
64
Control Flow Graph (CFG)
n1
n1
n1
n3
Join
Join
Sequence
If-then-else
Iterative
If-then
When drawing CFG, ensure that there is one exit
include the join node if needed
65
Example 1 CFG
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
Draw CFG
66
Example 1 CFG
1
2
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
3
9
4
5
6
7
8
67
Example 1 CFG
1
1,2
2
3
3
9
9
4
4,5
1. read (result) 2. read (x,k) 3. while
result lt 0 then 4. ptr ? false 5. if x gt k
then 6. ptr ? true 7. x ? x 1 8.
result ? result 1 9. print result
5
6
6
Join
7
7,8
8
68
In Class Write CFGs

• Example 2
• if (a lt b) then
• while(a lt n)
• 3. a ? a 1
• 4. else
• 5. while(b lt n)
• 6. b ? b 1

• Example 3
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• if c gt 10 then
• c ? max
• else c ? min
• 7. else print Done

• Example 4
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• 6. c ? a b

69
Write a CFG

• Example 2
• 1. if (a lt b) then
• 2. while (a lt n)
• 3. a ? a 1
• 4. else
• 5. while (b lt n)
• 6. b ? b 1

70
Answers

• Example 3
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. if c gt 10 then
• 5. c ? max
• else c ? min
• 7. else print Done

71
Answers

• Example 4
• 1. read (a,b)
• 2. if (a ? 0 b ? 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• 6. c ? a b

72
Coverage

• Statement
• Branch
• Condition
• Path
• Def-use
• Others

73
Statement Coverage

• Every statement gets executed at least once
• Every node in the CFG gets visited at least once

74
Example (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
75
Examples number of paths neededfor Statement
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
76
Examples number of paths neededfor Statement
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
4
1
5
5
77
Branch coverage

• Every decision is made true and false
• Every edge in a CFG of the program gets traversed
  at least once
• Also known as
• Decision coverage
• All edge coverage
• Basis path coverage

78
Examples number of paths neededfor Branch
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
79
Examples number of paths neededfor Branch
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
1
5
5
80
Condition coverage

• Every complex condition is made true and false by
  every possible combination
• E.G., (x and y)
• x true, y true
• xfalse, ytrue
• x true, y false
• x false, y false
• There are lots of different types of condition
  coverage Condition, multiple condition,
  condition/decision, modified condition/decision
  (MCDC), Im only covering the simplest.
• Under most circumstances, achieving Condition
  achieves Branch.

81
Condition Coverage CFGs

• One way to determine the number of paths is to
  break the compound conditional into atomic
  conditionals
• Suppose you were writing the CFG for the assembly
  language implementation of the control construct
• If (A AND B) then
• C
• Endif
• (short circuit eval) (no short circuit eval)
• LD A LD A in general, lots of
• BZ endif LAND B code for A and B
• LD B BZ endif
• BZ endif JSR C
• JSR C endif nop
• endif nop

82
AND Condition
1

• 1. read (a,b)
• 2. if (a 0 b 0) then
• 3. c ? a b
• 4. else c ? a b
• paths
• 1, 2A,2B,3, J
• 1, 2A, 2B, 4, J
• 1, 2A, 4, J

2A
2B
4
3
Join
83
OR Condition
1

• 1. read (a,b)
• 2. if (a 0 b 0) then
• 3. c ? a b
• 4. while( c lt 100)
• 5. c ? a b
• Paths
• 1, 2A, 3, 4, 5, 4 J
• 1, 2A, 3, 4, J
• 1, 2A, 2B, 3, 4, 5, 4, J
• 1,2A, 2B, J

2A
3
2B
4
5
Join
84
Path

• A path is a sequence of statements
• A path is sequence of branches
• A path is a sequence of edges

85
Examples number of paths neededfor Path
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
5
86
Examples number of paths neededfor Path
Coverage? (from Hutchinson, Software Testing
Fundamentals, Wiley, 2003)
1
6
1
6
2
7
2
7
3
8
3
8
4
9
4
9
5
16
5
5
87
Path Coverage-1

• Every distinct path through code is executed at
  least once
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid
• Test Paths
• 1, 2, 3, 4, 5, J1, 7, 8, J2
• 1, 2, 3, 4, 5, J1, 7, 9, J2
• 1, 2, 3, 6, J1, 7, 8, J2,
• 1, 2, 3, 6, J1, 7, 9, J2

1,2,3
4,5
6
Join1
7
8
9
Join2
88
Cyclomatic Complexity

• Software metric for the logical complexity of a
  program.
• Defines the number of independent paths in the
  basis set of a program
• Provides the upper bound for the number of tests
  that must be conducted to ensure that all
  statements been have executed at least once
• For Edges (E) and Nodes (N)
• V(G) E N 2

89
Examples Complexity of CFGs
3,4
3,4
1
1
5
5
6
2
Join
Join
3
N3 E2 E-N2 2-32 1V(G)
N4 E4 E-N2 4-42 2V(G)
N3 E3 E-N2 3-32 2V(G)
N1 E0 E-N2 0-12 1V(G)
90
Example
1
2,3
6
4,5
8
7
9
10
11
91
In Class Compute the cyclomatic complexity
1,2
3
9
4,5
6
Join
7,8
92
Example 1 CFG
1,2
N7 E8 E-N2 8-72 3V(G)
3
9
4,5
6
Join
7,8
93
Example 2
N6 E8 E-N2 8-62 4V(G)
94
Answers
N7 E8 E-N2 8-72 3V(G)
95
Answers
N6 E7 E-N2 7-62 3V(G)
96
Independent Path Coverage

• Basis set does not yield minimal test set
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid
• Cyclomatic complexity 3
• Test Paths
• 1, 2, 3, 4, 5, J1, 7, 8, J2
• 1, 2, 3, 4, 5, J1, 7, 9, J2
• 1, 2, 3, 6, J1, 7, 8, J2,

1,2,3
4,5
6
Join1
7
8
9
Join2
97
Def-Use Coverage

• Def-use coverage every path from every
  definition of every variable to every use of that
  definition is exercised in some test.
• Example
• 1. read (x)
• 2. read (z)
• 3. if x ? 0 then begin
• 4. y ? x z
• 5. x ? z end
• 6. else print Invalid
• 7. if y gt 1 then
• 8. print y
• 9. else print Invalid

Def x, z Use x
1,2,3
4,5
Def y, x Use x, z
6
Use none
Join
7
Use y
8
Use none
9
Use y

• Test Path 1, 2, 3, 4, 5, 7, 8, J

Join
98
Strength of Coverage
Statement
Arrows point from weaker to stronger
coverage. Stronger coverage requires more test
cases.
Branch
Def-Use
Condition
Path
99
What paths dont tell you

• Timing errors
• Unanticipated error conditions
• User interface inconsistency (or anything else)
• Configuration errors
• Capacity errors