Software Testing and Reliability Functional Testing

1
Software Testing and ReliabilityFunctional
Testing

• Aditya P. Mathur
• Purdue University
• May 19-23, 2003
• _at_ Guidant Corporation
• Minneapolis/St Paul, MN

Last update April 17, 2003
2
Learning Objectives

• What is functional testing?

• How to perform functional testing?

• What are clues, test requirements, and test
  specifications?

• How to generate test inputs?

• What are equivalence partitioning, boundary value
  testing, domain testing, state testing, and
  decision table testing?

3
References

• The Craft of Software Testing, Brian Marick,
  Prentice Hall, 1995.

4
What is Functional Testing?

• When test inputs are generated using functional
  specifications, we say that we are doing
  functional testing.

• Functional testing tests how well a program meets
  the functionality requirements.

5
The Methodology

• The derivation of test inputs is based on
  functional specifications.

• Clues are obtained from the specifications.

• Clues lead to test requirements.

• Test requirements lead to test specifications.

• Test specifications are then used to actually
  execute the program under test.

6
Test Methodology
Specifications
Program output is correct
7
Specifications

• Inputs and tasks

• Given inputs

• Perform tasks

8
Specifications (contd.)

• Input properties

9
Specifications (contd.)

• Two types of pre-conditions are considered

• Validated those that are required to be
  validated by the program under test and an error
  action is required to be performed if the
  condition is not true.

• Assumed those that are assumed to be true and
  not checked by the program under test.

10
Specification Example

• For the sort program
• Inputs are

• N

• pointer to a sequence of length N

• pointer to an area in memory where the output
  sequence is to be placed.

11
Specification Example (contd.)

• Tasks to be performed

• Sort the sequence in ascending order

• Return 1 if sorting is successful, -1 otherwise.

• Return 1 if sorting is successful, -1 otherwise.

12
Preconditions for sort

• Validated
• Ngt0
• On failure return -1 sorting considered
  unsuccessful.

• Assumed
• The input sequence contains N integers.
• The output area has space for at least N integers.

13
Deriving pre-conditions

• Pre-conditions result from properties of inputs.

• Example
• alpha_sequence(name)
• alpha_sequence is the string obtained from name
  by removing all characters other then A-Z and
  a-z. Thus, if name is A12C then alpha_name is
  AC.

14
Deriving pre-conditions (contd.)

• This leads to the following pre-condition
• Validated the string alpha_sequence(name) is
  identical to or shorter than the string name.
• On failure print invalid name.

• This property could also lead to the
  pre-condition
• Assumed the string alpha_ sequence(name) is
  identical to or shorter than the string name.

15
Post-conditions

• A post-condition specifies a property of the
  output of a program.

• The general format of a post-condition is
• if condition then effect-1 else effect-2

• Example
• For the sort program a post-condition is
• if Ngt0 then the output sequence has the same
  elements as in the input sequence and in
  ascending order.

16
Post-condition (contd.)

• This could be stated more formally as
• if Ngt0 then
• and each is a member
  of the input sequence and sort returns 1.
• else
• the output sequence is undefined and sort
  returns -1.

17
Post-condition (contd.)

• Another example
• if (AB) and (BC) then return equilateral

• Can you complete the above post-condition for a
  program that is required to classify a triangle
  given the length of three sides?

Convention Do not nest if-then-else conditions
while specifying a post-condition.
18
Incompleteness of Specifications

• Specifications may be incomplete or ambiguous.

• Example post-condition
• if user places cursor on the name field then
  read a string
• This post-condition does not specify any limit on
  the length of the input string hence is
  incomplete.

19
Ambiguous Specifications

• It also does not make it clear as to
• whether a string should be input only after the
  user has placed the cursor on the name field and
  clicked the mouse or simply placed the cursor on
  the name field.
• and hence is ambiguous.

20
Clues Summary

• Clues are
• Pre-conditions

• Post-conditions

• Variables, e.g. A is a length implying thereby
  that its value cannot be negative.

• Operations, e.g. search a list of names or
  find the average of total scores

• Definitions, e.g. filename(name) is a name with
  no spaces.

21
Clues (contd.)

• Ideally, variables, operations and definitions
  should be a part of at least one pre- or
  post-condition.

• However, this may not be the case as
  specifications are not always written formally.

• Hence look out for variables, operations, and
  definitions within a specification!

22
Test Requirements

• A test requirement is a description of how to
  test the program that is under test.

• Here is a sample test requirement for a program
  that classifies a triangle given the length of
  three sides.

• A, B, C are non-zero and positive.
• One of A, B, C is negative error condition.
• One of A, B, C is zero error condition.

23
Test Requirements-Derivation

• Test requirements are derived from clues.

• For example, consider the following
  pre-conditions (clues)
• Assumed A, B, and C are lengths
• Validated Agt0, Bgt0, Cgt0

• These pre-conditions on A, B, and C lead to the
  test requirement given on the previous slide.

24
Test Requirements-Derivation

• Note that we have combined the pre-condition for
  each input variable into one condition. This is
  done only for inconvenience!

• It is recommended that pre-conditions be
  separated for each variable.

25
Test Requirements-Derivation

• Note also that each validated pre-condition
  results in at least two requirements one for the
  validated part and the other for the failure part.

• In our example above we did not list all
  requirements. For example, we are content with
  testing one of A, B, C is negative error
  condition.

26
Test Requirements-Derivation

• Post-conditions also lead to test requirements.

• For example, the partial post-condition
• if (AB) and (BC) then return equilateral

• leads to the following test requirement
• AB and BC.

27
Compound Validated Pre-conditions

• Compound pre-conditions are ones that use one or
  more and and or connectors in some combination.

• Examples validated compound pre-conditions
• Pre-condition A and B

• Pre-condition user places the mouse over the
  name field and clicks it.

28
Compound Validated Pre-conditions

• The first of the above pre-conditions leads to
  four requirements
• A true, B true (This is the validated part)
• A false, B true (This and the remaining are
  failures)
• A true, B false
• A false, B false

• You may work out the requirements for compound
  pre-condition with the or connector.

29
Compound Validated pre-conditions

• Compound validated pre-conditions could become
  quite complex.

• Example (A and (B or C))

• Brute force method will lead to 8 test
  requirements.

30
Compound Validated pre-conditions

• In general this will lead to too many test
  requirements.

• We can prune them by leaving out those
  requirements that are unlikely to reveal a
  program error.

• For example, consider the validated
  pre-condition A or B.

31
Pruning Test Requirements

• There are four possible test requirements
• A true, B true
• A false, B true
• A true, B false
• A false, B false

• Consider a correct C implementation
• if (!(A B))
• exit_with_error(Error A is d, B is d, A,
  B)
• else.. / the validated code comes here./

32
Possible Errors

• Programmer forgets to check for one of the two (A
  and B) cases resulting in the code

• if (!A)
• exit_with_error(Error A is d, B is d, A,
  B)
• or
• if (!B)
• exit_with_error(Error A is d, B is d, A, B)

33
Possible Errors (contd.)

• Or, the programmer uses a wrong logical operator
  as in
• if (!(A B))
• exit_with_error(Error A is d, B is d, A, B)

• Let us analyze how the four different tests will
  perform in each of the four implementations one
  correct, and three incorrect ones.

34
Truth table or condition
Inputs
Incorrect implementations

• A B !(A B) !(AB) !A !B
• T F F T F T
• F T F T T F
• F F T T T T
• T T F F F F

Correct implementation
Notice this one will it help find any of the
three possible errors?
35
Truth Table Analysis

• Case 1

• A test input with Atrue and Bfalse will cause
  the correct program to evaluate the condition to
  false.

• The two incorrect implementations, !(AB) and
  (!B) will evaluate the condition to true.

36
Truth Table Analysis (contd.)

• Both incorrect implementations will print the
  error message.

• The oracle will observe that the correct and the
  incorrect implementations behave differently.

• It will therefore announce failure for each
  incorrect implementation thereby pointing to an
  error in the code.
• End of Case 1.

37
Truth Table Analysis (contd.)

• Case 2
• Test input Afalse and Btrue will reveal the
  error in the two incorrect implementations,
  !(AB) and (!A).

• Case 3
• Test input Afalse and Bfalse might find a fault
  in the then branch of the if condition.

38
Truth Table Analysis (contd.)

• Case 4
• Test input Atrue and Btrue might find a fault
  in the else branch of the if condition.

• Thus, all four test inputs are likely to be
  useful.

39
Truth Table Analysis (contd.)

• However, if we were to check for the correct
  implementation of the condition A or B, then only
  the first two inputs are necessary.

• In this example, reducing the number of test
  specifications from 4 to 2 does not lead to any
  significant savings.

• When will the savings be significant?

40
Assumed pre-conditions

• Each assumed pre-condition is likely to result in
  a test requirement.

• Example
• Assumed MODE is on ground or flying

• This leads to two requirements
• MODE is on ground , MODE is not flying
• MODE is not on ground , MODE is flying

41
Assumed pre-conditions

• These can be simplified to
• MODE is on ground
• MODE is flying

42
Clues from code?

• Yes, clues can also be derived by scanning the
  code.

• However, such clues might be redundant and
  incomplete if coverage measurement and its use is
  planned.

• In the absence of coverage measurement, it is a
  good idea to scan the code and find clues.

43
Clues from code (contd.)

• Examine internal variables. These may lead to new
  test requirements.

• Example Suppose that variable length is input
  and denotes the length of an array. In the code
  we find
• int last_indexlength1

This leads to the following test requirements
44
Clues from code (contd.)

• an array of length zero
• array of length 1
• array with more than one element

• Later we will see how these clues and
  requirements might be derived, with more
  certainty, using boundary-value analysis.

• Another example Consider the sort program for
  which we have seen the specifications.

45
Clues from Code (contd.)

• The specifications do not indicate what algorithm
  is to be used for sorting the input array.

• A programmer might decide to use different
  algorithms for different array sizes. When
  scanning the code we may see
• if (scanltmin_length) simple_sort()
• else quicksort()

46
Clues from Code (contd.)

• Variable size and the check against min_length
  give us a clue for new test requirements. These
  are
• size is equal to or greater than min_length

• Later we will see how this clue and requirements
  might be derived, with certainty, using branch
  coverage.

47
Test Requirements Checklist

• Obtaining clues and deriving test requirements
  can become a tedious task.

• To keep it from overwhelming us it is a good idea
  to make a checklist of clues.

• This checklist is then transformed into a
  checklist of test requirements by going through
  each clue and deriving test requirements from it.

48
Test Specifications

• A test requirement indicates how to test a
  program. But it does not provide exact values of
  inputs.

• A test requirement is used to derive test
  specification, which is the exact specification
  of values of input and environment variables.

49
Test specifications (contd.)

• There might not be a one-to-one correspondence
  between test requirements and test specifications.

• A test requirement checklist might contain 50
  entries. These might result in only 22 test
  specifications.

• The fewer the tests the better but only if these
  tests are of good quality !

50
Test specifications (contd.)

• We will discuss test quality when discussing test
  assessment.

• A test specification looks like this

• Test 2
• global variable all_files is initially false.
• next_record is set to 1.

• Upon return expect
• all_files to be true
• next_record is last_record1

51
Test specifications (contd.)

• Notice the format of a test specification

• Each test is given a number which serves as its
  identifier.

• There is a set of input values.

• There is a set of expected values upon return
  from execution. Any side effects on files or
  networks must also be specified here. In essence,
  all observable effects must be specified in the
  Expect part of a test specification.

52
Test Specifications (contd.)

• Any side effects on files or networks must also
  be specified. In essence, all observable effects
  must be specified in the Expect part of a test
  specification.

• Similarly, values of all input variables, global
  or otherwise, must also be specified.

53
Test Requirements to Specifications

• The test requirements checklist guides the
  process of deriving test specifications.

• Initially all entries in the checklist are
  unmarked or set to 0.

• Each time a test is generated from a requirement
  it is marked or the count incremented by 1.

54
Test Requirements to Specifications

• Thus, at any point in time, one could assess the
  progress made towards the generation of test
  specifications.

• One could also determine how many tests have been
  generated using any test requirement.

55
Test Requirements to Specifications

• Once a test requirement has been marked or its
  count is more than 0 we say that it has been
  satisfied.

• Some rules of thumb to use while designing tests
• Try to satisfy multiple requirements using only
  one test.
• Satisfy all test requirements.

56
Test Requirements to Specifications

• Avoid reuse of same values of a variable in
  different tests. Generating new tests by varying
  an existing one is likely to lead to tests that
  test the same part of the code as the previous
  one.

In testing, variety helps!

• Though we try to combine several test
  requirements to generate one test case, this is
  not advisable when considering error conditions.

57
Test Requirements to Specifications

• For example, consider the following
• speed_dial, an interval
• speed_diallt0 ,error
• speed_dialgt120, error
• zones, an interval
• zones lt5, error
• zonesgt10, error

58
Test Requirements to Specifications

• One test specification obtained by combining the
  two requirements above is
• speed_dial-1
• zone3

• Now, if the code to handle these error conditions
  is

• if (speed_diallt0 speed_dialgt120)
• error_exit(Incorrect speed_dial)
• if (zonelt6 zonegt10)
• error_exit(Incorrect zone)

59
Test Requirements to Specifications

• For our test, the program will exit before it
  reaches the second if statement. Thus, it will
  miss detecting the error in coding the test for
  zone.

• Also, do not assume an error test to satisfy any
  other test requirement.

• Example
• Consider the function
• myfunction(int X, int Y)
• A test for the erroneous value of X might not
  test the code that uses Y.

60
Test Requirements to Specifications

• Test specifications must not mention internal
  variables. Remember, a test specification aids
  in setting input variables to suitable values
  before the test begins. Values of internal
  variables are computed during program execution.

• However, there are exceptions to the above rule.
  Can you think of one ?

61
Equivalence Partitioning

• The input domain is usually too large for
  exhaustive testing.

• It is therefore partitioned into a finite number
  of sub-domains for the selection of test inputs.

• Each sub-domain is known as an equivalence class
  and serves as a source of at least one test
  input.

62
Equivalence Partitioning
Four test inputs, one selected from each
sub-domain.
Too many test inputs.
63
How to partition?

• Inputs to a program provide clues to partitioning.

• Example 1
• Suppose that program P takes an integer X as
  input X.
• For Xlt0 the program is required to perform task
  T1 and for Xgt0 task T2.

64
How to partition? (contd.)

• The input domain is prohibitively large because X
  can assume a large number of values.

• However, we expect P to behave the same way for
  all Xlt0.

• Similarly, we expect P to perform the same way
  for all values of Xgt0.

• We therefore partition the input domain of P into
  two sub-domains.

65
Two sub-domains
All test inputs in the Xlt0 sub-domain are
considered equivalent. The assumption is that if
one test input in this sub-domain reveals an
error in the program, so will the others. This
is true of the test inputs in the Xgt0 sub-domain
also.
66
Non-overlapping Partitions

• In the previous example, the two equivalence
  classes are non-overlapping. In other words the
  two sub-domains are disjoint.

• When the sub-domains are disjoint, it is
  sufficient to pick one test input from each
  equivalence class to test the program.

67
Non-overlapping Partitions

• An equivalence class is considered covered when
  at least one test has been selected from it.

• In partition testing our goal is to cover all
  equivalence classes.

68
Overlapping Partitions

• Example 2
• Suppose that program P takes three integers X, Y
  and Z. It is known that
• XltY
• ZgtY

69
Overlapping partitions
70
Overlapping Partition-Test Selection

• In this example, we could select 4 test cases as
• X4, Y7, Z1 satisfies XltY
• X4, Y2, Z1 satisfies XgtY
• X1, Y7, Z9 satisfies ZgtY
• X1, Y7, Z2 satisfies ZltY

• Thus, we have one test case from each equivalence
  class.

71
Overlapping Partitions-Test Selection

• However, we may also select only 2 test inputs
  and satisfy all four equivalence classes
• X4, Y7, Z1 satisfies XltY and ZltY
• X4, Y2, Z3 satisfies XgtY and ZgtY

• Thus, we have reduced the number of test cases
  from 4 to 2 while covering each equivalence class.

72
Partitioning using non-numeric data

• In the previous two examples the inputs were
  integers. One can derive equivalence classes for
  other types of data also.

• Example 3
• Suppose that program P takes one character X and
  one string Y as inputs. P performs task T1 for
  all lower case characters and T2 for upper case
  characters. Also, it performs task T3 for the
  null string and T4 for all other strings.

73
Partitioning using non-numeric data
lc Lower case character UC Upper case
character null null string.
74
Non-numeric Data

• Once again we have overlapping partitions.

• We can select only 2 test inputs to cover all
  four equivalence classes. These are
• X lower case, Y null string
• X upper case, Y not a null string

75
Guidelines for equivalence partitioning

• Input condition specifies a range create one for
  the valid case and two for the invalid cases.

• e.g. for altXltb the classes are
• altXltb (valid case)
• Xlta and Xgtb (the invalid cases)

76
Guidelines (contd.)

• Input condition specifies a value create one for
  the valid value and two for incorrect values
  (below and above the valid value). This may not
  be possible for certain data types, e.g. for
  boolean.

• Input condition specifies a member of a set
  create one for the valid value and one for the
  invalid (not in the set) value.

77
Sufficiency of Partitions

• In the previous examples we derived equivalence
  classes based on the conditions satisfied by the
  input data.

• Then we selected just enough tests to cover each
  partition.

• Think of the advantages and disadvantages of this
  approach!

78
Boundary Value Analysis (BVA)

• Another way to generate test cases is to look for
  boundary values.

• Suppose a program takes an integer X as input.

• In the absence of any information, we assume that
  X0 is a boundary. Inputs to the program might
  lie on the boundary or on either side of the
  boundary.

79
BVA (contd.)

• This leads to 3 test inputs
• X0, X-20, and X14.

Note that the values -20 and 14 are on either
side of the boundary and are chosen arbitrarily.

• Notice that using BVA we get 3 equivalence
  classes. One of these three classes contains only
  one value (X0), the other two are large!

80
BVA (contd.)

• Now suppose that a program takes two integers X
  and Y and that x1ltXltx2 and y1ltYlty2.

81
BVA (contd.)

• In this case the four sides of the rectangle
  represent the boundary.

• The heuristic for test selection in this case is
• Select one test at each corner (1, 2, 3, 4).
• Select one test just outside of each of the four
  sides of the boundary (5, 6, 7, 8)

82
BVA (contd.)

• Select one test just inside of each of the four
  sides of the boundary (10, 11, 12, 13).

• Select one test case inside of the bounded region
  (9).

• Select one test case outside of the bounded
  region (14).

• How many equivalence classes do we get?

83
BVA (contd.)

• In the previous examples we considered only
  numeric data.

• BVA can be done on any type of data.

• For example, suppose that a program takes a
  string S and an integer X as inputs. The
  constraints on inputs are
• length(S)lt100 and altXltb

• Can you derive the test cases using BVA?

84
BVA Applied to Output Variables

• Just as we applied BVA to input data, we can
  apply it to output data.

• Doing so gives us equivalence classes for the
  output domain.

• We then try to find test inputs that will cover
  each output equivalence class.

85
Finite State Machines (FSMs)

• A state machine is an abstract representation of
  actions taken by a program or anything else that
  functions!

• It is specified as a quintuple
• A a finite input alphabet
• Q a finite set of states
• q0 initial state which is a member of Q.

86
FSMs (contd.)

• T state transitions which is a mapping
• Q x A--gt Q
• F A finite set of final states, F is a subset of
  Q.

• Example Here is a finite state machine that
  recognizes integers ending with a carriage return
  character.

• A0,1,2,3,4,5,6,7,8,9, CR
• Qq0,q1,q2
• q0 initial state

87
FSMs (contd.)

• T ((q0,d),q1),(q1,d),q1), (q1,CR),q2)
• F q2

• A state diagram is an easier to understand
  specification of a state machine. For the above
  machine, the state diagram appears on the next
  slide.

88
State diagram
d denotes a digit
89
State Diagram-Actions
x/y x is an element of the alphabet and y is an
action.
i is initialized to d when the machine moves from
state q0 to q1. i is incremented by 10d when the
machine moves from q1 to q1. The current value of
i is output when a CR is encountered.
Can you describe what this machine computes?Can
you construct a regular expression that
describes all strings recognized by this state
machine?
90
State Machine Languages

• Each state machine recognizes a language.

• The language recognized by a state machine is the
  set S of all strings such that
• when any string s in S is input to the state
  machine the machine goes through a sequence of
  transitions and ends up in the final state after
  having scanned all elements of s.

91
State diagram-Errors
d/add 10d to I
d/set I to d
CR/output I
CR/output error
reset
q4
q4 has been added to the set of states. It
represents an error state. Notice that reset is a
new member added to the alphabet.
92
State Diagram-Code

• A state diagram can be transformed into a program
  using case analysis. Let us look at a C program
  fragment that embodies logic represented by the
  previous state diagram.

• There is one function for each action.

• digit is assumed to be provided by the lexical
  analyzer.

93
Program for integer state machine
/ state is global, with values q0, q1, q2. i is
also global./
void event_digit()

• case q0
• idigit / perform action. /
• stateq1 / set next state. /
• break / event digit is done. /
• case q1
• ii10digit / Add the next digit. /
• stateq1
• break
• /complete the program. /

switch (state)
94
Checking State Diagrams

• Unreachable state One that cannot be reached
  from q0 using any sequence of transitions.

• Dead state One that cannot be left once it is
  reached.

95
Test Requirements

• Every state must be reached at least once, Obtain
  100 state coverage.

• Every transition must be exercised at least once.
  Obtain 100 transition coverage.

• The textbook talks about duplicate transitions.
  No transitions are duplicate if the state machine
  definition we have given is used.

96
Example Test Requirements

• For the integer state machine
• state machine transitions
• event digit in state q0
• event CR in state q0
• event digit in state q1
• event CR in state q1
• event reset in state q4

97
More testing of state machines?

• Yes, it is possible!

• When we learn about path coverage we will discuss
  how more test requirements can be derived from a
  state diagram.

98
Test Specifications

• As before, test specifications are derived from
  test requirements.

• In the absence of dead states, all states and
  transitions might be reachable by one test.

• It is advisable not to test the entire machine
  with one test case.

• Develop test specifications for our integer
  state machine.

99
Decision Tables

• Requirements of certain programs are specified by
  decision tables.

• A decision table is useful when specifying
  complex decision logic

100
Decision Tables

• A decision table has two parts
• condition part
• action part

• The two together specify under what condition
  will an action be performed.

101
Decision Table-Nomenclature

• C denotes a condition
• A denotes an action
• Y denotes true
• Ndenotes false
• X denotes action to be taken.
• Blank in condition denotes dont care
• Blank in action denotes do not take the action

102
Bank Example

• Consider a bank software responsible for debiting
  from an account. The relevant conditions and
  actions are
• C1 The account number is correct
• C2 The signature matches
• C3 There is enough money in the account
• A1 Give money
• A2 Give statement indicating insufficient funds
• A3 Call vigilance to check for fraud!

103
Decision tables
104
Example (contd.)

• A1 is to be performed when C1, C2, and C3 are
  true.

• A2 is to be performed when C1 is true and C2 and
  C3 are false or when C1 and C2 are true and C3 is
  false.

• A3 is to be performed when C2 and C3 are false.

105
Default Rules

• Are all possible combinations of conditions
  covered?

• No! Which ones are not covered?

• We need a default action for the uncovered
  combinations. A default action could be an error
  report or a reset.

106
Example-Test Requirements

• Each column is a rule and corresponds to at
  least one test requirement.

• If there are n columns then there are at least n
  test requirements.

• What is the maximum number of test requirements?

107
Example-Test Specifications

• For each test requirement find a set of input
  values of variables such that the selected rule
  is satisfied.

• When this test is input to the program the output
  must correspond to the action specified in the
  decision table.

• Should the testing depend on the order in which
  the conditions are evaluated?

108
Summary

• Specifications, pre-conditions, and
  post-conditions.

• Clues, test requirements, and test
  specifications.

• Clues from code.

• Test requirements catalog.

• Equivalence partitioning and boundary value
  analysis.

• Testing state machines.

109
Summary-continued

• Finite state machine

• State diagram

• Events and actions

• Unreachable and dead states

• Test requirements and specifications for state
  machines

• Decision tables, rules, actions