Automated Generation of Tests

1
Automated Generation of Tests

• Generate tests from what?
• From model of requirements or specification
• Can check if implementation meets what is
  expected
• Difficulty level of abstraction
• Code
• Definitely provides required level of detail
• Cannot be used for functional testing
• Can be used for regression testing of next
  version of software

2
Strategy

• To generate tests automatically, we need a goal
• Choose units of measurement for test coverage
• structural
• data-oriented
• interactions
• We would like to generate a set of tests that
  maximizes the coverage, and minimizes the size of
  the set.

3
Example

• Suppose we want to automatically generate
  regression tests from code.
• Coverage strategy use criterion based on code
  structure.
• Example take all branches in code

4
Steps in Automated Test Generation (1)

• Parse the code, and generate a flow graph.
• Analyze the flow graph to determine units of test
  coverage criteria.
• Determine a set of sub-paths through each
  structural test unit of interest.
• Combine sub-paths into a complete path (from
  entry point to exit point).
• Trace path in reverse from exit to entry, and
  collect conditions at each branch point.

5
Steps to Generate Structural Tests (2)

• Solve set of data conditions required to take
  complete path.
• This has the effect of deriving equivalence
  classes of input data.
• Any member of the equivalence class should result
  in executing the same path through the code.
• Feasibility of data conditions must be checked
• If no data satisfies input conditions, return to
  step 4 to search for a different complete path to
  cover test unit.

6
Steps to Generate Structural Tests (3)

• Choose representative members from equivalence
  classes.
• Trace path in forward direction, in order to
  predict expected output.
• In many cases, this is extremely difficult.
• Combine input with test environment information.
• Generate test script.

7
Example Delete rows from array (1)

• public int deleteRows( int anArray,
• int firstRow, int
  lastRow )
• int result null
• int numRows anArray.length
• if ( ( firstRow gt numRows ) ( firstRow lt 0
  ) )
• System.out.println( Bad first row. )
• else if ( ( lastRow gt numRows ) ( lastRow
  lt 0 ) )
• System.out.println( Bad last row. )
• else if ( lastRow lt firstRow )
• System.out.println( Not a valid range. )

S1
D1
S2
D2
S3
D3
S4
8
Example Delete rows from array (2)

• else
• int numNewRows numRows - ( lastRow -
  firstRow 1 )
• result new intnumNewRowsanArray0.leng
  th
• int offset 0
• for ( int row 0 row lt numRows row )
• if ( ( row gt firstRow ) ( row lt
  lastRow ) )
• offset
• else
• resultrow-offset anArrayrow
• return result

S5
S8
D4
D5
S6
S7
S9
9
Flow graph
T
S2
D1
S1
S3
T
T
S4
D2
F
D3
T
F
S6
S9
T
S8
D5
S5
F
D4
S7
F
F
10
Potential Coverage Measures

• Statement blocks Cover S1 through S8
• Decision Cover true and false cases for each of
  D1 through D5
• Condition coverage cover true and false cases
  for each individual condition with D1 through D5
• Decision-Condition coverage combination of
  above two
• Example use decision coverage

11
Create a sub-path forunit of test coverage
T
S2
D1
S1
S3
T
T
S4
D2
F
D3
T
F
S6
S9
T
S8
D5
S5
unit of coverage
F
D4
S7
F
F
12
Extend sub-path to complete path
T
S2
D1
S1
S3
T
T
S4
D2
F
D3
T
F
S6
S9
T
S8
D5
S5
F
D4
S7
F
F
13
Solve conditions
T
S2
D1
S1
S3
T
T
S4
D2
F
D3
T
F
S6
S9
Need D2 true D1 false
T
S8
D5
S5
F
D4
S7
F
F
14
Solve conditions

• Decision D1 (false)
• ( firstRow gt numRows ) ( firstRow lt 0 )
• Decision D2 (true)
• ( lastRow gt numRows ) ( lastRow lt 0 )
• Determine input that affects numRows
• int numRows anArray.length
• Therefore, for this path, we need ! ((firstRow
  gt anArray.length) (firstRow lt 0))
  ((lastRow gt anArray.length) (lastRow lt 0 ))

15
Equivalence classes (not complete)

• firstRow (-?,-1 0,anArray.length-1
  anArray.length,?)
• lastRow
• (-?,-10,firstRow-1firstRow,anArray.length-1a
  nArray.length,?)
• Need firstRow in equivalence class0,anArray.leng
  th-1 and lastRow from equivalence class (-?,-1
  or anArray.length,?)

S2
S2
S3
S3
S4
16
Input Data Selection

• Need to find specific values for each input
• anArray anArray.length gt 1, number of columns
  unspecified, contents unspecified
• Choose anArray.length 3, number of columns 2,
  fill array with ones.
• firstRow must be between 0 and 2 inclusive
• Choose 1
• lastRow must be -1 or less, or 3 or greater
• Choose 5

17
Predict output (1)

• public int deleteRows( int anArray,
• int firstRow, int
  lastRow )
• int result null
• int numRows anArray.length
• if ( ( firstRow gt numRows ) ( firstRow lt 0
  ) )
• // not executed
• else if ( ( lastRow gt numRows ) ( lastRow
  lt 0 ) )
• System.out.println( Bad last row. )
• else if ( lastRow lt firstRow )
• // not executed

S1
S2
S3
S4
18
Predict output (2)

• else
• ... // not executed
• return result

S9
19
We have

• Output
• Bad last row.
• Return value
• result null

20
A JUnit test

• public void testD2S3branch()
• int anArray 1,1,1,1,1,1
• int firstRow 1
• int lastRow 5
• int result AClass.deleteRows( anArray,
• firstRow,
• lastRow )
• Assert.assertNull( result )
• // need to check output as well!

21
Issues

• Feasibility of solving conditions
• Non-linear terms, inequalities, ...
• Prediction of output
• Complex calculations
• Optimization of coverage of test units
• Choice of equivalence class representatives
• Knowledge of test environment

22
State-based Models (1)

• A state-based model consists of
• States points where the system is waiting for
  an event to occur.
• optional Variables values associated with
  the state machine.
• Transitions the change from one state to
  another. A transition is composed of
• Event stimulus to the system, that causes the
  implementation to exit from a state.
• optional, requires variables Guard
  condition(s) additional Boolean conditions that
  must be true to take the transition.
• optional Actions action(s) to perform as a
  result of the event
• Next state the new state that should be entered
  at the end of the transition.

23
State-based Models (2)

• An initial state must be identified.
• A initial transition from the initial state to
  another state can be specified. This transition
  should not have an event or a guard condition
  only action(s).
• The implementation is in the initial state at
  start up, and immediately executes the initial
  transition without stimulus.
• Analogy calling a constructor method.
• The implementation will never return to the
  initial state unless it is restarted no
  transition can have the initial state as its next
  state.
• Actions can consist of
• Output events (this can trigger events in the
  same or other models)
• Modification of state variables (if variables are
  used)

24
Example A stack
Event
Guard condition
Empty
push / add, size
-- / size 0
Initial State
push size lt max 1 / add, size
pop / error
pop size 1 / remove, size--
Partly Filled
push size max 1 / add, size
pop size gt 1 / add, size
State
Assumption max gt 1
Actions
Full
push / error
pop / remove, size--
25
Potential Faults (1)

• This slide and the next represent the error model
  for state-based models. Adapted from Binder.
• Action fault
• The actions on a transition are incorrect, or
  missing.
• Guard condition fault
• The guard condition on a transition is not
  correct.
• Next state fault
• Transition is to a legal, but incorrect, state.

26
Potential Faults (2)

• State fault
• There are extra states, or missing states.
• Unspecified event / missing transition
• There is no transition specified for a legal
  event at a particular state.
• Extra transition
• A legal event is accepted in a particular state,
  but it should not be.
• Illegal event failure
• Unexpected event causes a failure
• Trap door
• Implementation accepts events that are not
  intended to be accepted at any time.

27
Testing strategies

• Coverage criteria during the execution of the
  test suite ...
• all states are visited.
• all events occur.
• all actions taken.
• all transitions taken (includes all of above)
• Conformance testing for communications has a
  standardized strategy (ISO 9646)

28
ISO 9646 Conformance Tests

• Developed as a standard for conformance testing
  of communications protocols that used an FSM
  model to describe protocol behaviour.
• For a particular protocol, this is a standard
  methodology for deriving a set of tests that must
  pass in order for an implementation to be
  certified for use.
• Two objectives
• Check that the implementation has states
  corresponding to the model.
• Check that the implementation has transitions
  corresponding to the model.

29
ISO 9646 Conformance Tests

• A test consists of
• Preamble Series of events that take the system
  from a known / initial state to a targeted state,
  or the state at the head of the transition being
  targeted.
• Event Cause the system to take the targeted
  transition.
• Response Check that the appropriate actions for
  the transition are taken, based on observed
  response.
• Next state verification Collect evidence that
  the implementation is now in the correct state.
• How to do this?
• Postamble Put the implementation into a known
  state (to set up the next test).

30
State Verification

• How can we check that a transition ends in a
  specific state?
• If we have an implementation that has a
  queryState() method, we can call this after a
  transition has been executed, and see if we are
  in the correct state.
• Requires an implementation that has been designed
  with testing in mind.
• Otherwise, we have to use indirect evidence.

31
Indirect state verification

• If all we can observe are responses from the
  implementation, we can try to observe
  event-response combinations that could only have
  happened if we were in the desired state.
• Example Suppose we are in an unknown state, and
  the following event / response sequence is
  performed.
• pop / error
• This could only have happened if the state we
  were in initially was Empty.
• Suppose we have a transition x / y that should
  lead to the state Empty.
• The sequence x / y pop / error will check the
  transitions action (y) in response to event x,
  and that after the transition is taken, the
  transition entered the correct state.
• If we see a response other than error to the pop
  event, then the transition x / y must have gone
  to an incorrect state.

32
Indirect state verification

• Distinguishing sequence (DS) a sequence of
  inputs that produces a different sequence of
  responses for every state in the implementation.
• This is the most useful property if there is such
  a sequence apply the DS and identify the state
  from the response.
• No guarantee that a DS exists
• Unique input/output sequence (UIO) for a given
  state, there is a sequence of inputs that
  produces a response sequence different from any
  other state.
• This can only provide a yes/no answer to the
  question, am I in the given state?
• No guarantee that a UIO sequence exists but it
  is more likely to exist than a DS.

33
Preambles and Postambles

• Preambles
• Objective take the system from the designated
  start state to any targeted state in the system,
  in order to set up a test.
• Requirements a sequence of events that will
  move the system to a specified state.
• The start point of a preamble is fixed the state
  where a preamble ends will vary depending on the
  test purpose.
• Postamble
• Objective take the system from the state in
  which it is believed the system is in, to a final
  state or the common start/end state.
• Requirements a sequence of events that will
  move the system to a designated final state, in
  order to leave the system in a known state.
• The start point of a postamble will depend on
  which state the system is in after state
  verification which may not necessarily be the
  end state of a targeted transition.

34
Sensor example

• Two motion sensors, one at door 1 and the other
  at door 2. When someone enters the detection
  range, a sensor reports (). When someone leaves
  the detection range, a sensor reports (-).
• Integrate the two sensor inputs to produce
• P (present) event when someone enters the area
• A (absent) event when the area becomes clear.

35
Sensor example
-1 / --
Note that -1/A isunique to state Only1.
-2 / --
Neither
2 / P
1 / P
-2 / A
Only1
Only2
-1 / A
1 / -- - 2 / --
2 / --
- 1 / --
1 / --
2 / --

• 1 sensor 1 on
• 1 sensor 1 off
• P present
• A absent
• -- no output

-2 / --
-1 / --
Both
Test sequence 1/P, 2/--, -2/--, -1/A
1 / -- 2 / --
36
Developing Conformance Tests

• Step 1 Determine if a state verification
  sequence exists.
• A DS to identify all states
• A UIO sequence for each state.
• Step 2 Determine the start state for testing.
• All test cases will start in this state.
• All test cases will end in this state, if the
  system is intended to run continuously.
• Choose this to be a state which is easily
  reachable from any other state, as well as on
  system start-up.
• Usually, this is the state after the initial
  transition, and represents a ready or idle
  state.

37
Developing Conformance Tests (2)

• Step 3 Determine a path from the start state to
  each of the other states.
• Path includes set of events and expected
  observable responses.
• This will allow for preambles to put the system
  in a known state at the start of any transition
  (except the initial transition).
• Step 4 Determine a path from each state to ...
• ... the start state, for a continuously running
  system.
• ... a final state, if such states are present.
• Choose a path that is as short as possible.
• This will allow for postambles, to return the
  system to a known state, or to terminate the
  system gracefully.
• Objective set system up for the following test.

38
Developing Conformance Tests (3)

• Step 5 Determine if we can observe a distinct
  set of states.
• This will be done by checking that the
  distinguishing sequence OR unique input-output
  sequences derived from the state model actually
  work in the implementation
• See next slide for how to do this.
• This will allow us to discover if the
  implementation has at least as many states as
  the state model.
• Does not detect the presence of extra states in
  the implementation.
• If these test paths do not result in test cases
  that pass, the remainder of the test cases that
  will be created in step 6 are not ready to
  execute.

39
Developing Conformance Tests (4)

• Step 5 continued
• Objective verify that we can observe distinct
  states.
• To check that a distinguishing sequence really
  distinguishes the states, use preamble, DS,
  postamble for each of the states where DS is the
  distinguishing sequence input and the expected
  output corresponds to the state.
• To check that the unique-input-output (UIO)
  sequences distinguish the states
• for each state S do
• for each state T do
• preamble(T), UIO(S,T), postamble
• In the above, UIO(S,T) represents applying the
  UIO input for state S, but with expected results
  forstate T.

40
Developing Conformance Tests (5)

• Step 6 Create a test case path for each
  transition, consisting of
• The preamble for the state from which the
  transition exits.
• The event and expected response for the
  transition.
• The state verification sequence for the next
  state after the transition.
• The postamble for the state expected to be
  reached at the end of the state verification
  sequence.

41
Error model What is detected

• Using the ISO 9646 conformance test approach will
  detect
• All implementations with fewer states than the
  model.
• All implementations with an output error
• An incorrect observable response on a transition.
• All implementations with a next state error
• A transition that has the correct response, but
  goes to an incorrect state.
• All implementations with a missing transition.

42
Error model What may not be detected

• What may not be detected
• Incorrect non-observable actions on transitions
  (e.g. problems with state variables)
• Implementations with more states than the model.
• Extra transitions, if the model is not completely
  specified
• Completely-specified transitions are present
  for all legal events in every state.
• Illegal transitions acceptance of non-legal
  events.

43
Sensor Example

• For each state, we need
• A preamble leading to the state
• A sequence that will verify that
• Also record the state at the end of the sequence,
  to select the state postamble
• A postamble taking us to a known state (in this
  case, Neither).

44
State verification for both

• It turns out that there is not a single sequence
  that we can use to verify the state both.
• However
• -2 / --, -1 / A reduces the possibilities to
  both or only1
• -1 / --, -2 / A reduces the possibilities to
  both or only2
• We can use the combination of the two sequences
  to reduce the possible states to just both. If
  the two sequences pass, we must have been in our
  desired state.
• However, we need to be at the end of the
  transition being checked before executing each of
  the state verification sequences.
• This means that we have to use two test cases for
  the transition
• Preamble, transition, state verification 1,
  postamble
• Preamble, transition, state verification 2,
  postamble
• Both test cases have to pass to verify the
  transition.

45
Check the states

• This set of tests will identify the existence of
  these states in the actual implementation
• Each row in the table represents a test case.
• Since we are using UIO sequences for state
  verification, we will have to apply each UIO
  sequence to each state.
• Results in a minimum of n2 tests for n states in
  the model.
• State both has 2 partial UIO sequences, so we
  have to verify the combination is distinctive.
• This means applying both UIO sequences to every
  state.
• Result more than n2 test cases to check states
  for the sensor example.

46
Check the states (1/3)
47
Check the states (2/3)
48
Check the states (3/3)
49
Check the transitions

• This set of tests will check the response on each
  transition, and also verify that the transition
  goes to the correct next state.
• Each row in the table represents a test case.
• For each transition, there will be at least one
  test case.
• Results in a minimum of m tests for m transitions
  in the model.
• For any transition that is expected to end in the
  both state, two test cases will be needed as
  there are two partial state verification
  sequences for this state.

50
Check the transitions (1/2)
51
Check the transitions (2/2)
52
The conformance test suite

• 41 test cases 20 to check states, and 21 to
  check transitions.
• If an incorrect response is observed during the
  execution of the preamble of a test case, the
  verdict should be declared as inconclusive
• Some other test will target the transition on the
  preamble that didnt work. This test case would
  fail.
• Doing this will better identify the location of
  problems.
• What about an incorrect response during the
  postamble?
• Something to think about
• If the last few steps of a test are the same as
  the first few steps of some other test, could you
  overlap the tests? What are the implications?

53
Issues for state-based automated test generation

• An automated tool has to replicate the process we
  have just followed.
• What is needed
• Parser for state machine format.
• Algorithm to find state verification sequences
• Computation of DS or UIO sequences is an
  NP-complete problem in the general case, although
  the can be found easily in many cases.
• Sequences do not necessarily exist.
• Path feasibility checking
• Path output prediction

54
Issues for state-based automated test generation

• What is needed (continued)
• Algorithm to find preambles and postambles for
  states.
• Theoretic approach shortest path to/from
  desired state
• Practical approach user may want to identify
  preferred preambles and postambles to keep the
  system as close to normal operation as possible.
• That is, avoid using something like an emergency
  shutdown system as a postamble.

55
Issues for state-based automated test generation

• Transient states
• These states exist only for a brief amount of
  time, normally because one of the events is
  expected quickly in real time.
• A state-based test generation tool will probably
  not take this into account and generate a test
  for other events from the state.
• Such tests are difficult, if not impossible, to
  execute reliably.

transition that will normally be taken quickly
transient state
event that could happen from this state
56
Issues for state-based automated test generation

• Ordering of multiple responses on one transition
• As a result of one event, several responses may
  be detected. They may appear in a specific order
  in the state model, but may not be observed in
  that order from the system.
• If the state model was a design abstraction, the
  ordering may not have been preserved in the
  implementation.
• Test equipment may not observe the events in the
  same order.

state 1
state 2
event / action1, action2, action 3
57
Issues for state-based automated test generation

• Multiple, concurrent timers
• Example
• Send a message, and start two concurrent timers.
• Duration of timers timer1 2s, timer2 60s
• Timer 1 expiry indicates resend of message, timer
  2 expiry assumes connection is broken and aborts.
  
• Path generator has to determine that to take
  timer 2 expiry transition, one must take many
  loops of the timer 1 expiry transition
  beforehand.

timer1 expires / askReply,
start(timer1)
Await response
... / askReply, start(timer1),
start(timer2)
reply / cancel(timer1), cancel(timer2)
timer2 expires / ...
Abort
58
Information pointers

• FSM / conformance testing conference series
• FORTE (formal description techniques)
• PSTV (protocol specification, testing, and
  verification)
• IWTCS / Testcom (testing of communicating
  systems)
• ISSTA / ISSRE (software testing and analysis /
  software reliability engineering)