Principles and Methods of Testing Finite State Machines

1
Principles and Methods of Testing Finite State
Machines A Survey
David Lee, Senior Member, IEEE and Mihalis
Yannakakis invited paper
Part about Conformance Testing
2
Overview

• Main concepts FSM, sequences, equivalence
• Conformance testing problem and assumptions
• Separating family of sequences
• Status messages and reset
• Distinguishing sequences
• Identifying sequences
• Polynomial time randomised algorithm
• Heuristic procedures and optimisations
• Extensions one machine, more states, partially
  specified

3
Concepts FSM

• Finite State Machine (Mealy machine)
• Is a tuple M(I, O, S, ?, ?)
• I, O input and output symbol sets (alphabets),
• S state set,
• ? S?I ? S transition function,
• ? S?I ? O output function.
• Example
• When the machine is in state s?S and receives
  input symbol a?I it moves to state ?(s, a) and
  produces output symbol ?(s, a).

4
Concepts sequences and state equivalence

• We extend transition and output functions
• Let x be an input string xa1,,ak, ai?I, for
  i1..k
• then ?(s1, x)sk1 where si1?(si, ai), for
  i1..k
• and ?(s1, x)b1bn where bi?(si, x), for i1..k
• x is a sequence and b1bn is a response to it.
• Equivalent si, sj ?S ? ?x?I ?(si, x) ?(sj, x)
• If si and sj are not equivalent then
• ? separating sequence x?I ?(si, x) ? ?(sj, x)

5
Machine equivalence

• Equivalent A, B ?FSM ?
• ?sA?SA ?sB ?SB sB equivalent to sA
• ?sB?SB ?sA ?SA sA equivalent to sB
• There are many equivalent machines
• Each equivalent machine class contains minimized
  machine M with minimum number of states (all
  states are unique).
• Minimized machine is unique.

6
Conformance testing problem

• Given
• Complete information of specification machine A
  (states, transition and output function)
• Implementation machine B, black box, only I/O is
  observable
• Goal
• Determine whether B is correct implementation of
  (conforms to, is equivalent to) A by applying a
  test sequence to B and observing the output.
• Checking sequence for machine A with n states is
  an input sequence x that distinguishes A from any
  non-equivalent machine B with n states
• ?(sB, x) ? ?(sA, x) ? B is not equivalent to A

7
Conformance testing assumption

• Specification A is strongly connected
• It must be possible to reach all states
• A is reduced (minimized)
• We can determine equivalence only to minimized
  machine, since equivalent states are not
  distinguishable.
• B does not change during experiment and has the
  same input alphabet as A
• B has no more states than A
• Assume, faults do not increase number of states,
  only
• Wrong output on transition
• Wrong state in transition destination

8
Conformance test structure

• Algorithm structure
• Initialization move to some known state s1
• If s1 is given ? verify it (not always possible)
• Else apply homing sequence that takes to some
  known state s1 (possible for minimised machines)
• Verify similarity of B to A
• Verify each transition ?(si, a)sj
• Apply sequence that moves machine to si
• Apply a
• Verify that machine is in sj
• Methods
• Status and Reset messages
• Distinguishing sequences
• Identifying sequences
• Randomised sequences

9
Separating family of sequences

• Separating family of sequences for A is a
  collection of n sets ?i (one set for each state)
• For every pair of states si, sj (si?sj) there is
  sequence ?
• ?(si, ?) ? ?(sj, ?)
• ? is a prefix in some xi??i and a prefix in some
  xj??j
• ?i is called a separating set of state si
• Elements of ?i are separating sequences of state
  si
• Examples
• A has a preset distinguishing sequence x then all
  sets ?i may be equal to x.
• UIO sequences fulfil 1) but may violate 2).

10
Separating family properties

• General construction algorithm for reduced FSMs
• If A is reduced ? ?si,sj ?x separating sequence
• Partition states into blocks based on ?(sk, x)
• For every sk put x into Zk
• Repeat procedure for each block until all blocks
  become singletons.
• Every pair of states has a separating sequence
  with common prefix.
• Zi contains ltn-1 sequences of length ?n.
• Key property only one state at most gives the
  same response to all elements of particular ?i

11
State and machine similarity

• State similarity
• qi?SB is similar to si?SA ? ?(qi, x)?(si, x)
  ?x??i
• Because of key property qi can be similar to at
  most one state of A
• Machine similarity
• B is similar to A ? ?si?SA ?qi?SB qi is similar
  to si
• All qi are distinct and B has at most n states ?
  one-to-one correspondence between A and B states.

12
Status messages and Reset

• FSM has a reset capability if special input r
  takes the machine from any state to initial s1.
• If r input is defined for all states of B then
  reset is reliable.
• Status message tells the current state of machine
  without changing it.
• Reliable status message guarantees that state
  will stay the same as before message.
• Unreliable status messages must be applied twice
  when the state is expected to change.

13
Test with reliable reset

• Let ?i be a family of separating sets
• Build a spanning tree with states in nodes from
  diagram of machine A
• B similarity to A check
• For every si?SA
• for every x?Zi
• Reset B to state s1 by applying r
• Move to state si according to tree path from s1
  to si
• Apply x
• ? we are sure that B moved to a state similar to
  si
• ? we are sure that B has states similar to A
  ones,
• ? B has no more states than A has, ? B is similar
  to A

14
Test with reliable reset (2)

• Check all other transitions ?(si, a)sj
• For every x?Zj
• Reset to s1 by r
• Move to state si according to tree path from s1
  to si
• Apply a
• Apply x
• ? we are sure that transition ?(si, a)sj is OK
• ? B is isomorphic to A

15
Test with distinguishing sequences

• Let A have an adaptive distinguishing sequence,
  then Zixi, where xi is a path in decision tree
  from root to state si
• ?(si, sj) is a pre-computed transition sequence
  that takes machine from state si to state sj
• ti ?(si, xi) some state after separation
• Similarity test
• x1 ?(t1, s2)x2 ?(t2, s3)x3 xn ?(tn, s1)x1
• ?we have visited all n states and observed all
  distinguishing responses to distinguishing
  sequences
• Transition ?(si, a)sj test when in state tk
• ?(tk, si-1)xi-1?(ti-1, si) a xj

16
Test with identifying sequences

• Problem to verify state si similarity we need to
  apply separating sequence Zi times precisely on
  the same state.
• Example
• Separating family Zia, b
• Machine is in state s1
• Apply a?a?a?, observe 000
• B was in q0, moved to q1, q2, q3
• However, at least two of states are the same
• ? q3qi for some i0..2 ? we have already applied
  a on q3 ? lets apply b on it!
• ? we have successfully applied a and b on the
  same state, ? B has state q3 similar to state s1
  in A.

17
Test with identifying sequences (2)

• A has n states
• Similarity to si with separating set Ziz1, z2
• Let qr?(si, z1?(ti, si))r ), where ti?(si, z1)
• Apply qn
• Then ?rltn qr qn, i.e. we have already applied
  z1 successfully on qn
• Apply z2
• ? sequence (z1?(ti, si))nz2 identifies a state in
  B which is similar to si
• Transition check reuse reliable reset idea,
  because identifying sequences actually reset the
  state.

18
Polynomial time randomized test

• Similarity
• For i1,,n
• Repeat ki times
• Apply sequence that takes to si from current
  state
• Choose a separating sequence z?Zi uniformly at
  random
• Apply z
• Let x be random input formed sequence from
  similarity test with
• kiO(n Zi min(p, Zi) log n)
• Then it can be shown that B is similar to A with
  high probability

19
Heuristic procedures and optimizations

• Checking sequences guarantee complete fault
  coverage, but sometimes they are too long.
• Success example circuit testing is based on
  faults model significantly limit possible faults.
• Covering paths
• Transition checking with UIO sequences leads to
  Postman Tour Problem ? NP-hard.
• Random walk may be trapped if system has
  narrow passages or has just few faults ?
  exponential.
• Guided random walks records partial history and
  makes random choice based on priorities from
  history.
• Test sequences from combinations of
  sub-sequences overlap in sub-sequences.

20
Summary of algorithms

• Complexity is very sensitive to preliminary
  conditions and requirements to the system
• pI number of inputs, nS - number of states

Sequences Length Time
Reliable reset pn3 pn3
Distinguishing pn3 pn3
Identifying Exponential Exponential
Randomized Polynomial Polynomial
21
Conformance test extensions

• One black-box only
• No difference for deterministic test algorithms
• Randomized algorithms are more effective on a
  single fixed machine fault probability is
  squared while test length is doubled
• More states than in specification
• Unknown states and transitions have combination
  lock property that require exponentially long
  sequences to cover.
• Partially specified machines
• Special treatment for undefined transitions may
  transform machine to fully specified machines.
• One fault - exponentially many machines
• Only randomised checking has polynomial complexity

22
Evaluation and critics

• Valuable digest of completely solved problems.
• Good introduction to partially solved ones.
• Easy to read concepts are presented following
  the pattern motivation, definition and simple
  example.
• Most of algorithms are encoded in complex
  sentences instead of nested blocks gives
  motivation, but looses clarity.

23
Thank you for your attention!

• Questions?

24
Interesting proposition

• Let A and B satisfy the assumptions, then the
  following are equivalent
• A and B are isomorphic (there exists a bijection
  between equivalent A and B states)
• A and B are equivalent
• At least one state of A has an equivalent state
  in B