Specification-based%20Protocol%20Testing

1
Specification-based Protocol Testing

• Hyoung Seok Hong
• Oleg Sokolsky
• CSE 642

2
Outline

• Introduction
• Specifications, implementations
• Conformance relations
• Tests, testing architectures
• Assumptions, etc.
• Formal protocol specifications
• Finite-state machine (FSM) testing
• Extended FSM (EFSM) testing
• Test generation for StateCharts

3
Testing problem

• A specification gives a description of the system
  behavior.
• An implementation should behave according to the
  specification.
• Is this really true?

a
a
b
c
b
b
a
4
Black-box vs. White-box testing

• Black-box unknown system structure
• Testing against reference specification
• Use the specification interface for
• test selection, coverage criteria, result
  analysis
• White-box known system structure
• Testing against reference specification
• Use system structure for
• test selection, etc.

5
Protocol testing architectures
Local architecture
6
Protocol testing architectures
Distributed architecture
7
General testing architecture
IUT
Tester
Tester
PCO
PCO
PCO
Tester
8
Tests and test suites

• A test case defines
• A finite sequence of inputs
• A finite sequence of expected outputs
• A test suite is a collection of tests to achieve
  a certain test coverage
• Testing process
• test generation construct a test suite
• test application execute each test
• result evaluation interpret outputs

9
Test structure

• Test purpose
• e.g., identify a state of the implementation
• Test preamble
• lead the IUT into known state
• Test body
• invoke behavior corresponding to the test purpose
• Checking of test results

10
Modeling approach

• Testing criteria on a concrete system are hard to
  formulate

implementation
detailed specification
?
11
Outline

• Introduction
• Formal protocol specifications
• Formal specification languages
• Finite-state machine (FSM) testing
• Extended FSM (EFSM) testing
• Test generation for StateCharts

12
Formal specification languages

• Extended FSM
• SDL
• Estelle
• StateCharts
• Process Algebras
• LOTOS
• Algebraic Specification
• Z, VDM

13
Outline

• Introduction
• Formal protocol specifications
• Finite-state machine (FSM) testing
• FSM model
• FSM fault models
• FSM test selection
• Extended FSM (EFSM) testing
• Test generation for StateCharts

14
FSM model

• Finite sets of input (X) and output (Y) events

15
FSM-based fault model

• Output faults
• Transfer faults
• Transfer faults with additional states
• Additional or missing transitions
• Control and data flow faults?

16
Testing based on FSM models

• Testing methods based on control flow
• Transition-tour (TT) method
• a single sequence of inputs to traverse all
  transitions
• simple but weak
• Unique-input-output (UIO) method
• a test to identify each state in the
  specification
• Distinguishing sequence (DS) method
• identify each state by the same test body
• provides full coverage, but may not have a DS

17
UIO testing method

• Assumption for each state, there is an input
  sequence that produces a unique output
• Each test identifies one state
• s1 b/1
• s2 a/0 a/0 b/1
• s3 b/1 a/1
• preamble body

b/1
s1
s2
b/0
b/0
a/0
a/0
a/1
s3
18
Outline

• Introduction
• Formal protocol specifications
• Finite-state machine (FSM) testing
• Extended FSM (EFSM) testing
• EFSM model
• EFSM fault models
• EFSM test selection
• Test generation for StateCharts

19
EFSM model

• Finite sets of input (X) and output (Y) events

• A finite set of variables (V)

20
Tests for EFSM models

• EFSM executions depend on input signals and data
• A test consists of
• test sequence
• test data
• Executability problem
• Find data to execute the test sequence
• Undecidable, in general ?

21
Test validation

• Executability problem introduces an additional
  step in the testing process
• test generation construct a test suite
• test application execute each test
• result evaluation interpret outputs

22
Control and data flow testing

• EFSM executions are data-dependent
• Control flow FSM testing methods are not adequate
  for EFSM models
• Data flow testing methods account for data
  dependencies

23
Data flow definitions and uses

• Data variables are
• defined by inputs and updates (def)
• used in
• updates (c-use)
• guards (p-use)
• Data-flow graph captures data dependencies

24
Data flow graph

• Directed graph with nodes labeled with
  definitions and uses of variables

p-use v def u
v0?input u
ult0?vu1
p-use u c-use u def v
25
Data-flow coverage criteria

• all-def
• test suite covers each definition at least once
• all-use
• cover each def-use association at least once
• all-du-paths
• exercise all paths from each definition of a
  variable to every its use.

26
all-use coverage

• Find at least one definition-free path for every
  def-use association

no definitons of v
def v
use v
27
Outline

• Introduction
• Formal protocol specifications
• Finite-state machine (FSM) testing
• Extended FSM (EFSM) testing
• Test generation for Statecharts

28
Statecharts

• EFSMs (FSMs variables) concurrency
  hierarchy communication real-time
• Widely used for specifying real-time embedded
  HW/SW controllers
• Also used in most of object-oriented
  methodologies, e.g., UML

29
A coffee vending machine
Statecharts
30
Test sequences for Statecharts

• The main purpose
• What should implementations do?
• power-on/light-on, inc/, coffee/start
• What should not implementations do?
• power-on/light-on, coffee/
• The main issue
• How can we generate a finite and reasonable
  number of test sequences?

31
State coverage
32
Transition coverage
33
Data flow oriented coverage
34
Test generation from Statecharts (I)

• Basic idea
• Transforms Statecharts into EFSMs
• Advantage
• Can reuse the existing methods and tools for
  EFSMs
• Can handle infinite state space
• Disadvantage
• Cannot determine the executability of test
  sequences

35
(No Transcript)
36
Test generation from Statecharts (II)

• Basic Idea
• Test generation can be reduced into model
  checking
• TG for each state s, is there a path leading to
  s
• MC a temporal logic formula EFs express this
  property
• Advantage
• Fully automatic
• Generates only executable test sequences
• Disadvantages
• Can handle only finite state space

37
Another perspective

• Testing classes in object-oriented programs
• Can we show an CLASS implementation conforms to a
  Statechart specification?
• Variables of Statecharts data members of classes
• Events of Statecharts member functions of
  classes
• Testing inheritance in object-oriented programs
• Can we reuse the test sequences of a super-class
  for testing its subclasses ?

38
A coffee-cocoa vending machine