Fast Functional Test Generation Using an SDL Model

1
Fast Functional Test GenerationUsing an SDL Model

• Robert L. Probert
• Alan W. Williams
• School of Information Technology and Engineering
• University of Ottawa
• Ottawa Canada
• bob,awilliam_at_site.uottawa.cawww.site.uottawa.ca
  /awilliam

2
Overview

• Goals of the project
• Project Steps
• Functional Testing
• Development of
• requirements models
• test models
• abstract test cases
• Productivity Results
• Issues identified
• Conclusions and further work

3
Goals of the Project

• A local company had obtained equipment with a
  TTCN interface for automated testing of their
  telephone switching systems.
• Questions to answer
• Can TTCN be used for functional testing?
• Are the tools for these methods industrial
  strength?
• How much effort does this process take?
• Can MSCs, SDL, be used to speed up the process of
  test generation, and decrease the time to market?
• For this project, Telelogic Tau (version 3.3) was
  used as the tool set.

4
Reduced Time-to-Market
5
Project StepsIterative and Incremental Process
Develop MSCUse Cases
Create/Revise SDL model
Iterations for errors, added use cases

Produce MSCs by executing SDL model,and compare
with requirements MSCs
Create TTCN abstract test cases byexecuting SDL
model
6
Functional versus Conformance Testing

• Focus of conformance testing
• Exchange of protocol messages both for sequence
  and for content.
• Goal is to verify system interoperability, by
  conforming to a standard.
• Focus of functional testing
• Users view of the system, with respect to
  actions and reactions.
• More indirect than conformance testing.
• Needed to ensure that customer requirements are
  met.

7
Comparison of Test Actions and Reactions

• Test actions
• Functional testing go off hook on a phone
  set.
• Conformance testing send protocol message to
  the telephone switch as a result of going off
  hook.
• Dealing with a response to a test action
• Conformance testing check exact sequence and
  data format of messages that result from going
  off hook.
• Functional testing check that a set of responses
  occurs within a suitable time interval dial
  tone lamps, displays updated.
• Persistent events a dial tone comes on, and
  stays on.

8
Test Architecture

• Functional test view test environment provides
  a set of phone emulators that do user-oriented
  actions according to a script .
• Adaption layer provides set of valid TTCN
  messages to user (e.g. off hook).
• TTCN message is converted to actual protocol
  message.

TTCN Test Cases
9
SDL - TTCN - Test Environmentcorrespondence
Signal a, b, c
! a ? b ! c
...
a
...
b
...
c

• SDL message names are automatically extracted for
  use in the TTCN test cases.
• To match the test environment, the SDL model
  should use the names provided by the test
  environment adaption layer of the specific
  functionality to be tested.

10
Mapping the MSC modelto the test environment
phone1
user1
phone2
user2
switch
test tool phone command
test tool command to cause switch action
test tool command to detect switch response
test tool phone response
11
Creating the Message Sequence Charts
Environment
System

• The MSC has only two actors the environment
  (user) and the system.
• This models the interaction between the user(s)
  and the system.
• For testing, this models the exchange of messages
  between the test environment and the system under
  test.

userOffhook 1
userDialToneOn 1
userKey 1,2
userDialToneOff 1
userKeyTone 1,2
userKey 1,2
userKeyTone 1,2
userAlertingOn 2
userRingbackOn 1
userOffhook 2
12
High-Yield Scenarios

• Yield a rich harvest of bugs in requirements or
  design

Tester must think like a user and influence
designer to do likewise
13
Creating the SDL test model
Exceptions will produce high-yield test cases
Normal scenarios will produce low-yield test
cases
14
SDL Model considerations

• The model is not intended to replicate the entire
  system behaviour of the switch and phone sets.
• The model should reflect the test architecture.
• The model should include
• the functionality to be tested.
• additional behaviour extending to points of
  control and observation, for test tool commands.
• enough internal behaviour to execute model.
• The model does not need to include
• internal behaviour of the system that cannot be
  observed by test equipment, and is not part of
  the protocol.

15
Simulated Execution of the SDL model

• User can interact with the model, and view
  execution graphically.
• MSCs that record SDL model execution were
  produced.
• Execution MSCs were compared and verified to
  original requirements MSCs to check that the SDL
  model is correct.
• Coverage report was produced
• Created abstract test cases in Tree and Tabular
  Combined Notation (TTCN), generated directly from
  the SDL model.
• First iteration generate complete test cases.
• Second iteration (manually) identify test
  steps, and regenerate modular test cases.

16
Comparing Requirements and Execution MSCs
System structure added
States added
Digit timer added
Unexpired timer
17
Coverage-based Test Generation
transient state inputs
18
TTCN test case
19
Finding a bug with the SDT validator
Bug ringing should have been turned off when
call back activated
Unspecified message reception answering
the ringing phone
20
Productivity Results

• People working on SDL model
• One company employee with product knowledge, who
  had recently taken SDL and Tau tool training.
• Two graduate students, one with experience in
  telephony, SDL, and the Tau tool.
• One professor, who was consulted for reviews.
• Time to produce basic call model
• Initial version took about a week of intensive
  work.
• After several restructurings, call model
  stabilized after four weeks.
• All work (SDL model with 5 features, initial test
  cases, verified feature MSCs) took 3 months.

21
Productivity Chart
22
Issues for functional testing Event order

• Forced ordering of events on SDL transitions,
  TTCN test cases.
• Example Update of screen, lamps on a telephone.
• Order is irrelevant for user.
• Puts unnecessary constraints on designers.
• Creates unnecessary alternatives in test case.
• Potential remedy
• SDL unordered transitions.
• Allow TTCN to detect an interleaved set of
  responses
• ! off hook
• ? screen main lamp line lamp dial tone

23
Issues for functional testing Durations

• Example situation Test case generated with the
  intention of expiring the off hook timer
• phone1 ! off hook
• phone1 ? dialToneOn
• phone1 ? dialToneOff
• phone1 ? announcementOn
• The duration of the off hook timer is specified
  in the SDL model.
• The duration does not get carried into the test
  case.
• TTCN does have timer facilities, so this is not a
  language deficiency.
• If the announcement appears too early from users
  view point, this test case will still pass.

24
Conclusions and Further Work

• Advantages of the process
• User-oriented testing is based on what the
  customer wants.
• Simulated execution and validation can find bugs
  earlier in the process.
• TTCN test cases are generated faster (15
  minutes/test case) than with manual approach.
• Well-defined level of test coverage.
• Graphical documentation, in the form of MSCs and
  SDLs.
• A new project is underway that will
• Verify the scalability of the approach.
• Bring design groups into the process.
• Continue to refine the metrics for productivity
  measurement.

25
Early Detection of Design Faults

• Design Information
• Designers
• UCMs, LOTOS traces, LOTOS semantics

MSC Editor

High-Yield Iteration
Validator
Autolink, TTCNlink
SDL Design Model
design regression tests
Navigator