UniTesK: Model Based Testing in Industrial Practice

1
UniTesKModel Based Testing in Industrial
Practice

• Victor Kuliamin
• Alexander Petrenko
• Alexander Kossatchev
• Igor Burdonov

Institute for System Programming Moscow, Russia
2
Outline

• Origin of UniTesK Method
• UniTesK Manifesto
• UniTesK Solutions
• Case Studies

3
Origin of UniTesK Method

• 1994 1996ISP RAS Nortel Networks project
  onfunctional test suite development for Switch
  Operating System kernel
• Few hundreds of bugs found in the OS kernel,
  which had been in use for 10 years
• KVEST technologyAbout 600K lines of Nortel code
  tested by 2000
• BUT Failed to be introduced in Nortel processes

4
UniTesK Test Construction
Testing Model
Behavior Model
Coverage Model
System under Test
Test Input
Behavior Correctness Checking
5
Test Development Process
Interface Definition
Design
Requirements
Specification Development
Behavior Specifications
Scenario Development
Test Scenarios
Test Quality Criteria
Mediator Development
Mediators
System under Test
Test Execution
Test Reports
Test Results Analysis
6
Engineering Problems
To increase productivity, but not to loose
flexibility

• How to simplify transformation of requirements
  into formal specifications?
• How to minimize human involvement in test
  development, but make it possible in necessary
  points?
• How to support a large variety of problem domains
  and test completeness criteria?
• How to decouple tests and implementation, but
  keep them working together?
• How to measure testing quality without
  implementation?

To make easier their usage for ordinary software
engineers
To make relation between them more clear
To make possible early start of test development
To make specifications and tests more reusable
7
UniTesK Manifesto

• Maximum simplification of MBT techniques
• Automation where possible
• Accommodation to current industrial practice
• Integration with widely used tools
• Integration with widely used languages

8
Tools

• J_at_T
• CTesK
• Ch_at_se

9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
UniTesK Test Architecture
Test action construction
Test Engine
Test action construction
Test Action Iterator
Test Scenario
Specification
System under Test
16
Example

• Client Data Management System
• A number of clients
• Clients can be related as parent-subsidiary
• Client can have only one parent, but may have
  several subsidiaries
• No cycles along parent-subsidiary links
• When parent is removed, all its subsidiaries
  should be removed

17
Example Interface

• class ClientManager
• Client addClient ( String name )
• boolean removeClient ( Client client )
• class Client
• boolean addSubsidiary ( Client client )

18
Example Client Data

• class Client
• String name
• ClientSpecification parent null
• HashSet children new HashSet()
• invariant ParentChildLink()
• if(parent ! null !parent.children.contains
  (this)) return false
• Iterator j children.iterator()
• while(j.hasNext()) if(((Client)j.next()).paren
  t ! this) return false
• return true
• invariant NoCyclesAlongLinks()
• Client r this
• do r r.parent if(r this) return
  false while(r ! null)

19
Example addClient() Method

• class ClientManager
• specification Client addClient ( String name )
• updates clients.?
• post
• if(name null clients.containsKey(name)
  )
• // Client cannot be created
• return addClient null
  clients.equals(_at_clients.clone())
• else
• // Client can be created
• HashMap oldClients (HashMap)clients.clon
  e()
• oldClients.remove(name)
• return addClient ! null
  addClient.name.equals(name)

20
Test Coverage Criteria
post
branches Single branch Single branch Single branch
predicates ( a b ) ( a b ) !( a b )
disjuncts a !a b !a !b
Single branch
branches
if(a b)
if(a b)
if(a b)
predicates
( a b )
!( a b )
disjuncts
a
!a !b
!a b
branch Single branch
branch Single branch
return

21
Example addClient() Method

• class ClientManager
• specification Client addClient ( String name )
• updates clients.?
• post
• if(name null clients.containsKey(name)
  )
• branch "Client cannot be created"
• return addClient null
  clients.equals(_at_clients.clone())
• else
• branch "Client can be created"
• HashMap oldClients (HashMap)clients.clon
  e()
• oldClients.remove(name)
• return addClient ! null
  addClient.name.equals(name)

22
Automaton from Coverage Goals
parameters
operation domain
2
3
coverage goals
1
states
23
Implicit Automata Description

• Implicit specifications cannot be resolved
• To decrease effort they should not be resolved
• Huge automata can be described shortly

24
Test Scenarios

• User can describe an automatonimplicitly in the
  form of Test Scenario
• Functions
• Provide the current state identifier
• Provide the next admissible stimulus in this state

25
Test Scenario State

• scenario class ClientManagerTestScenario
• ClientManager target
• AbstractGenState getGenState()
• IntToIntMapGenState genstate new
  IntToIntMapGenState()
• int n 0, m 0
• Iterator j objectUnderTest.clients.keySet().
  iterator()
• while(j.hasNext())
• String s (String)j.next()
• n Integer.parseInt(s)
• if(((Client)target.clients.get(s)).parent
  null) genstate.setValue(n, 0)
• else
• m Integer.parseInt(((Client)target.clien
  ts.get(s)).parent.name)
• genstate.setValue(n, m)

26
Test Scenario Stimuli

• scenario class ClientManagerTestScenario
• scenario boolean addClient()
• iterate(int i 0 i lt numberOfClients i
  )
• String name (i 0)?(null)("" i)
• if( target.precondition_addClient( name ) )
  
• target.addClient( name )
• return true
• scenario boolean removeClient()
• iterate(int i -1 i lt numberOfClients i
  )
• Client client (i lt 0)?
  ClientMediator.create(new ClientImpl("1"))
• 
  (Client)target.clients.get(("" i))

27
Test Execution First Stage
Test Engine
Test Scenario
28
Factorization
29
Testing Concurrency Modeling
?a
?b
How to model responses on all possible
multistimuli?
?a,b
?a,a
?b,b
?a,a,a
?a,a,b
Plain concurrency axiom reaction on
multistimulus is the same as on some sequence of
its stimuli
30
Asynchronous Reaction Specification

• specification reaction UDPPacket UDPResponse ( )
• pre return !ModelUDPPackets.isEmpty ( )
• post
• return
• (_at_ModelUDPPackets.clone()).contains (
  UDPResponse )
• !ModelUDPPackets.contains ( UDPResponse
  )

31
Providing Concurrent Inputs
s11
s12
s13
s14
System under Test
s21
s22
s23
s24
s31
s32
s33

• Multisequence is used instead of sequence of
  stimuli

32
Collecting Asynchronous Reactions
11
21
31
r12
r11
System under Test
12
22
32
r23
r22
r21
23
r33
r32
r31
33
Time

• Reactions form a partially ordered set

33
Evaluation of Reactions
Stimuli
Reactions
Plain concurrency axiom
?
34
UniTesK Test Architecture, 2
Test input construction
Test Engine
?
?
Test Action Iterator
Synchronization Manager
Serializer
Oracle
Mediator
System under Test
Reaction Collector
35
UniTesK Tools

• J_at_T (C Link)Java (C) / NetBeans, Eclipse
  (plan)
• CTesKC / Visual Studio 6.0, gcc
• Ch_at_seC / Visual Studio .NET 7.1
• OTKSpecialized tool for compiler testing

36
Technology Transfer

• Forms
• Training
• Pilot projects
• Joint projects
• Project supervising
• Successful transfers
• Lanit-TercomCTesK, Feb 2002
• Luxoft (IBS group)J_at_T, Jul 2003
• IntelOTK, Nov 2003

37
Case Studies

• IPv6 implementations - 2001-2003
• Microsoft Research
• Mobile IPv6 (in Windows CE 4.1)
• Oktet
• Intel compilers - 2001-2003
• Web-based client management system in bank
• Enterprise application development framework
• Billing system

38
References

• V. Kuliamin, A. Petrenko, N. Pakoulin, I.
  Bourdonov, and A. Kossatchev. Integration of
  Functional and Timed Testing of Real-time and
  Concurrent Systems. Proc. of PSI 2003. LNCS,
  Springer-Verlag, 2003.
• V. Kuliamin, A. Petrenko, I. Bourdonov, and A.
  Kossatchev. UniTesK Test Suite Architecture.
  Proc. of FME 2002. LNCS 2391, pp. 77-88,
  Springer-Verlag, 2002.
• A. K. Petrenko, I. B. Bourdonov, A. S.
  Kossatchev, V. V. Kuliamin. Experiences in using
  testing tools and technology in real-life
  applications. Proceedings of SETT01, India,
  Pune, 2001
• I. B. Bourdonov, A. S. Kossatchev, V. V.
  Kuliamin. Using Finite State Machines in Program
  Testing. "Programmirovanije", 2000, No. 2 (in
  Russian). Programming and Computer Software, Vol.
  26, No. 2, 2000, pp. 61-73 (English version)
• I. Bourdonov, A. Kossatchev, A. Petrenko, and D.
  Galter. KVEST Automated Generation of Test
  Suites from Formal Specifications. Proceedings of
  World Congress of Formal Methods, Toulouse,
  France, LNCS, No. 1708, 1999, pp. 608-621
• http//www.ispras.ru/groups/rv/rv.html

39
Contacts

• Victor V. Kuliamin Alexander K.
  Petrenko
• E-mail kuliamin_at_ispras.ru, petrenko_at_ispras.ru
• 109004, B. Kommunisticheskaya, 25
• Moscow, Russia
• Web http//www.ispras.ru/groups/rv/rv.html
• Phone [phone]
• Fax [phone]

40

• Thank you!