TOBIAS a tool for test purposes design

1
TOBIASa tool fortest purposes design

• Laboratory LSR-IMAG Grenoble
• P. Bontron, O. Maury, S. Beghdadi, H. Bouldjedri,
  
• L. du Bousquet,Y. Ledru, M.L. Potet C. Oriat
• Workshop AFADL, June 2002

2
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

3
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

4
The COTE project

• A RNTL Project
• Partners
• SOFTEAM (tool objecteering) leader
• IRISA (tools UMLAUT/TGV)
• GEMPLUS (smart cards/software components)
• FT RD (OO technologies/components)
• LSR-IMAG (Software engineering)

5
The context of COTE

• Conformance testing of software components
• The component is tested against its UML
  specification
• Black box testing the internal behaviour of the
  product under test is hidden

6
Aim of the COTE project

• UML sequence diagrams provide an abstraction
  for tests cases
• provide tools for automatic executable test
  generation from UML test cases
• target three component technologies

7
Aim of the COTE project
8
Test case / Test purpose
An example A coffee vending machine
A test purpose a sequence of transitions to reach
A test case a complete path in the behavioural
specification
9
General structure of COTE
UML Specification
UML Test purposes
Test synthesis
Behaviour computation UMLAUT/TGV
UML Test cases
Automatic tests generation
corba
ejb
.net
10
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

11
Why TOBIAS ?

• Behavioural testing requires the specification of
  many tests.
• Test campaign gt (potentially) thousands of test
  cases
• UMLAUT/TGV one test purpose and one execution
  one test case
• New values gt creation of new test purposes
• Hugues MARTINs doctoral thesis (GEMPLUS)
•  Une méthodologie de génération automatique
  de suites de tests pour applets java-card. 
• A methodology for automatic generation of
  test suites for java-card applets.

12
Generation of test purposes

• Idea notion of high level test purpose
• (test pattern)
• gt abstraction of test purposes on objects,
  methods or values
• A test pattern gt a set of test purposes gt a set
  of test cases
• Significantly reduces the repetitive tasks of
  test definitions

13
TOBIAS
Test pattern
TOBIAS
UML Specification
Test synthesis
UML Test cases
14
Overview

• Context the COTE project (COmponent TEsting)
• Why TOBIAS ?
• TOBIAS description of the tool
• Test purpose generation
• Test case generation with VDM
• Smarter generation

15
Principles of TOBIAS

• TOBIAS provides
• help for test purposes design
• test pattern descriptions
• Idea
• instantiation of test patterns according to
  different criteria (objects, parameters )

16
TOBIAS inputs
UML specification
deployment diagram
class diagram
test patterns
TOBIAS
Test purpose in UML
17
A little example (1)

• UML class diagram

1
0..1
1..5
18
A little example (2)

• UML deployment diagram (one for each test
  campaign)

TestConfiguration
DrinkMachineVendingMachine
FrontHCI
CoffeeDrink
TeaDrink
CappucinoDrink
ChocolateDrink
TestorActor
ResetCancel
19
Some test purposes
5 test purposes
For - 4 different drinks - 1 cancel touch
(Reset.GiveChange) - 5 different coins 0.1,
0.2, 0.5, 1, 2 The number of different test
purposes is around (55²) x (4 1) 150 test
purposes
20
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

21
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

22
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

23
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

24
Principle of test patterns

• Provide the generation of test purposes according
  to different criteria
• the method parameters
• the objects
• the groups
• the number of group or method calls

25
An example
GroupAction Drink.ClickButton(),
Cancel.GiveChange()
Using the following test pattern
insertCoin(x)1..2 GroupAction

• TOBIAS generates the 150 test purposes previously
  showed

26
Advantages of using TOBIAS

• TOBIAS is useful to generate many test purposes
  ? generating many test cases using UMLAUT/TGV
• Generating test purposes for new values is easier
• TOBIAS can be used to generate test suites for
  VDM in combination with VDMTools

27
Using TOBIAS with VDM

• VDMtools provide an environment to specify an
  application, to execute test suites and to
  evaluate the test results
• The output of TOBIAS is directly used as a test
  suite.
• Using TOBIAS for the automatic generation of VDM
  test cases - VDM Workshop 2002 at Floc02

28
Using TOBIAS with VDM
VDM specification
Test pattern
UML specification
TOBIAS
VDM test suites
VDMTools

• Results
• fault detection
• code coverage

29
Using TOBIAS with VDM

• We did the following experiment
• specify an application managing groups of
  students
• manually specify a test suite (about 15 test
  cases)
• generate a test suite using TOBIAS (about 4300
  test cases)

30
Using TOBIAS with VDM

• We obtained the following results
• the generated and the manually written test suite
  have the same code coverage
• The development effort is similar
• the generated test suite detect more errors
• one unknown error
• one known error (from reading the specification)
• new activations of a known error
• the generated test suite takes significantly more
  time to be executed
• TOBIAS generates non conforming test cases
• TOBIAS generates redundant test cases

31
Smarter generation

• Preventing non conforming test purposes / test
  cases
• using more elements from the specification
• relations between classes
• pre/post conditions
• state diagrams
• Reducing the test suite size
• eliminating redundant test purposes / test cases
• defining constraints on the schemas

32
Using more elements
TOBIAS extract classes from the UML class diagram.
Class A
Class C
Class B
Class D
the class A calls the methods of the class C
A!C ? C!A ? B!C ? C!B ? A!B ? B!A ?
A!D ? D!A ? B!D ? D!B ? C!D ? D!C ?
A!A ? B!B ? C!C ? D!D ?
33
Using more elements
Relations between classes
A!C ? C!A ? B!C x C!B x A!B x B!A x
A!D x D!A ? B!D ? D!B ? C!D x D!C x
A!A x B!B x C!C x D!D x
34
Using more elements
Deployment diagram
35
Using more elements
Deployment diagram that gives the initial state
of the application
aA
cC
bB
dD
A!C ? a!c? C!A ? c!a? B!C x C!B x A!B x B!A x
A!D x D!A ? d!a? B!D ? b!d? D!B ? d!b? C!D
x D!C x
A!A x B!B x C!C x D!D x
36
Adding constraints

• The user can associate a constraint to each test
  pattern
• Constraints deal with parameters
• Constraints are used to filter test purposes /
  test cases

37
Adding constraints
GroupAction Drink.ClickButton(),
Cancel.GiveChange()
A test pattern insertCoin(x)1..2
GroupAction
- insertCoin(x_1) GroupAction - insertCoin(x_1)
insertCoin(x_2) GroupAction
?
A constraint
Number of selected test purposes
(59)x(41) 70 test purposes
38
Conclusion

• TOBIAS provides a new abstract level for test
  design the test patterns
• A test pattern gt a set of similar test cases
• To avoid combinatorial explosion, the user can
  associate constraints to the schemas
• TOBIAS can be used to generate test sequences
• TOBIAS will be used and validated on an
  industrial test case in the COTE project

39
Prospects

• Refining the test generation
• Deleting redundant
• Using pre/post conditions on operation to
  eliminate non-conforming sequences
• Helping the decision to stop test campaign
• Specification coverage by test patterns ? using
  more elements from the specification

40
THANKS !