An Automated Test Strategy Based on UML Diagrams

1
An Automated Test Strategy Based on UML Diagrams

• F. Basanieri , A. Bertolino , E. Marchetti ,
  A. Ribolini , G. Lombardi , G. Nucera
   PISATEL LAB, IEI-CNR, Pisa, Italy 
  ERICSSON LAB ITALY

2
PISATEL_LAB
The joint conduction by the academia in Pisa and
ERI of strategic research projects on software
for telecommunications, selected on an annual
basis
3
Pisatel Objectives

• Main areas of interest are
• software engineering methods and tools,
• software architectures, distibuted systems,
  networks, real time systems, and protocols.
• Currently the active projects are
• Integrating Wireless, Wireline and Internet
  Networks
• Analysis and Development of Real Time Software
• Quality And Validation of Software Architecture
• More details
• http//www.iei.pi.cnr.it/ERI/

4
An Automated Test Strategy Based on UML Diagrams
5
Objectives

• Use for test planning the same UML diagrams
  developed (e.g., by Rational Rose) for design and
  analysis
• Tool-supported generation and maintenance of an
  updated and strategic test plan, as the design
  evolves.

6
(Realistic) Preconditions

• It is plausible that .
• in early design phases many of the UCs are
  defined at a high level and only few of them are
  properly refined.
• there is not yet the specification of every
  possible scenario, but only few SDs already exist
  that realize the defined UCs.
• different people or teams take part to the
  development, everyone familiar with a different
  component or functionality.

7
Fundamental Guidelines

• FG1 use the UML diagrams developed during
  specification and design, without imposing to the
  UML designers any additional formalism, or ad-hoc
  effort specifically for testing purposes.
• FG2 test planning for integration testing must
  cope with high level diagrams, even though not
  yet completely specified, or refined

8
Trade-off

• UML is a notation and does not impose a specific
  process.
• A test strategy needs to refer to a systematic
  and documented design process
• We want to impose minimal requirements
• Use Cases (UCs) organized into a hierarchy
• Express relevant scenarios as Sequence Diagrams
  (SDs)

9
Requirements

• The reference documentation is derived by
  collecting together and automatically organizing
  the diagrams developed during the design phase
  without additional manual effort of
  formalization.
• Inconsistencies and incompleteness are
  highlighted (Design weaknesses)

10
Cow_Suite approach

• The processing of the Cow_Suite approach starts
  by analysing the project description, and
  proceeds in parallel with the design and analysis
  refinement steps.
• the internal description of the project
  model, given by Rational Rose in a .mdl file

11
Cow_Suite tool

• The Cow_Suite tool consists of three elements
  working in combination

12
Cow_Suite Scheme
Test Procedures list
13
Main steps

• Derive a Reference Tree
• Deduce the Critical Profile
• Select the Test Strategy
• Derive the Test Cases
• Implement the Test Procedures

14
1.1 Tree Derivation

• Use Cases (UCs) and Sequence Diagrams (SDs) of
  the .mdl project design realized with Rational
  Rose are automatically organized by Cow_Suite
  tool in a sort of hierarchical (pseudo-)tree,
  starting from the Main Use Case Diagram, UCD.

The case study is ArgoUML, a tool supporting OO
design with UML
15
1.2 Tree Derivation

• Looking at the tree the Cow_Suite users can get a
  complete view of the status of the specification
  of the various functionalities
• The tree represents an ordered and organized
  documentation, that is continuously updated with
  the latest changes and progressively completed

16
2.1 Deduce the Critical Profile

• Systems are naturally composed of different parts
  realizing one or more functionalities, each with
  different importance for the overall system
  performance.
• The effort devoted to the test of different
  functionalities must be planned and scheduled
  considering their relative importance
  (risk-based testing, reliability guided testing,
  etc).
• Following level by level the tree structure of
  the UCs and SDs, the strategy allows the people
  in charge of the development of one or more
  functionalities to simply annotate the
  corresponding node in the tree with a number
  (weight).

17
2.2 Weighted tree derivation

• The user annotates every node with a value, leaf
  weight, representing the relative importance of
  this node (be it a UC or a SD) with respect to
  the other nodes at the same level in the tree

18
2.3 Weights management

• The weight value belongs to the 0,1 interval
  and must be assigned so that the sum of the
  weights associated to all the children of one
  node is equal to 1.
• The weights are user modifiable as the
  development proceeds.

19
Integration Testing vs Integration Stage

• To realize a complete system functionality, it is
  generally necessary to execute several actions
  realizing lower level functionalities
• A system functionality is realized by the
  different components interaction.
• Integration Testing testing interactions between
  system components (processes, packages,
  subsystems ) at an appropriate level of
  abstraction.
• Integration Stage is useful because generally
  the functionalities are not specified at the same
  level of detail and with the same number of UCs
  and SDs

20
2.4 Integration Stage Selection

• The first integration stage is represented by the
  main UC and the SDs (if any), which are children
  of this node (hence they are at level 2 of the
  tree).
• The i-th integration stage is represented by the
  UCs positioned in at i-th level of the tree and
  every SDs, children of these nodes, situated at
  i1-th level.
• NOTE the SDs at level i1 represent the
  interaction between the different components
  that realize the functionalities described in the
  UCs at i-th level.

21
Example 4th Integration Stage

• The Cow_Suite user selects the integration stage
  (4) at which the test is conducted.
• The nodes that are considered by Cow_Suite tool
  for the subsequent evaluations are all and only
  those belonging to the 4th integration stage plus
  all the tree leaves existing at higher levels
  (3rd, 2nd, 1st).

22
2.5 Final weights assignment

• The weights assigned to each node are used to
  define a relative importance factor, in terms of
  how risky is that node and how much effort should
  be put to test it.
• Considering every node, from the root down to the
  elements belonging to the integration stage
  considered, its final weight is computed as the
  product of all the nodes weights on the complete
  path from the root to this node.
• The final resulting weight is an element of
  discrimination for choosing amongst the tests to
  execute.

23
3. Cow_Test Strategy

• Two different situations are considered in test
  cases planning
• 1 Cowtest_ing with fixed number of tests
• A certain budget is available, which is
  translated in terms of number of tests, NT, to
  execute.
• In this case the tool select the most adequate
  selection of NT tests from among the (many)
  various test cases that could be potentially
  conceived.

24
3. Cow_Test Strategy

• 2 Cowtest_ing with fixed functional coverage
• A certain percentage of functionalities must be
  covered.
• In this case by the tool is possible to define in
  advance which are the functionalities to be
  covered and the minimum number of tests to
  execute.

Functional Coverage 90
25
3.1 Select the Test Strategy
26
Cowtest_ing with fixed number of tests

• Using the resulting final weight at the chosen
  integration stage, nw, for each SD the number of
  Test Procedures, NTtest,
• can be easily derived

27
3.2 Select the Test Strategy
28
Cowtest_ing with fixed functional coverage

• Considering the final weights, nw, of every node
  the tool can
• select the functional test cases to be run, by
  ordering in decreasing manner the nw100 values
  and summing them, starting from the heaviest one,
  until the fixed coverage, C, is reached.
• Calculate the minimum number of tests with
  respect to the fixed coverage percentage.

29
Notes

• To make a more practical prediction in terms of
  man/hours, or required budget for testing, it
  would be necessary to be able to estimate the
  cost of the different test cases.
• An alternative application of Cowtest is
  considering a certain budget, B, planned for the
  testing instead of the fixed number of test NT.
  In this case is possible to evaluate the
  relative cost, b, for the testing of every leaf
  of the fixed integration stage.

30
Cow_Suite Scheme
31
3.3 Sequence Diagrams Selection

• For each fixed integration stage, a weighted
  subtree is derived according to the choosen test
  criterion. Only the relative SDs are considered
  for Test Case generation.

32
4. Test Case Derivation

• The tool uses the UIT method, Use Interaction
  Test, to derive the Test Cases.
• The method is mainly based on the SDs
  (particularly objects, messages and parameters)

33
4.1 Use Interaction Test methodology

• A method to systematically derive Integration
  Test suites from UML diagrams.
• Inspired at large by the Category Partition
  method
• it defines classes of functional equivalence by
  selecting significant values (choices) of
  relevant categories (parameters or data)
• Incremental test strategy (over the Use Case
  organizational structure)

34
4.1.1 Sequence Diagram analysis

• Horizontal axis shows a set of objects, Test
  Units, that interact through messages.
• Those are system units separately testable (class
  testing) for exercising a possible use of system.
• Example
• DecisionModel, Decision, GoalModel, Goal.

35
4.1.2 Sequence Diagram analysis

• Vertical axis shows
• the exchanged messages, Interactions Categories,
  involved in object interactions
• their parameters and relevant inputs, Settings
  Categories.
• DecisionModel
• Settings _decisions
• Interaction DecisionModel() getDecisions()
• Decision Interactions Decision(nameString,prior
  ityint) GetName(), GetPriority() ............

36
Additional Information Class Diagram
Complementary analysis of the Class Diagram
description (mainly used for searching the
Settings Categories). These are attributes (or a
subset of them) of a class (and the corresponding
SDs object) that affect object interactions and
are represented by input parameters used in
messages or data structures.
37
4.1.3 Search of Message Sequences
MessageSequences set of Messages (in temporal
order), involved in a SD, used by objects to
define and elaborate specific functionalities. Ex
ample getDecisions() getPriority()
38
4.2 Test Case Derivation

• For each traceable Message Sequence a Test Case
  is generated. It contains the list of all its
  Settings and Interactions Categories and their
  values.

39
Conditions

• A Test Case (TC) can be associated to some
  feasibility conditions formally expressed (notes
  or specifications tab).
• In this case TC is divided in different subcases.

40
Number of Test Procedures

• The final weight of every SDs is used to
  automatically derive the number of Test
  Procedures associated to each Test Case.

41
Cow_Suite Scheme
Test Procedures list
42
5.1 Test Procedures Specification

• Test Specification
• For each identified category, we consider all its
  possible values and constraints (choices).

43
5.1.1 InteractionsCategories Specification

• Interactions
• Categories objects interactions, exchanged
  messages.
• The tool implements two possible choices for the
  user
• the default values,
• the user values

44
State Diagram
If a State Diagram is linked to a specific Test
Unit (class), further information can be found
about the relevant configurations of
InteractionCategories
State diagram of the Critic class
45
5.1.2 Specification of SettingsCategories

• Settings
• Categories
• parameters and inputs relevant for the Test Unit.
  
• The tool implements two possible choices for the
  user
• the default values,
• the user values.

46
5.2 Test Procedures Implementation

• A Test Procedure is automatically generated from
  a Test Specification, for each possible
  combination of choices of every category involved
  in a Messages Sequence.

47
Test Procedures an example

• Test Specification
• DecisionModel
• Settings
• _decisions
• Naming
• Storage
• Modularity
• Inheritance
• Stereotypes
• Relationships..
• Interactions
• getDecisions
• Opening a new file
• Opening a saved file
• After a modification and before saving
• After a modification and after saving
• DecisionModel()
• Class constructor

Test Procedure getDecisions() getName()
Opening a saved file _decisions
Naming Note Such a Test Case must be repeated
for all possible values of _decisions
48
Constraints on Choices

• Some choices combinations could result
  contradictory or meaningless. Following the
  Category Partition methodology, constraints must
  be added to the SettingCategories choices.
• Constraint are specified using
• Properties, that can be assigned to certain
  choices and testing for by other choices.
• IF Selector, is a conjunction of properties that
  were previously assigned to other choices.

49
Example

• SettingsCategories
• pattern size
• empty Property Empty
• single character Property Not Empty
• many character Property Not Empty
• quoting
• pattern is quoted Property quoted
• pattern is not quoted if Not Empty
• pattern is improperly quoted if Not Empty

50
SettingsCategories Specification Constraints

• Test Specification
• Decision(_name, _priority)
• _name
• Naming if prioritylt5
• Class Selection if priority0
• Storage
• Inheritance
• Modularity..
• _priority
• 0 Property priority0
• 1
• 2
• 3
• 7 Property not prioritylt5

51
SettingsCategories Specification Constraints

• Test Procedure
• DecisionModel()
• Opening an existing document
• Decision( _name, _priority)
• _name
• Naming if prioritylt5
• _priority
• 7 Property not prioritylt5

Test Procedure DecisionModel() Opening an
existing document Decision( _name,
_priority) _name Naming if prioritylt5 _prior
ity 4
X
ok
52
5.2 Test Suite

• The set of all generated Test Procedures is named
  Test Suite

53
Note

• The Test Procedures, as derived by the tool, are
  not directly executable (? Test Scripts). The
  derived Test Procedures have to be executed and
  logged by a test driver.
• CowSuite goal is
• giving automatic and systematic support to
  construct a meaningful Test Suite.
• It is not to directly execute the tests.

54
Remarks

• The degree of detail of the Test Suite depends
  directly on
• the design detail level
• granularity of the inserted values (selection of
  the constraints and choices of the UIT
  categories).
• Currently the final result is a text document
  containing the Test Suite.
• In future different format (e.g. XML), that can
  be accepted in input by the adopted test driver.

55
Features under Construction

• Two different situations can occur during the
  tree construction

56
Disconnected tree

• Analysing the .mdl file, the structure derived by
  the Cow_Suite tool is not a connected tree. Some
  holes in the design could be highlighted if
  existing pieces of a system have not a logical
  connection.
• More than one disconnected subtrees are derived
  by the Cow_Suite tool and visualized to the user.

Possible solution inserting dummy nodes and arcs
where necessary (according to the logical
relation between the disconnected trees).
57
Subsystems Reuse

• The reuse of a same subsystem in different parts
  of the system design causes anomalous
  connections, i.e. cycles, in the derived
  structure.

58
Subsystems Reuse

• Possible solution
• the first time a subsystem is encountered
  associate to it a node and derive for it a
  corresponding subtree, as usual
• for the following occurrences of the same
  subsystem, add to the pseudo-tree a dummy node
  referring back to the subtree already derived.

59
Conclusions
Systematic and automated support for test
planning and test case generation, based on the
same UML diagrams developed for design and
analysis
OK
Tool integrated with the design tool (e.g.
Rational Rose)
OK
60
Future Work
Test case derivation (UIT) to be validated on ERI
case studies
To be continued
Practicality of a test strategy based on manual
weights (COW_test)? How to assign the weights?
To refine
How to use Cow_Suite within a multi-phase test
process (different test stages)?
To do
61
Questions?
62
Cow_Suite Architecture