Mattias Gransson Theis Meggerle

1
Mattias Göransson Theis Meggerle

• Test Process Improvement Based on the TPI Model

2
Agenda

• Why TPI?
• Testways Approach
• The Model and Key Elements
• Some Examples
• What does a Process View Imply?
• A Wider View on Process and TPI
• Borderline issues?

3
Why Bother with Test Process Improvement?

• Testing is a necessary prerequisite for
  successfully building and implementing
  Information Technology Systems!
• BUT Testing is often regarded as a necessary
  evil. Unfortunately, in many cases this opinion
  is justified!
• In only a few organizations does management spend
  the appropriate amount of time needed to manage
  such a large sub-process (25-50 of the total
  project budget).
• Structuring the Test Process can solve many of
  the problems related to Testing Time, Budget,
  and Quality.

4
TPI Survey 2004

• 61 of the people responded that implementation
  of TPI lead to better software quality e.g. less
  failures in production.
• 83 of the people responded that implementation
  of TPI lead to (much) better testing control.
• 77 of the people responded that the return on
  the TPI investment is (very) good.
• The two main factors that had a positive
  influence on the implementation were management
  commitment (23) and involvement of people within
  the test process (23). As a negative influence,
  18 answered unrealistic expectations.
• 92 of the people responded that the contribution
  of the TPI-model to the improvement process is
  (very) good.

5
The Testway Approach

• Assess the current situation.
• Create and document long term, as well as short
  term, goals for test process improvement.
• Report suggested improvements as well as the
  dependencies between these improvement
  suggestions.

6
The Testway Approach Building Blocks

• TPI Model (Test Process Improvement Model).
• Features from TPI Automotive version 1.0.
• Requirement process evaluation.
• Configuration and Change Management process
  evaluation.
• Risk Management.

7
The TPI Model

• The first version of the TPI Model was publicized
  1997 by Tim Koomen and Martin Pol.
• The TPI Model offers a frame of reference for
  determining the strong and weak points of a
  organizations current Test Process.
• The TPI Model also supports the work of creating
  the setting up of specific and realistic
  proposals for the improvement of this Test
  Process in terms of lead time, cost, and quality.

8
TPI Model Key Elements
9
Maturity Matrix
10
Example - Key Area Test Strategy

• The Aim of the Test Strategy is Finding the most
  important defects as early and as cheaply as
  possible
• The Test Strategy defines which requirements and
  (product) risks are covered by what tests. The
  better each test level defines its own strategy
  and the more the different test level strategies
  are adjusted to each other, the higher the
  quality of the overall test strategy.

11
Example - Checkpoints level A

• A motivated consideration of the product risks
  takes place, for which knowledge of the system,
  its use and its operational management is
  required.
• There is a differentiation in test depth,
  depending on the risks and, if present, the
  acceptance criteria not all subsystems are
  tested equally thoroughly and not every quality
  characteristic is tested (equally thoroughly).
• One or more test specification techniques are
  used, suited to the required depth of a test.
• For retests, also, a (simple) strategy
  determination takes place, in which a motivated
  choice of variations between test solutions
  only and full retest is made.

12
Example - Dependencies

• Test specification techniques, lvl A, Informal
  techniques.
• Techniques are necessary to make concrete the
  choice for lighter or heavier testing.
• Commitment and motivation, lvl A, Assignment of
  budget and time.
• The Strategy needs to be discussed with the
  customer of the test, because it is strongly
  related to the time and mony needed.

13
Example - Improvement suggestions

• Determine quality characteristics.
• Test focus and report focus.
• Determine relative importance of quality
  characteristics.
• How to distribute the test effort.
• Break down into subsystems
• Total system and subsystems.
• Determine relative importance of system parts.
• How to distribute the test effort.
• Determine the measuring techniques to be used
• Measuring and in particular test specification
  techniques are selected.

14
TPI Automotive version 1.0

• TPI Automotive version 1.0 is an improved TPI
  Model developed for the car industry. It is
  currently under evaluation.
• If the evaluation result is positive the
  enhancements will be incorporated into the
  Regular TPI Model.

15
Our Approach The Process

• What is a process?
• The view of the Test process
• Scope
• The assessment

16
Definition of Process A mechanistic view

• A process is a collection of related activities
  that transform an input to create an output.
• Simple
• Reduced to a machine
• Creates no value to a receiver (should it exist?)
• No receiver
• One-shot

17
Definition of Process The Customer

• A process is a chain of activities that in a
  repeating flow creates a value for a customer.
• Value to a customer
• Repeating
• To a certain part still a closed system
• A value it not especially precise

18
Definition of Process Social system

• A process is a repetitive network of related
  activities that uses information and resources to
  transform input to output, from the
  identification to the satisfaction of a
  customers need.
• A network of performed activities.
• A more precise scope with a distinct beginning
  and end that takes its point of departure in the
  fulfillment of a customer need.
• A social system that depicts the necessity of
  information and human resources.

19
Definition of Process A representation
20
Process-oriented view on TPI
21
A more process-oriented view

• A wider view on the test process
• Includes the relation to processes that affect
  the test process
• Alignment with ISO-90012004 (customer, process,
  resources) and other system theories
• It is a Social System!

22
Requirement Process Border Issues

• The relationships between the Requirement Process
  and the Test Process can be defined in different
  ways in different organizations. These
  relationships have to be made clear.
• The earlier a defect is detected the cheaper and
  faster it is to correct it.
• A well working requirement process allows early
  static tests which saves both time and money as
  well as early preparation of the dynamic test.
• A well working requirement process also assures
  that the tests addresses the right requirements,
  at the right depth, and at the right time.

23
Requirement Process Border Examples?

• Continuous learning
• Requirement workshop
• Collaborative development of requirements
• Static tests
• Walkthroughs
• Technical Reviews
• Management Reviews

24
Configuration and Change Management Process
Border Issues

• The relationships between the Configuration and
  Change Management process and the Test Process
  can be defined in different ways in different
  organizations. These relationships have to be
  made clear.
• Changes to the scope will occur and the Tests
  Process have to address these changes.
• Changes to artifacts will occur and the Tests
  Process have to address these changes.

25
Configuration and Change Management Process
Border Examples

• Test Base
• Release Notes
• Release Walkthroughs

26
Risk Management

• Risk Management is vital in order to prioritize
  the work, address the right requirements at the
  right depth, and at the right time.
• A widely used method is FMEA (Failure Mode and
  Effect Analysis).

27
Other Examples

• Implementation process
• Working for Quality

28
When the Assessment is Done the Work Begins!

• The Test Process Improvement Assessment points
  out what needs to be improved and in what order
  changes should take place.
• It also gives suggestions on how to improve
  specific Key Areas.
• BUT it does not give you the holistic plan or
  guides the implementation work.