ART Automated Regression Testing EGOS Workshop 2005 ESOC, Darmstadt

1
ART - Automated Regression TestingEGOS Workshop
2005ESOC, Darmstadt
2
ART - Project Overview

• Project GoalAutomate Regression Testing of
  SCOS-2000
• Approach
• Reuse and customise the Intelligent Test
  Framework (ITF) approachResult of preceding GSP
  study
• Identify requirements specific to the ESA
  environment
• Enhance simulator in order to support test
  automation
• Develop automated Test Plans and Test Scripts

3
ART- Project Data

• TRP study initiated by ESOC/OPS-GIC
• Kicked-off 04/2005
• Technical Officer Eduardo Gomez
• Prime contractor Siemens Austria
• Space Business division
• Test Technology Support Centre
• Subcontractors
• Terma Space Division
• Critical Software

4
ART - Motivation

• Testing of distributed systems (like SCOS-2000)
  is difficult
• Majority of the test steps are performed manually
• ? time consuming
• ? tedious
• ? stress endurance tests are hard to do
• Test pass criteria must be checked visually
• ? time-consuming
• ? unreliable
• These difficulties become major cost drivers and
  source of problems with each test repetition
• ? Test automation is a must, especially with
  regression tests

5
ART Basic Concepts

• Intelligent Test Framework (ITF)
• Generic tools concepts developed in preceding
  GSP study
• Open to plug-in arbitrary Test Drivers
• Applicable to virtually any distributed and/or
  heterogeneous system

• SCOS-2000 as System Under Test (SUT)
• Analysis of existing manual Test Procedures
• Identify SUT-specific requirements (e.g.
  Simulator)
• Identify appropriate Test Drivers ?? Mapping of
  manual to automated Test Interfaces
• Reporting Requirements for Regression Testing

6
ART Framework Architecture
7
ART Framework principles

• Separation of
• Test Preparation vs.
• Test Execution time.
• Distinct levels of abstraction during test
  specification improves test maintenance
• Abstract Level corresponds to requirements?
  independent of implementation detail
• Implementation Level? encapsulates low-level
  detail

8
ART - Test Preparation
9
ART Test Preparation

• Different levels of abstraction
• Level of Requirements
• Derive implementation-independent Test Cases from
  requirements
• Link with Requirements ? Traceability
• Using a Test Specification and Generation Tool
• Level of implementation detail
• Develop test drivers scripts that implement the
  Test steps
• Iterative (similar to normal SW development)

10
ART - Test Preparation / TSGT

• Test Specification Generation Tool
• Graphical Editors for abstract specification
• Mapping and tracing of requirements
• Supports reusable building blocks
• Generation of Test Cases
• ART adopts (and extends) Siemens IDATG

IDATG Integrating Design andAutomated Test
Case Generation
11
ART IDATG Example
12
ART Test Case Generation

• IDATG Generation Features
• Specific Test Cases are generated from the
  abstract Test Specification.
• Generation algorithm covers all edges in the task
  flow graphs yielding C1 coverage (all edges)(?
  above example will yield 2 test cases)
• Test cases are exported to XML for
  execution(also COTS formats like WinRunner,
  Silktest - not used by ART)

13
ART - Test Execution
14
ART - Test Execution Model

• Problem
• SUT exposes many different test interfaces (many
  technologies parallel and distributed system)
• Parallel Execution requires coordination
• Solution
• ? Serialise test case into single-IF steps

15
ART - Test Execution

• Test Case consists of Test Steps
• Test Steps are Test Instructions parameters
• Test Commander executes Test Instructions
• Test Instructions invoke (external) Test Drivers
• Test Drivers interact with Test Interface of the
  SUT
• Test Drivers and Test Interfaces used in ART
• Python with OmniORBpy ? CORBA I/F
• Shell scripts ? File I/F
• Simulator control client ? Simulator control I/F

16
ART Test Commander

• More Features of TECO
• Load and compare against reference Test Result in
  order to detect regression deviations
• Fine-grained Test verdicts adopted from POSIX
  PASS, FAIL, ERROR, Expected FAIL, ..
• Test Result Test Report file generation
  (XML/HTML)
• Debugging features
• Breakpoints
• Step-by-step execution
• Out-of-order execution
• Edit test case
• Very useful for investigating problems
• Test case will be marked as UNRESOLVED in the
  report

17
ART Test Commander
18
ART Space Ground Simulator

• Based on SIMSAT Kernel
• Automated control using SGS Control Client
• Simulator is treated as a part of the SUT

19
ART Preliminary Results

• Project is currently on-going
• Experience gained so far
• ITF concepts are a solid base for ART
• Conversion of manual to automated testing is not
  straight-forward
• Automated building blocks are the base to improve
  the test coverage
• Model-View separation is a key to successful test
  automation
• Initial effort for test driver development is
  high, but paid back with increasing number of
  test repetitions

20
ART Project Contacts

• Agency
• Eduardo Gomez (ESOC)eduardo.gomez_at_esa.intTel.
  49(0)615190 2921
• ART Project Team
• Hannes KubrSiemens Austria / Space Business
  Unithannes.kubr_at_siemens.com Tel. [phone]
  42696
• Gerfried KramesSiemens Austria / Space Business
  Unitgerfried.krames_at_siemens.com Tel.
  43(0)51707 42628
• Stefan Mohacsi
• Siemens Austria / Test Technology Support
  Centerstefan.mohacsi_at_siemens.com Tel.
  43(0)51707 43417