Title: Who wrote the article
1(No Transcript)
2Who wrote the article ?
Alan Hartman
Kenneth Nagin
Andrei Kirshin
3What is it all about ?
- Executing abstract tests on distributed software.
- Deriving tests from specifications abstract
tests. - Tester must manually convert those tests into
their concrete executable form. - Could be used to verify distributed software.
- The same abstract tests can be extended to verify
issues related to concurrency, stress and input
data variations.
4Main issues
- Introduction.
- Test execution environment design.
- Abstract tests extended with object partitioning.
- Industrial experience.
- Future work.
5Introduction
6- What is an abstract test ?
- A test which describes test logic without
concerning itself with platform or language
issues. - In order to run the test, the execution
environment must deal with the messy details of
concrete execution. - The abstract tests may be reused and extended to
verify additional software requirements such as
concurrency.
7- In recent years, software modeling has enjoyed
tremendous popularity due to the adoption of
object oriented models. - Abstract tests are generated from such software
models.
8- However, the generation of test suites from a
specification based modeling has yet to gain
industrial momentum. - Mainly because most current test execution
environments were not designed to run the
generated tests.
9System Under Test
- Abstract model of System Under Test (SUT) -
10System Under Test cont.
- SUT may be perceived of as a set of objects.
- The SUT is not restricted to any programming
language or operating platform. - The abstract test describes the objects expected
behavior, state, and output in response to a
stimuli from the tester. - The concrete implementation of the SUT runs
within different process address space.
11Test Execution Environment
- Software testing environment are applications
that assist testers in executing tests and
collecting results against the SUT. - Testing environments should run under the same
operating system and use the same programming
language in order to interact with the SUT. - The cost of testing environments is a significant
part of the overall development cost.
12Test Execution Environment cont.
- The cost and complexity of the testing
environment increases when the SUT is
distributed. - The main problem is that distributed software may
run in different host environments, separated by
a network and its components may be coded in
different programming languages.
13Test Execution Environment cont.
- Example of distributed software
Unix C
Linux Java
Windows C
TCP/IP
14Test Execution Environment cont.
- The main goal of the test execution environment
is to provide an infrastructure to support the
heterogeneous application environment and
languages. - Of course, it must scale down to support more
simple non-distributed software testing.
15Abstract Test Execution
- An abstract test specifies a sequence of tester
stimuli to the SUT, observation of SUT responses,
and the assignment of a verdict based on the
observations. - An abstract test is designed to exercise a
particular execution sequence or verify
compliance with a specific requirement.
16Abstract Test Execution cont.
- Abstract tests have the advantage that they
express the testing logic without getting bogged
down in the details of how to execute the test. - Thus, the abstract test must be translated in
order to execute it on the SUT. - Abstract stimuli is mapped to method invocation,
mouse clicks, or system commands. - Abstract observations are mapped to actual value
checking.
17Abstract Test Execution cont.
- The test execution environment has 2 inputs
- Abstract test, telling what to execute against
the SUT. - Test execution directives, telling how to execute
the abstract test.
Test Driver
Abstract Test
Test Exec. Directives
18Test Execution Environment Design
19- The test execution environment design supports
the basic functionality for testing distributed
systems.
20- The main components of the test execution
environment are - Test Driver manages the test execution.
- Host Manager replicated on each target host
(each host can have different operating system). - Process Controllers interact directly with the
SUT objects. - The test driver and SUT need not be coded in the
same programming language.
21- The test driver communicates with host managers
and process controllers in order to accomplish
the following tasks - Create, destroy and locate process controllers.
- Create and destroy SUT objects.
- Control input to the SUT objects.
- Observe output and state of the SUT objects.
via process controllers
via host manager
22- Only the test driver understands the syntax and
semantics of the abstract test cases. - All other components handle real world
interactions. - This separation of concerns makes the test
execution environment design well-suited for the
testing of distributed software.
23- The test driver is implemented only once since it
has no SUT platform or language dependencies. - Host manager and process controller
implementation is dependent on the target SUT. - Host managers must be implemented in all target
SUT platforms (windows, Linux, ). - Process controller runs in the same process as
the SUT object and they are both coded in the
same programming language.
24Abstract Tests Extended with Object partitioning
25Partitioning
- The SUT consist of different entities whose
behavior is described in the abstract test. - The tester can partition the abstract test into a
set of executable objects.
- The object partitioning idea is everywhere.
26- Motivation for object partitioning
- Extend the test usage to verify issues related to
concurrency, stress and input data variations.
27- The test modeling language does not have to be
object oriented. - However, it is required that the abstract stimuli
and observable responses be identifiable so they
can be mapped to concrete objects.
28Example
- Consider an SUT that is client/server
application. - Server role - connect its clients to a database
of accounts. - Unique keys index the database of accounts.
- Client role create, remove, or update accounts
based on the keys.
29Example cont.
- We illustrate a simple test case
- Start the application server.
- Observe that no exception occurred.
- Create an account with key1.
- Observe that no exception occurred.
- Create another account with key1.
- Observe that an exception is thrown.
- Stop the application server.
30Example cont.
- If all the test case observations are as
expected, the test case passes. otherwise, it
fails. - The test can be partitioned into a client object
and a server object in the test execution
directives. - Create(), remove(), update() mapped to a client
object. - Start(), stop() mapped to server object.
31Example cont.
- When execution the test, our execution
environment does the following - Creates both the application server and the
client object. - Calls the client objects methods create(),
remove() and update(). - Calls the application server objects methods
start() and stop(). - Compare the responses from both the server and
client objects with expected responses.
32Object distribution
- When the SUT is a distributed application, the
test runs on multiple hosts. - The test driver coordinates all interaction
between or within the test objects , based on the
abstract test. - This includes object initialization and cloning.
33Object multiplication and cloning
- The abstract test may be re-used for concurrency
and stress testing by multiplying the SUT objects
and the environment in which they execute.
34- Abstract test is executed in a stepwise fashion.
- Multiple independent operations may run
concurrently or sequentially. - At each step, the test driver runs the required
number of independent operations and waits for
their completion. Only then it checks the
results - If as expected continue to the next test step.
- If failed the exact sequence that triggered the
fault is known.
35- The sequence can be replayed when a fix is
provided for the fault. - The test driver also allows the user to step
through a test case interactively when debugging
a fault.
36Mutual independence of objects
- When the number of objects does not affect the
test behavior, these objects are said to be -
37Mutual independence of objects cont.
- We refer to the creation of many instances of the
same SUT object as cloning. - Clones are created alike but they are made
distinct by initializing each clone with
different values.
38Mutual independence of objects cont.
- The value that is used to initialize each clone
must be set according to the following rule - The value must be a shared attribute that
distinguishes the SUT objects, but allow the test
behavior to remain invariant when many objects
are instantiated.
39Mutual independence of objects cont.
- In the client/server system there may be many
clients and servers referring to the same
database. - It is necessary to verify that the SUT works
properly in the presence of many clients and
servers. - Object mutual-independence is achieved by
initializing each client clone with a different
set of keys used to access the database.
40Mutual independence of objects cont.
- Object multiplication and cloning is an efficient
way to validate concurrency and stress since it
reuses an existing abstract test. - Once the test has run successfully without object
multiplication, any problems discovered
subsequently can be attributed to the
introduction of more objects into the system.
41Object synchronization
- The test driver can reproduce the precise
synchronization scenario that introduced the
fault, thereby facilitating the fixing of the
bug. - The test driver implements two types of
synchronization of cloned objects concurrent
and sequential.
42Concurrent synchronization.
- Concurrent synchronization causes each abstract
stimulus and observation to start on all clones
at the same time. - The next stimulus and observation does not start
until the previous one is complete on all clones.
43Concurrent synchronization.
44Sequential synchronization
- Sequential synchronization causes each abstract
stimulus and observation to start and complete on
each clone, before proceeding to the next one.
45Sequential synchronization
46Object initialization
- Object initialization is used to create mutual
independence between objects. - The test driver creates process controllers and
objects dynamically during testing. - It uses a hierarchical structure to organize them.
47Object initialization cont.
- Each host manager, process controller or object
is initialized with initialization sets
containing name-value pairs specified in the test
directives. - Value pairs may reference values belonging to
other members that are further up in the
hierarchy.
48Object initialization cont.
- Example
- Host initialization set includes
- IP 9.148.32.112 .
- One of its processes initialization set includes
- USER JOE .
- Any object of the process may include
- IP HOST.IP USER PROCESS.USER .
- In some cases those pairs may not be enough and
there should be at least one more unique pair,
why ?
49Industrial Experience
50GOTCHA - TCBeans
- IBM internal test automation toolset.
- Developed at the IBM Research Laboratory in
Haifa. - GOTCHA modeling tool used to generate abstract
tests. - TCBeans prototype of IBMs test execution
environment. - Has been used successfully to test IBM
applications for the past four years.
51Customer support center
- SUT dispatcher of an electronic customer
support center. - The customer support center is a component based
distributed application written in java. - Customers of the support center contact agents
and request support.
52Customer support center cont.
- Agents are represented by software components
called - access points. - Each access point generates interaction objects
to represent its communications. - A dispatcher schedules interactions to be
serviced by access points. - The dispatcher has a complex scheduling policy
based on the skills required to deal with the
communication and the skills possessed by the
available agents.
53Customer support center cont.
- GOTCHA was used to generate a set of abstract
tests. - TCBeans was used by an independent testing group
to execute the test. - 35 defects were discovered.
- Most of the defects were attributed to sequencing
problems.
54Customer support center cont.
- Testing with many access points was a requirement
in the system test to validate the dispatcher
under stress. - The test team reused the abstract test but
created many access points using object
distribution, multiplication and cloning. - Without any additional coding, they discovered an
additional 5 defects.
55Future Work
56Future work
- Go beyond the stepwise concurrency scheme and run
more complex synchronization patterns generated
from an abstract model. - Example synchronize the execution of more than
one instance of the abstract tests. - When to synchronize the stimuli and observations
between the multiple test instances.
57THE END