Who wrote the article - PowerPoint PPT Presentation

1 / 57
About This Presentation
Title:

Who wrote the article

Description:

Deriving tests from specifications abstract tests. ... express the testing logic without getting bogged down in the details of how to execute the test. ... – PowerPoint PPT presentation

Number of Views:37
Avg rating:3.0/5.0
Slides: 58
Provided by: eya3
Category:

less

Transcript and Presenter's Notes

Title: Who wrote the article


1
(No Transcript)
2
Who wrote the article ?
Alan Hartman
Kenneth Nagin
Andrei Kirshin
3
What is it all about ?
  • Executing abstract tests on distributed software.
  • Deriving tests from specifications abstract
    tests.
  • Tester must manually convert those tests into
    their concrete executable form.
  • Could be used to verify distributed software.
  • The same abstract tests can be extended to verify
    issues related to concurrency, stress and input
    data variations.

4
Main issues
  • Introduction.
  • Test execution environment design.
  • Abstract tests extended with object partitioning.
  • Industrial experience.
  • Future work.

5
Introduction
6
  • What is an abstract test ?
  • A test which describes test logic without
    concerning itself with platform or language
    issues.
  • In order to run the test, the execution
    environment must deal with the messy details of
    concrete execution.
  • The abstract tests may be reused and extended to
    verify additional software requirements such as
    concurrency.

7
  • In recent years, software modeling has enjoyed
    tremendous popularity due to the adoption of
    object oriented models.
  • Abstract tests are generated from such software
    models.

8
  • However, the generation of test suites from a
    specification based modeling has yet to gain
    industrial momentum.
  • Mainly because most current test execution
    environments were not designed to run the
    generated tests.

9
System Under Test
  • Abstract model of System Under Test (SUT) -

10
System Under Test cont.
  • SUT may be perceived of as a set of objects.
  • The SUT is not restricted to any programming
    language or operating platform.
  • The abstract test describes the objects expected
    behavior, state, and output in response to a
    stimuli from the tester.
  • The concrete implementation of the SUT runs
    within different process address space.

11
Test Execution Environment
  • Software testing environment are applications
    that assist testers in executing tests and
    collecting results against the SUT.
  • Testing environments should run under the same
    operating system and use the same programming
    language in order to interact with the SUT.
  • The cost of testing environments is a significant
    part of the overall development cost.

12
Test Execution Environment cont.
  • The cost and complexity of the testing
    environment increases when the SUT is
    distributed.
  • The main problem is that distributed software may
    run in different host environments, separated by
    a network and its components may be coded in
    different programming languages.

13
Test Execution Environment cont.
  • Example of distributed software

Unix C
Linux Java
Windows C
TCP/IP
14
Test Execution Environment cont.
  • The main goal of the test execution environment
    is to provide an infrastructure to support the
    heterogeneous application environment and
    languages.
  • Of course, it must scale down to support more
    simple non-distributed software testing.

15
Abstract Test Execution
  • An abstract test specifies a sequence of tester
    stimuli to the SUT, observation of SUT responses,
    and the assignment of a verdict based on the
    observations.
  • An abstract test is designed to exercise a
    particular execution sequence or verify
    compliance with a specific requirement.

16
Abstract Test Execution cont.
  • Abstract tests have the advantage that they
    express the testing logic without getting bogged
    down in the details of how to execute the test.
  • Thus, the abstract test must be translated in
    order to execute it on the SUT.
  • Abstract stimuli is mapped to method invocation,
    mouse clicks, or system commands.
  • Abstract observations are mapped to actual value
    checking.

17
Abstract Test Execution cont.
  • The test execution environment has 2 inputs
  • Abstract test, telling what to execute against
    the SUT.
  • Test execution directives, telling how to execute
    the abstract test.

Test Driver
Abstract Test
Test Exec. Directives
18
Test Execution Environment Design
19
  • The test execution environment design supports
    the basic functionality for testing distributed
    systems.

20
  • The main components of the test execution
    environment are
  • Test Driver manages the test execution.
  • Host Manager replicated on each target host
    (each host can have different operating system).
  • Process Controllers interact directly with the
    SUT objects.
  • The test driver and SUT need not be coded in the
    same programming language.

21
  • The test driver communicates with host managers
    and process controllers in order to accomplish
    the following tasks
  • Create, destroy and locate process controllers.
  • Create and destroy SUT objects.
  • Control input to the SUT objects.
  • Observe output and state of the SUT objects.

via process controllers
via host manager
22
  • Only the test driver understands the syntax and
    semantics of the abstract test cases.
  • All other components handle real world
    interactions.
  • This separation of concerns makes the test
    execution environment design well-suited for the
    testing of distributed software.

23
  • The test driver is implemented only once since it
    has no SUT platform or language dependencies.
  • Host manager and process controller
    implementation is dependent on the target SUT.
  • Host managers must be implemented in all target
    SUT platforms (windows, Linux, ).
  • Process controller runs in the same process as
    the SUT object and they are both coded in the
    same programming language.

24
Abstract Tests Extended with Object partitioning
25
Partitioning
  • The SUT consist of different entities whose
    behavior is described in the abstract test.
  • The tester can partition the abstract test into a
    set of executable objects.
  • The object partitioning idea is everywhere.

26
  • Motivation for object partitioning
  • Extend the test usage to verify issues related to
    concurrency, stress and input data variations.

27
  • The test modeling language does not have to be
    object oriented.
  • However, it is required that the abstract stimuli
    and observable responses be identifiable so they
    can be mapped to concrete objects.

28
Example
  • Consider an SUT that is client/server
    application.
  • Server role - connect its clients to a database
    of accounts.
  • Unique keys index the database of accounts.
  • Client role create, remove, or update accounts
    based on the keys.

29
Example cont.
  • We illustrate a simple test case
  • Start the application server.
  • Observe that no exception occurred.
  • Create an account with key1.
  • Observe that no exception occurred.
  • Create another account with key1.
  • Observe that an exception is thrown.
  • Stop the application server.

30
Example cont.
  • If all the test case observations are as
    expected, the test case passes. otherwise, it
    fails.
  • The test can be partitioned into a client object
    and a server object in the test execution
    directives.
  • Create(), remove(), update() mapped to a client
    object.
  • Start(), stop() mapped to server object.

31
Example cont.
  • When execution the test, our execution
    environment does the following
  • Creates both the application server and the
    client object.
  • Calls the client objects methods create(),
    remove() and update().
  • Calls the application server objects methods
    start() and stop().
  • Compare the responses from both the server and
    client objects with expected responses.

32
Object distribution
  • When the SUT is a distributed application, the
    test runs on multiple hosts.
  • The test driver coordinates all interaction
    between or within the test objects , based on the
    abstract test.
  • This includes object initialization and cloning.

33
Object multiplication and cloning
  • The abstract test may be re-used for concurrency
    and stress testing by multiplying the SUT objects
    and the environment in which they execute.

34
  • Abstract test is executed in a stepwise fashion.
  • Multiple independent operations may run
    concurrently or sequentially.
  • At each step, the test driver runs the required
    number of independent operations and waits for
    their completion. Only then it checks the
    results
  • If as expected continue to the next test step.
  • If failed the exact sequence that triggered the
    fault is known.

35
  • The sequence can be replayed when a fix is
    provided for the fault.
  • The test driver also allows the user to step
    through a test case interactively when debugging
    a fault.

36
Mutual independence of objects
  • When the number of objects does not affect the
    test behavior, these objects are said to be -

37
Mutual independence of objects cont.
  • We refer to the creation of many instances of the
    same SUT object as cloning.
  • Clones are created alike but they are made
    distinct by initializing each clone with
    different values.

38
Mutual independence of objects cont.
  • The value that is used to initialize each clone
    must be set according to the following rule
  • The value must be a shared attribute that
    distinguishes the SUT objects, but allow the test
    behavior to remain invariant when many objects
    are instantiated.

39
Mutual independence of objects cont.
  • In the client/server system there may be many
    clients and servers referring to the same
    database.
  • It is necessary to verify that the SUT works
    properly in the presence of many clients and
    servers.
  • Object mutual-independence is achieved by
    initializing each client clone with a different
    set of keys used to access the database.

40
Mutual independence of objects cont.
  • Object multiplication and cloning is an efficient
    way to validate concurrency and stress since it
    reuses an existing abstract test.
  • Once the test has run successfully without object
    multiplication, any problems discovered
    subsequently can be attributed to the
    introduction of more objects into the system.

41
Object synchronization
  • The test driver can reproduce the precise
    synchronization scenario that introduced the
    fault, thereby facilitating the fixing of the
    bug.
  • The test driver implements two types of
    synchronization of cloned objects concurrent
    and sequential.

42
Concurrent synchronization.
  • Concurrent synchronization causes each abstract
    stimulus and observation to start on all clones
    at the same time.
  • The next stimulus and observation does not start
    until the previous one is complete on all clones.

43
Concurrent synchronization.
44
Sequential synchronization
  • Sequential synchronization causes each abstract
    stimulus and observation to start and complete on
    each clone, before proceeding to the next one.

45
Sequential synchronization
46
Object initialization
  • Object initialization is used to create mutual
    independence between objects.
  • The test driver creates process controllers and
    objects dynamically during testing.
  • It uses a hierarchical structure to organize them.

47
Object initialization cont.
  • Each host manager, process controller or object
    is initialized with initialization sets
    containing name-value pairs specified in the test
    directives.
  • Value pairs may reference values belonging to
    other members that are further up in the
    hierarchy.

48
Object initialization cont.
  • Example
  • Host initialization set includes
  • IP 9.148.32.112 .
  • One of its processes initialization set includes
  • USER JOE .
  • Any object of the process may include
  • IP HOST.IP USER PROCESS.USER .
  • In some cases those pairs may not be enough and
    there should be at least one more unique pair,
    why ?

49
Industrial Experience
50
GOTCHA - TCBeans
  • IBM internal test automation toolset.
  • Developed at the IBM Research Laboratory in
    Haifa.
  • GOTCHA modeling tool used to generate abstract
    tests.
  • TCBeans prototype of IBMs test execution
    environment.
  • Has been used successfully to test IBM
    applications for the past four years.

51
Customer support center
  • SUT dispatcher of an electronic customer
    support center.
  • The customer support center is a component based
    distributed application written in java.
  • Customers of the support center contact agents
    and request support.

52
Customer support center cont.
  • Agents are represented by software components
    called - access points.
  • Each access point generates interaction objects
    to represent its communications.
  • A dispatcher schedules interactions to be
    serviced by access points.
  • The dispatcher has a complex scheduling policy
    based on the skills required to deal with the
    communication and the skills possessed by the
    available agents.

53
Customer support center cont.
  • GOTCHA was used to generate a set of abstract
    tests.
  • TCBeans was used by an independent testing group
    to execute the test.
  • 35 defects were discovered.
  • Most of the defects were attributed to sequencing
    problems.

54
Customer support center cont.
  • Testing with many access points was a requirement
    in the system test to validate the dispatcher
    under stress.
  • The test team reused the abstract test but
    created many access points using object
    distribution, multiplication and cloning.
  • Without any additional coding, they discovered an
    additional 5 defects.

55
Future Work
56
Future work
  • Go beyond the stepwise concurrency scheme and run
    more complex synchronization patterns generated
    from an abstract model.
  • Example synchronize the execution of more than
    one instance of the abstract tests.
  • When to synchronize the stimuli and observations
    between the multiple test instances.

57
THE END
Write a Comment
User Comments (0)
About PowerShow.com