Software%20Testing:%20A%20Selection%20of%20Papers%20by%20Elaine%20Weyuker

1
Software TestingA Selection of Papers by Elaine
Weyuker

• Peter Greenwood
• for EEE 599

2
Elaine Weyuker

• PhD (CS) Rutgers U 77
• Software Testing Researcher at ATT Bell Labs,
  Murray Hill, NJ
• CS Professor Courant Institute of NYU
• 30 papers, primarily on Software Testing

3
The Papers

• Axiomatizing Software Test Data Adequacy (TSE Dec
  86)
• The Automatic Generation of Load Test Suites and
  the Assessment of the Resulting Software (TSE Sep
  95, with Avitzer)
• Reliability Testing of Rule-Based Systems
  (Software Jun 96, with Avritzer and Ros)

4
Goal of Presentation

• highlight some of Weyukers prolific work in
  software testing
• contrast the general and theoretical character of
  her early work with the specific and practical
  nature of her current work

5
Axiomizing Software Test Data Adequacy (Dec 86)

• How much testing is enough to determine if a
  program is acceptable?
• very important question
• relatively little research activity (1986)
• no industry standards
• test until testing time expired
• test until all cases in current test set pass

6
When Should Testing Stop?

• Terminate when an adequacy criterion is met
• a measure of how well the testing process has
  been performed
• relates a test set to the program, the
  specification, or both
• could relate a test set to the programs
  operational profile

7
Constraints of this Approach

• deals only with dynamic testing (running the
  program on test cases)
• does not involve any static testing (analysis of
  code, structure, etc.)

8
Constraints (cont)

• Weyukers paper deals only with program-based
  criteria
• ignores specifications
• most adequacy criteria are of this type
• easier to automate

9
Weyuker on Testing

• The purpose of testing is to uncover errors, not
  to certify correctness.

10
The Axioms (1 2)

• Axiom 1 (Applicability) For every program, there
  exists a finite adequate test set.
• Axiom 2 (Nonexhaustive Applicability) There is a
  program P and a test set T such that P is
  adequately tested by T, and T is not an
  exhaustive test set.

11
Axioms 3 4

• Axiom 3 (Monotonicity) If T is adequate for P,
  and T is a subset of T, then T is adequate for
  P.
• Axiom 4 (Inadequate Empty Set) The empty set is
  not adequate for any program.

12
Axioms 5 6

• Axiom 5 (Anti-extensionality) There are programs
  P and Q such that P is equivalent to Q, T is
  adequate for P, but T is not adequate for Q.
• Axiom 6 (General Multiple Change) There are
  programs P and Q which are the same shape, and a
  test set T such that T is adequate for P, but T
  is not adequate for Q.

13
Axiom 7

• Axiom 7 (Anti-decomposition) There exists a
  program P and a component Q such that T is
  adequate for P, T is the set of vectors of
  values that variables can assume on entrance to Q
  for some t of T, and T is not adequate for Q.

14
Axiom 8

• Axiom 8 (Anti-composition) There exist programs
  P and Q such that T is adequate for P and P(T) is
  adequate for Q, but P is not adequate for PQ.

15
Summary of Axiomatizing

• axioms are applied to 5 adequacy criteria
• path, branch, statement, mutations, modified size
• path, branch, statement fail application and
  antidecompositon
• branch and statement also fail anticomposition

16
Summary of Axiomatizing
17
Criticism

• Zweben Gourlay, On the Adequacy of Weyukers
  Test Data Adequacy Axioms (IEEE TSE Apr 89)
• The properties proposed by Weyuker are neither
  fundamental enough nor precise enough to serve as
  a useful formal system for discussing adequacy or
  comparing criteria

18
More Criticism

• Professor Norman Fenton, keynote address to 2nd
  QUALMS User Group Meeting, South Bank University,
  Jul 92
• Elaine Weyuker has presented a list of axioms
  Measurement Theory reveals that some of these
  axioms are contradictory.

19
Dr. Weyukers Current Work
20
Automatic Generation of Load Test Suites (Sep 95)

• develops 3 algorithms for automatic test case
  generation
• applies them to 5 real industrial software
  systems
• assess the results

21
Failure Types

• Hard failure
• complete loss of traffic ( ie cable break, power
  failure, router failure)
• Soft failure
• degraded performance (ie activity burst)
• important that this does not occur
• telecoms designed with excess capacity
• avoid permanent soft failure

22
Goals

• a smoothly degrading system
• graceful recovery

23
Load Testing Strategies

• system functionality has already been tested
• assess system performance under a known load
• black-box oriented
• based on system operational profile

24
Typical Telecom Operational Profile

• number of different types of call
• average service demands and probability
  distribution of each call type
• average external arrival rate
• routing probabilities for each node
• this information is usually readily available

25
Purpose of Algorithms

• tests resource allocation mechanisms associated
  with software systems
• system behavior under load is not measured by
  functionality testing
• each variable vector is a distinct system state

26
Basis of Algorithms

• too many states to exhaustively test
• calculate probability of system being in each
  state (from operational profile)
• heuristically determine some threshold ?
• generate a test case for those states where P(S)
  gt ?

27
The Algorithms

• Deterministic State Testing (DST)
• generate a test case for each state S where P(S)
  gt ?
• DST with State Transition Checking (TDST)
• as for DST, but also generate test cases for
  every S reachable from S

28
The Algorithms (cont)

• DST with Length N Cycles (CDST)
• as for DST, but also generate test cases for
  every S within a cycle of length N from S with a
  probability gt ?

29
Target Systems

• Project 1 telephone call rerouter
• Project 2 800 number call traffic monitor
• Project 3 routing automator
• Project 4 transaction processing system
• Project 5 network interface software

30
Testing Results

• automated test generation routines were developed
  for all five projects
• automated test results were presented for
  projects 1 2

31
Project 1 Results

• telephone call rerouter software
• test results accurately predicted system
  performance
• testing revealed a serious architectural problem

32
Project 2 Results

• 800 number call traffic monitor
• led to better requirements analysis
• used automated test case generation exclusively,
  replacing hand-crafting
• more test cases with more variety
• uncovered four potentially severe faults
• minimal additional cost

33
Reliability Testing of Rule-Based Systems (Sep 96)

• applies same automatic test case generation
  algorithm to an industrial network surveillance
  system
• system was a reimplementation, so a large body of
  operational data was available from the original
  system
• system designed to run 24 hours/day 365 days/year

34
Approach

• system has very large state space
• 355 days of operational data from 93 switches
  were used to identify 857 states entered
• 0.999 coverage required testing 843 states

35
State Testing Requirements
36
Findings

• historical data is useful for selecting test
  cases based on state probabilities
• selection algorithm must consider
• reliability requirements
• size of the (limited) state space
• comprehensiveness of individual test cases

37
Summary

• change of focus from theory to application
• extension from telecom to general application
• accomplished, respected, and prolific researcher
  in software testing and reliability