Regression Testing

1
Regression Testing

• SE 760, Software Quality Engineering
• Mark Ardis, RIT

2
Acknowledgement

• Some material in these slides is taken from
• "An empirical study of regression test selection
  techniques" by Graves, Harrold, Kim, Porter and
  Rothermel, ACM Transactions on Software
  Engineering and Methodology 10(2), 184-208, April
  2001.

3
Outline

• Purpose of regression testing
• Types of regression testing
• Effectiveness of test selection

4
Testing in the Large

• Integration testing putting the pieces together
• System testing function and performance
• Regression testing making sure that fixes do not
  break other functionality

5
Regression

• Loss or deterioration of functionality
• Causes
• Introduction of new features creates an undesired
  interaction with old features
• Repair of one defect introduces a new defect

6
When is Regression Testing Done?

• After each new version of a product is created
• for scheduled new versions
• for maintenance versions
• Often late in the lifecycle
• need to pass other tests first
• last step before delivery

7
Principles

• Results of all tests should be known in advance
• Features and modules tested should be identified
  for each test
• All features and modules should be covered by a
  complete test suite

8
Problems

• Update maintenance of test suite as product
  changes
• Some tests become obsolete
• Some tests need to be edited
• Overall suite size grows over time
• Selection determining which tests to run after a
  change

9
Selection Methods

• Positive
• Negative
• Minimization strategies
• Random
• Hybrid combination of above

10
Positive Regression Testing

• Collect successful tests from previous phases of
  testing
• Emphasize frequently-used paths of the product
• Cover all features of product
• Does not provide evidence of defect repair

11
Negative Regression Testing

• Collect tests from users, or that exercise
  user-reported defects
• Provides evidence that defects have been repaired
  (and stay repaired)
• Does not cover entire product

12
Minimization Strategies

• Use control-flow or data-flow information to
  partition test suites
• Use knowledge of changes to source code to derive
  needed tests
• find tests that execute changed code
• avoid tests that execute constant code
• May not always be safe

13
Random Selection

• Specify test inputs by types
• Randomly generate values of appropriate types
• Manually determine appropriate outputs for all
  inputs, or write code that can perform
  intelligent check

14
Regression Testing Definitions

• P is the original program
• P' is the modified version of P that needs to be
  tested
• T is a suite of tests for P
• S is a specification of P

15
Regression Testing Process

• Delete obsolete tests
• Select T' ? T to execute on P'
• Test P' with T'
• Create T'', new tests for P'
• Test P' with T''
• Update test suite from T, T' and T''

16
Delete Obsolete Tests

• A test is obsolete if
• the input is no longer valid for P'
• the expected output is no longer valid for P'
• Cannot wait to discover these during testing
• may cause abortion of execution process
• need to use automation to do checking for
  correctness

17
Regression Test Selection

• Want to find a subset T' of T that will
  adequately test P'
• Select T' ? T to execute on P'
• If T' is too big, then little value over T
• If T' is too small, may omit fault-revealing tests

18
Coverage Identification

• Need to identify areas of P' or S' that are not
  tested by T
• Create T'', new tests for P'
• May use traditional unit testing coverage
  criteria
• Need to track which unit tests cover which parts
  of P' and S'

19
Test Suite Execution

• Need to efficiently execute tests and check for
  correctness
• Test P' with T'
• ...
• Test P' with T''
• Requires automation, especially checking for
  correctness

20
Test Suite Maintenance

• Need to update and store useful regression tests
  for future use
• Update test suite from T, T' and T''
• Need automation

21
Regression Test Selection Techniques

• Minimization
• Dataflow
• Safe
• Random
• Retest-All

22
Minimization Techniques

• Divide P into basic block segments
• Use linear programming to identify a subset T' of
  T that ensures
• Every segment that is reachable from a modified
  segment of P' is exercised by at least one test
  case in T' that also exercises the modified
  segment.

23
Dataflow Techniques

• Use du-pairs to select tests
• For example, test every du-pair that was
• deleted from P, or
• new in P', or
• modified for P

24
Safe Techniques

• Ensure that T' contains all test cases that can
  reveal faults in P'
• E.g., Rothermel and Harrold
• all tests that execute statements that were
  deleted from P
• all tests that execute new statements in P'
• all tests that execute modified statements in P'

25
Random Techniques

• Randomly select a subset
• May choose different subset sizes
• Requires practically no analysis cost

26
Retest-All Technique

• Let T' T
• Requires no analysis cost

27
Cost-Benefit Analysis Graves et al.

• Random gt Minimization
• equivalent effectiveness for less analysis cost
• Safe gt Dataflow
• equivalent effectiveness for less analysis cost
• Safe more effective than Random
• Random more efficient than Safe