SENG 521 Software Reliability

1
SENG 521Software Reliability Testing

• Executing Test
• (Part 7a)

Department of Electrical Computer Engineering,
University of Calgary B.H. Far (far_at_enel.ucalgary.
ca) http//www.enel.ucalgary.ca/far/Lectures/SENG
521/07a/
2
Contents

• Allocating test time
• How to allocate test time among system components
  based on test type (feature test, regression
  test, load test) and operation modes?
• Invoking the test
• In what order the test should be carried on?
• How many test cases and runs should be invoked?
• Identifying failures that occur
• What deviation could be discovered?
• How to document it?

3
Test Time Allocation

• Allocate test time
• Among systems to be tested
• Among feature, regression and load test for each
  associated system under reliability growth test
• Among operational modes for each associated
  system under load test

4
1. Test Time Systems /1

• Various systems involved get their share in test
  time
• Allocate time to interface to other systems based
  on estimated risk
• Allocate time to the rest in the same proportion
  that test cases were allocated

Interface to other systems
Acquired components
Developed Components
OS
5
1. Test Time Systems /2

• Example
• Total test time 320 h
• Test case proportion
• Associated system components 71.4
• Operating system component 28.6

Interface to other systems (40 h)
Associated System Components 200 h
Acquired components
Developed Components
OS (80 h)
6
2. Test Time Reliability Growth Test

• For each associated system components during
  reliability growth test
• for all new test cases Allocate time to feature
  test (first release)
• for all new test cases Allocate time to
  regression test (subsequent releases)
• In this way testing all critical new operations
  will be guaranteed.
• The remaining time goes to load test.

7
3. Test Time Load Test

• Allocate time for load test based on the
  proportion of the occurrences in operational
  modes
• Example

The remaining 20 hours go to feature and
regression tests.
8
Test Invocation /1

• Sequence of system test
• Acquired components
• certification test only
• Developed product
• Feature test and then load test for a new product
• Feature test, and then regression test for
  subsequent releases
• Other systems interface
• load test only
• It is possible to change this sequence or testing
  different systems in parallel.

9
Test Invocation /2

• In the reliability growth test, perform feature
  test first and then load test. Then conduct
  regression test after each build that has a
  significant change.
• Invocation of test cases should occur at random
  times.
• In feature test select in random order from the
  set of all new test cases plus the regression
  test cases of the previous release.
• In load test, invoke each operational mode for
  its allotted proportion of time.
• The number of test cases invoked will be
  determined based on the time the operational mode
  runs and its occurrence rate.

10
Test Case Selection /1

• Selection should be with replacement for test
  cases in load test but not in feature or
  regression test.
• In feature or regression test, the runs are much
  less likely to be different because indirect
  input variables are tightly controlled.
• In load test, the resulting runs will almost be
  different, because the indirect variables can
  vary, causing to explore the failure behavior of
  different runs.
• Repetitions are inefficient. But in load test,
  the number of runs is so large that the
  probability of wasting test resources by
  repeating many runs is infinitesimal.

11
Test Case Selection /2

• Therefore, in load test, after each test, replace
  the element in the population, allowing
  reselection.
• Why?
• Operations may be associated with multiple
  faults. If selection is performed without
  replacement, the operation can be selected only
  once therefore reducing the probability of
  finding another bug.

12
Repetition /1

• In invoking test cases, repeat a run in certain
  special circumstances
• To collect more data to improve failure
  identification, either in the sense of more
  failures or better describing known failures
• To verify failure resolution (fault removal)
• Usually you will require some number n of runs
  per operation to achieve reasonable confidence
  that the operation is failure-free.
• How to select number of runs n?
• The number n cannot be chosen in general it
  depends on the nature of the operation and the
  code that implements it.

13
Repetition /2

• How to select number of runs n?
• If the occurrence probability of the operation is
  very small and the operation is non-critical,
  usually the risk of remaining fault(s) is
  acceptable.
• If occurrence probability is very small and the
  operation is critical, the number of runs for
  that operation must be increased.
• If the occurrence probability is not very small,
  the total amount of testing planned may be
  increased.

14
Random Run Selection /1

• The occurrence probabilities with which the
  operations for execution are selected, should be
  stationary, or reasonably unchanged with time.

• Example
• Set 1
• ABAABAAABA
• Set 2
• AAAAAAABBB

15
Random Run Selection /2

• For set 1, deviations from expected occurrence
  probability is reasonably small and can be
  considered stationary. For set 2 it is not.
• Non-stationary probabilities generally occur
  because the tester reorders the selection of
  operations from what would occur at random.