SENG 521 Software Reliability

1
SENG 521Software Reliability Testing

• Preparing for Test
• (Part 6a)

Department of Electrical Computer Engineering,
University of Calgary B.H. Far (far_at_enel.ucalgary.
ca) http//www.enel.ucalgary.ca/far/Lectures/SENG
521/06a/
2
Concepts /1

• Run The smallest division of work that can be
  initiated by external intervention. Run is
  associated with input state (set of input
  variables) and runs with identical input state
  are of the same run type.
• Operation An operation is a group of run types
  as conceived at the development stage.
• Test procedure A controller that sets up
  environmental conditions and invokes randomly
  selected test cases at random times.

3
Concepts /2

• Types of test
• Feature Test A single execution of an operation
  with interaction between operations minimized.
• Load Test Testing with field use data and
  accounting for interactions
• Regression Test Feature tests after every build
  involving significant change, i.e., check whether
  a bug fix worked.

4
Concepts /3

• Direct input variable is a variables that
  controls the operation directly
• Example arguments, selection menu, entered data
  field.
• Indirect input variable is a variable that only
  influences the operations or its effects are
  propagated to the operation
• Example traffic load, environmental variable

5
Example

• Example of direct and indirect input variables
  for an operation

6
What is a Test Case?

• A test case is a partial specification of a run
  through the naming of its direct input variables
  and their values.
• Test case is independent of operational mode
• The same test case can execute in different
  operational modes. Thus a test case can generate
  multiple runs each with different potential
  failure behavior

7
Properties

• A good test case satisfies
• It has a reasonable probability to catch an error
• It is not redundant
• It is the best of its breed
• It is neither too simple nor too complex
• It makes program failure obvious

8
Example

• Test case is specified with its direct input
  variables
• In theory, it is possible to record all the input
  variables needed to initiate the runs that make
  up the whole execution space of the software

9
Test Case Run

• Specification of the indirect input variable
  gives a test case the necessary context so it can
  become a run.
• During feature and regression test, the influence
  of indirect input variables should be kept to
  minimum to ensure that the operation is reliable.
• Indirect input variables are effective during
  load test. The load test is divided into a number
  of operation modes each derived by a test
  procedure.

10
Procedure

• The procedure involves two steps
• Preparing test cases (Test case management)
• Using either field data recording or operational
  profiles
• Preparing test procedures
• Test procedure is a controller that sets up
  environmental conditions and invokes randomly
  selected test cases at random times.
• Prepare one test procedure for each operational
  mode using operational profile and the operation
  occurrence rate

11
Test Cases Management

• The procedure for preparing test cases involve
• Estimate the number of new test cases needed for
  the current release
• Allocate the number of new test cases among the
  systems to be tested
• Allocate the number of new test cases for each
  system among its new operations
• Specify the new test cases
• Adding the new test cases to the ones already
  available (may be from a previous release)

12
1. Estimate New Test Cases

• Affected by two factors time and cost
• Method
• Compute the number of test cases for the given
• Time
• (available time X available staff)/
• (average time to prepare a test
  case)
• Cost
• (available budget) / (average preparation cost
  per test case)
• Select the minimum number of the two

13
2. Test Case Allocation System

• Allocate the bulk of the test cases to the
  product itself.
• Give weight to the differing operations that have
  high occurrence probabilities.
• Do not allocate test cases to acquired components
  unless the newly acquired components are
  substituted for previous ones.
• Allocate test cases to acquired components based
  on their size relative to the product and the
  estimated risk.
• In the case of Fone Follower (example in Musas
  Book), the operating system represented a
  substantial part of the product and its
  reliability was unknown. Hence 500 test cases
  were allocated to the product and 200 to the
  operating system.

14
3. Test Case Allocation Operations

• Convert graphical representation of the
  operational profile to the tabular representation
  by walking through all the paths, obtain an
  occurrence probability for each operation (path)
  by multiplying the branch probabilities together.
  
• Identify the rarely occurring critical new
  operations and determine how many test cases to
  pre-assign to each.
• Determine the allocation probabilities for the
  other new operations.
• Pre-assign one test case to each infrequent other
  new operation.
• Assign the remaining test cases to the remaining
  other new operations in accordance with the
  allocation probabilities.

15
Critical Operations

• A critical operation is one for which successful
  execution adds a great deal of extra value and
  failure causes a great deal of impact with
  respect to human life, cost, or system
  capability.
• An example of a critical operation is the SCRAM
  operation in nuclear power plants that shuts down
  a reactor when it starts to overheat. The
  operation that handles this condition is very
  rarely used, but it is extremely critical.
• Identify only the rarely occurring critical
  operations because sufficient test cases will be
  allocated to the frequently occurring critical
  operations because of their substantial
  occurrence probabilities.

16
New Operations

• In the case of a new release, set the allocation
  probabilities for the new operations equal to the
  occurrence probabilities of the system
  operational profile.
• This is a satisfactory approximation the sum of
  the occurrence probabilities for the other new
  operations will be very close to 1.
• For subsequent releases, divide the system
  occurrence probabilities for the other new
  operations by the total of the system occurrence
  probabilities for the other new operations.

17
Infrequent Operations

• Infrequent operations are those that would not
  normally be assigned test cases by virtue of
  their occurrence probabilities.
• Ensure that we assign each operation at least one
  test case.
• The occurrence probability of an operation is
  often such that, because of the fact that runs
  usually substantially outnumber test cases, the
  operation might well invoke a run even though the
  occurrence probability might not normally result
  in test case selection.
• Thus determine the threshold occurrence
  probability below which you would not normally
  assign a test case to an operation.
• This is 0.5 divided by the number of new test
  cases.
• We then pre-assign one test case to each other
  new operation whose occurrence probability falls
  below the threshold.