Software Reliability Estimates Projections, Cumulative

1
Rivier College Mathematics Computer Science
Lecture Series March 23, 2006

• Software Reliability Estimates/ Projections,
  Cumulative Instantaneous
• Presented by Dave Dwyer

With help from Ann Marie Neufelder, John D.
Musa, Martin Trachtenberg, Thomas Downs, Ernest
O. Codier and Faculty of Rivier College Grad.
School Math and Computer Science
2
Martin Trachtenberg (1985)

• Simulation shows that, with respect to the number
  of detected errors
• Testing the functions of the software system in a
  random or round-robin order gives linearly
  decaying system error rates.
• Testing each function exhaustively one at a time
  gives flat system-error rates
• Testing different functions at widely different
  frequencies gives exponentially decaying system
  error rates Operational Profile Testing, and
• Testing strategies which result in linear
  decaying error rates tend to require the fewest
  tests to detect a given number of errors.

3
Thomas Downs (1985)
In this paper, an approach to the modeling of
software testing is described. A major aim of
this approach is to allow the assessment of the
effects of different testing (and debugging)
strategies in different situations. It is shown
how the techniques developed can be used to
estimate, prior to the commencement of testing,
the optimum allocation of test effort for
software which is to be nonuniformly executed in
its operational phase.
4
There are Two Basic Types of Software Reliability
Models

• Predictors - predict reliability of software at
  some future time. Prediction made prior to
  development or test as early as concept phase.
  Normally based on historical data.
• Estimators - estimate reliability of software at
  some present or future time based on data
  collected from current development and/or test.
  Normally used later in life cycle than predictors.

5
A Pure Approach Reflects the True Nature of
Software

• The execution of software takes the form of the
  execution of a sequence of M paths.
• The actual number of paths affected by an
  arbitrary fault is unknown and can be treated as
  a random variable, c.
• Not all paths are equally likely to be executed
  in a randomly selected execution profile.

6
Start
x1
xN
x2
3
M
1
2
2 paths affected by x1
M total paths
1 path affected by x2
N total faults initially
c paths affected by an arbitrary fault
7
Further...

• In the operational phase of many large software
  systems, some sections of code are executed much
  more frequently than others.
• Faults located in heavily used sections of code
  are much more likely to be detected early.

8
Downs (IEEE Trans. on SW Eng. April, 1985) Showed
that approximations can be made

• Each time a path is selected for testing, all
  paths are equally likely to be selected.
• The actual number of paths affected by an
  arbitrary fault is a constant

9
My Data Assumptions

• Cumulative 8 Hr. test shifts are recorded VS the
  number of errors.
• Each first instance is plotted
• The last data point will be at the end of the
  test time, even though there was no error,
  because a long interval without error is more
  significant than an interval with an error.

10
Other Assumptions

• Only integration system test data are used.
• Problems will be designated as priority 1, 2 or 3
  (Ref DoD-STD-2167A) where
• Priority 1 Prevents mission essential
  capability
• Priority 2 Adversely affects mission essential
  capability with no alternative workaround
• Priority 3 Adversely affects mission essential
  capability with alternative workaround

11
Downs Showed ? faults/path

• ?j (N j)?, where
• N the total number of faults,
• j the number of corrected faults,
• ? -r log(1 c/M),
• r the number of paths executed/unit time,
• c the average number of paths effected by each
  fault and
• M the total number of paths

12
Failure Rate is proportional to failure number,
Downs ?j ? (N j)r(c/M)
13
Failure rate plots against failure number for a
range of non-uniform testing profiles, M1, M2
paths and N1, N2 initial faults in those paths.
(Logarithmic?)
14
Imagine two main segments
Segment 2
Segment 1
15
After testing segment 1, someone asks

• Given 10 faults found, whats the reliability of
  the code?
• Responses
• Dont know how many other faults remain in
  section 1, let alone are in section 2
• Dont know how often sections 1, 2 are used.
• Did we plot failure intensity vs faults?
• Why didnt we test to the operational profile?

16
By reference to Duanes derivation for hardware
reliability, (Ref. E. O. Codier, RAMS - 1968)
17
Failure Intensity (Instantaneous Failure Rate)
Derivation - Hardware Software
Duanes Instantaneous ? for HW
Daves Instantaneous ? for SW
Failure Intensity
Similar Result
18
Priority 1 Data Plotted
19
Priority 1 and 2 Data Plotted
20
Point Estimates vs Instantaneous
21
(No Transcript)
22
For copy of paper, e-mail request to

• david.j.dwyer_at_baesystems.com