Chin-Yu Huang

1
Optimal Allocation of Testing-Resource
Considering Cost, Reliability, and Testing-Effort
The 10th International Symposium Pacific Rim
Dependable Computing (PRDC 2004)

• Chin-Yu Huang
• Department of Computer Science
• National Tsing Hua University
• Hsinchu, Taiwan

2
Outline

• Introduction
• Software Reliability Growth Model with
  Generalized Logistic Testing-Effort Function
• Testing Resource Allocation for Software Module
  Testing
• Experimental Studies and Results
• Numerical Example and Sensitivity
  Analysis
• Conclusions

3
Introduction

• A software testing process consists of several
  testing stages including module testing,
  integration testing, system testing, and
  installation testing.
• The quality of the tests usually corresponds to
  the maturity of the software test process, which
  in turn relates to the maturity of the overall
  software development process.
• Most popular and commercial software products are
  complex systems and composed of a number of
  modules.

4
Introduction (contd.)

• Project managers should know how to allocate the
  specified testing-resources among all the modules
  and develop quality software with high
  reliability.
• We provide a systematic method for the project
  managers to allocate specific amount of
  testing-resource expenditures for each module
  under some constraints minimizing the cost of
  software development, with a given fixed amount
  of testing-effort and a reliability objective.

5
SRGMs

• Software Reliability represents a
  customer-oriented view of software quality and is
  dynamic rather than static.
• SRGMs describe failures as a random process,
  which is characterized in either times of
  failures or the number of failures at fixed time.
• We use an SRGM with generalized logistic
  testing-effort function to describe the
  time-dependency behaviors of detected faults and
  the testing-resource expenditures spent during
  module testing.

6
Generalized Logistic Testing-Effort Function

• The generalized logistic testing-effort function
  (LTEF) was proposed by Huang, Kuo, and Lyu (1999)
  and can be depicted as
• where
• N is the total amount of testing
  effort to be eventually
• consumed,
• a is the consumption rate of
  testing-effort expenditures,
• A is a constant, and
• k is a structuring index with a
  large value for modeling
• well-structured software development
  efforts.

7
SRGM with Generalized Logistic Testing-Effort
Function

• This SRGM with Generalized LTEF was proposed by
  Huang, Kuo, and Lyu (1999) and the mean value
  function is

where m(t) is the expected mean number of
faults detected in time (0, t)
, Wk(t) is the current
testing-effort consumption at time t,
a is the expected number of initial faults,
and r is the error detection rate
per unit testing-effort at
testing time t rgt0.
8
SRGM with Generalized Logistic Testing-Effort
Function (contd.)

• When t?8, the expected number of faults to be
  detected is
• The reliability of a software system is defined
  as the ratio of the cumulative number of detected
  faults at time t to the expected number of
  initial faults

(if A gtgt N)
9
Testing-Resource Allocation for Module Testing

• Assumptions
• The software system is composed of N independent
  software modules that are tested individually.
  The number of faults remaining in each module can
  be estimated by an SRGM with generalized LTEF.
• For each module, the failure data have been
  collected and the parameters of each module can
  be estimated.
• The total amount of testing resource expenditures
  available for the module testing processes is
  fixed and denoted by W.

10
Testing-Resource Allocation for Module Testing

• Assumptions (contd.)
• If any of the modules is faulty, the whole
  software system fails.
• The system manager has to allocate the total
  testing resources W to each software module to
  minimize the number of faults remaining in the
  system during the testing period.
• The desired software reliability after the
  testing phase is greater than or equal to the
  reliability objective R0.

11
Mean Value Function of a Software System with N
Modules

• The number of detected faults in the system can
  be estimated by
• where vi is a weighting factor to measure the
  relative importance of a fault removal from
  module i in the future.
• If vi 1 for all i 1, 2,, N, the objective is
  to minimize the total number of faults remaining
  in the software system after the testing phase.

12
Mean Value Function of a Software System with N
Modules (contd.)

• Therefore, the number of remaining faults can be
  estimated by

13
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
The objective function is
minimizing the cost of software testing Subject
to the constraints (1) The total
amount of testing resource
expenditures for all modules would not be
more than available resource W. (2)
Reliability of each module should not be less
than R0 ? guarantee that
reliability of system will not be
less than R0 .
14
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
The objective function is Minimize
Subject to the constrains
, i1, 2,..., N.
15
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
Therefore, we have
, i 1,2,,N
Let
We can have ,
,i1,2,.,N
and ,where
Let , we can transform
the above equations to The following two
equation
16
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
Minimize Subject to
,i1,2, ..., N.
where
17
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
According to the Lagrange multiplier, above
equations be simplified as follows
Minimize
18
Minimizing the Software Cost with a Given Fixed
Amount of Testing-Effort and a Reliability
Objective
Based on the Kuhn-Tucker conditions (KTC) ,
the necessary conditions for a minimum value
of above equation are in existence . The
results will be applied in the Algorithm for
solving our problem.
19
Algorithm 1
Step 1 Set l0.
Step 2 Calculate the following equations
,i1, 2,..., N- l.
Step 3 Rearrange the index i such that
stop (i.e., the solution is optimal)
then
Step 4 IF
Else ll1
End-IF.
Step 5 Go to Step 2.
20
Observation 1

• The optimal solution has the following form

21
Observation 2

• Algorithm 1 always converges in, at worst, N-1
  steps and thus the value of objective function at
  the optimal solution is

22
Numerical Examples

• Suppose the total amount of testing-effort
  expenditures W is given.
• We apply the proposed model to actual software
  failure data and have to allocate the
  expenditures to each module to minimize the
  expected cost.
• All the parameters ai and ri for each software
  module have been estimated by using the maximum
  likelihood estimation (MLE) or the least squares
  estimation (LSE).

23
An example for Proposed Algorithm

• Table 1 the estimated values of ai , ri , vi ,
  and k .

Module ai ri k vi
1 89 4.182310-4 1 1.0 7632
2 25 5.092310-4 1 0.6 3158
3 27 3.961110-4 1 0.7 4009
4 45 2.295610-4 1 0.4 4329
5 39 2.533610-4 1 1.5 8964
6 39 1.724610-4 1 0.5 4568
7 59 8.81910-5 1 0.5 6023
8 68 7.27410-5 1 0.6 9112
9 37 6.82410-5 1 0.05 0
10 14 1.530910-4 1 1 2203
24
Sensitivity analysis

• If a1 is increased by 40,
• then the estimated value of optimal
  testing-effort
• expenditure for module 1 is changed from 7632
  to
• 8400 and its relative change (RC) is
  0.100628931
• about 10 increment).
• for modules 2, 3, 4, 5, 6, 7, 8, and 10, the
  estimated
• values of optimal testing-effort expenditures
  are
• decreased by about 0.95, 0.95, 1.52,
  0.67,
• 1.93, 2.87, 2.30, and 4.49, respectively.

25
Sensitivity analysis (contd.)
Figure 1 Relative change of OTEE for the case
of 40, 30, 20, and 10 increase to a1.
26
Sensitivity analysis (contd.)

• If a1 is decreased by 30
• the estimated value of optimal testing-effort
  expenditure
• for module 1 is changed from 7632 to 6818 and
  its RC is
• 0.106656184 (about 10.66 decrement).
• for modules 2, 3, 4, 5, 6, 7, 8, and 10, the
  estimated
• values of optimal testing-effort expenditures
  are increased
• by about 1.01, 1.02, 1.64, 0.71, 2.06,
  3.05,
• 2.44, and 4.86, respectively.

27
Sensitivity analysis (contd.)
Figure 2 Relative change of OTEE for the case
of 40, 30, 20, and 10 decrease to a1.
28
Sensitivity analysis (contd.)

• If a1 a2 both are increased by 40
• the estimated values of optimal testing-effort
• expenditure for modules 1 and 2 are changed
  from
• 7632 to 8370 (about 9.67 increment) and 3158
  to
• 3764 (about 19.18 increment), respectively.
• for modules 3, 4, 5, 6, 7, 8, and 10, the
  estimated
• values of optimal testing-effort expenditures
  are
• decreased by about 1.75, 2.79, 1.23, 3.52,
• 5.25, 4.19, and 8.22, respectively.

29
Sensitivity analysis (contd.)
Figure 3 Relative change of OTEE for the case
of 40, 30, 20, and 10 increase to a1 a2.
30
Conclusions

• In this paper, we proposes a method to optimize
  the software testing-resource allocation problem
  minimizes the cost of software development, with
  a given fixed amount of testing-effort and a
  reliability objective.
• We develop a comprehensive strategy for module
  testing in order to help software project
  managers make the best decisions in practice.

31
Conclusions (contd.)

• An extensive sensitivity analysis is presented to
  study the effects of various principal parameters
  on the optimization problem of testing-resource
  allocation.
• Using Algorithm 1, project managers can allocate
  limited testing-resource easily and efficiently
  and thus achieve the lowest cost objective during
  software module and integration testing.