A Guest Lecture on

1
CSE 565 Software Verification, Validation, and
Testing
A Guest Lecture on Testing-Based Software and
SystemReliability Evaluation (Part 2)
Dr. Yinong Chen
2
Basic Input Domain Models
Examples MacWilliams73, BrownLipow75, Nelson78.
Software reliability is defined as the
probabilityR(N) Probno of failures over N
application runs where N is the exposure period
whose time unit is the number of application
runs. Assuming that input cases are selected
independently, then R(N) can be expressed
as R(N) (R(1))N RN where, R ? R(1) is the
expected reliability per application run. Now
the question is how to estimate R.
3
Estimate R
R, the reliability per test run, can be defined
by the ratio of the number of test runs in which
failures are observed and the total number of
test runs when infinite number of different input
cases are applied for test runs R 1 F 1
Because of test time limit only a subset of
the entire input domain can be applied to test
the program in practice. Thus the reliability per
test run, R, is usually estimated by
4
Example
Test the program 10 000 time (test runs) Five (5)
failures are observed F 5 / 10 000 1 / 2
000 R 1 - 1 / 2 000 0.9995 Reliability in N
application runs are R(N) RN
5
MacWilliams 73 and BrownLipow 75 With partition
and profiling
MacWilliams 73 the s input cases are selected
randomly from the input domain. BrownLipow 75
the input domain is partitioned into m classes.
If si input cases are selected from class Ci and
fi failures are observed, the reliability can be
calculated by where P(Ci) is a probability
function reflecting the input profile in terms of
classes.
6
Example
The input domain is partitioned into 10
sub-domains.



-

1 0.020938 0.979063
R
1
F
7
Case Study 1Anti-lock Braking System (ABS)

• Requirement
• To obtain the maximum braking effect
• Algorithm
• Define (or measure) the wheel diameter
• Measure the wheel rotations per seconds rps
• Compute the wheel velocity wv
• Measure the body velocity bv
• Error detection and action
• if (bv gt wv), reduce braking force
• else if (bv lt wv), reduce acceleration force
• else no action

8
Sample Code in C
include ltiostreamgt using namespace std const
float mile_inch 63360 const float pi
3.1416 float wheel_diameter 15 //
inches float wheel_sensor() float rps cout
ltlt "get rotations per second " ltlt endl rps
ReadWheelRotationSensor() return rps
9
float wheel_velocity(float rps) float wv wv
(pi wheel_diameter rps 3600)/mile_inch r
eturn wv float body_velocity() float
bv cout ltlt "get miles per hour " ltlt endl bv
ReadBodySpeedSensor() return bv
10
void error_detection(float wv, float bv) if
(abs(bv - wv) lt 0.01) cout ltlt "no action" ltlt
endl else if (bv gt wv) cout ltlt "reduce
brake force!" ltlt endl else cout ltlt "reduce
acceleration force!" ltlt endl void evaluation()
float rps, wv, bv rps wheel_sensor() wv
wheel_velocity(rps) bv body_velocity() err
or_detection (wv, bv) void main() for (i
1, I lt 10000, i) evaluation()
11
Apply the input domain reliability model

• Write the random function that simulates
  ReadWheelRotationSensor()
• Write the random function that simulates
  ReadBodySpeedSensor()
• Test the program and collect date
• Apply the input domain model to evaluate the
  reliability of the program

12
Case Study Testing greatest common divisor
program
Source Y. Chen, W.T. Tsai, Introduction to
programming languages Programming in C, C,
Scheme, Prolog, C, and SOA, second edition,
Kendall/Hunt Publishing Company, 2006, ISBN
0-7575-2974-7. Section 1.5.2.
13
greatest common divisor program
include ltstdio.hgt int gcd (int n0, int m0) //
n0 ? 0, m0 ? 0 int n, m // n0 or m0 ? 0 n
n0 m m0 while (n ! 0 n ! m) if (n lt
m) m m - n else n n - m return
m void main() int i, j, k scanf("d\nd",
i, j) //input k gcd(i, j) // call
gcd printf("d\n", k) // output
14
Random Test Case Generation
Input Output k (i, j) (6, 9) 3 (i, j) (10,
5) 5 (i, j) (0, 4) 4 (i, j) (5, 7) 1 (i, j)
(8, 29) 1
The program produces correct outputs for all
these different test cases. Can we claim that
the program is correct?
15
Example Input Domain Partition Analysis

• Input Domain Analysis
• The program takes two integers as input.
• The branches of the program are controlled by the
  relative values of the two integers.
• Input Domain Partitioning
• Partition each integer input into three groups
  lt 0, 0, and gt 0.
• For this program, lt 0 is not allowed. The group
  that has only one value is called boundary value.
• Considering the semantics of the program, prime
  numbers and nonprime numbers play a role in the
  program. Thus, the positive integers are further
  divided into prime and nonprime numbers.

16
Input Domain Partition
Based on the analysis, we have a partition of
i 0 2, 3, 5, 7, 11, 4, 6, 8, 9, j
0 2, 3, 5, 7, 11, 4, 6, 8, 9, The
combination of the two inputs generates
following cases (0, 0) This case is not allowed
according to the specification. (0, 2), (0, 3),
(0, 5), (0, 7), (0, 11), (0, 4), (0, 6), (0,
8), (0, 9), (2, 0), (2 2), (2, 3), (2, 5), (2,
5), (2, 4), (2, 6), (2, 8), (2, 9), (3, 0),
(3, 2), (3, 3), (3, 5), (3, 7), . . . (9, 0),
(9, 2), (9, 3), (9, 5), (9, 7),
17
Coverage Consideration Branch Coverage
?
?
no
n ! 0?
?
yes
?
no
n ! m?
yes
?
?
no
n lt m?
?
yes
m m - n
return m
n n - m
?
?
?
Exit
(0, 2),(0, 3),(0, 9),(0, 10) ??? (2, 2),(3, 3),
(9, 9),(10, 10) ??? (2, 3),(2, 9), (2,
10) ?????... (3, 2),(9, 2),10, 2),(10, 3),(10,
9) ?????... (2, 0),(3, 0), (9, 0),(10,
0) ?????...
18
Testing the Program Using Test Cases
Inputs Output k (i, j) (0, 2) 2 (i, j) (2,
2) 2 (i, j) (2, 3) 1 (i, j) (3, 2) 1 (i, j)
(2, 0) ?
19
Apply the partition-based input domain
reliability model

• Modify the main program, so that it
  systematically generate (large number of) input
  cases from different sub-domain
• Test the gcd function and collect data
• Apply the partition-based input domain model to
  evaluate the reliability of the program

20
SUMMARY SOFTWARE RELIABILITY MODELS

• Basic concepts and terminology
• Faults and failure rates
• Reliability R(t) and availability A(t)
• Software reliability models
• Classifications
• Time-domain models
• Fault count model
• Input-domain models
• Fault seeding model
• Sample code of real-time ABS software
• Sample code of real-time ABS software
• Partition-based input domain testing

21
Modeling Complex Systems (Software and Hardware)

• A large system can be decomposed into smaller
  components.
• Evaluate the reliability of the components
• Evaluate the reliability of the system based on
  known component reliabilities
• Combinatorial Models
• Markov Models

22
Markov Models
Markov models are more generic than combinatorial
models. They can handle repairs and much more
complex situations. Assumption Any component
may in one the two states working or
failed Probability of state transition depends
only on the current state. ß Failure rates and
repair rates are constants. ß Transition
probability is proportional to the time that the
component stays at a state. ß Exponential
distribution of the reliability/availability
23
Steps of Applying Markov Models
A system consists of multiple components ß Constr
uct state transition diagram
(1)
24
Step 1 Construct state transition diagram
Example 1 Simplex system with repair
25
Step 1 Construct state transition diagram
Example 2 Reliability of TMR system with repair
26
Step 1 Construct state transition diagram
Example 3 A ring system with different node and
link failure rates a and b. Assume that the
system fails if any two or more than components
failed.
Failed
27
Step 2 Construct differential equations
A(t) p0(t)
The question is how to obtain the probability of
each state.
p0 (t Dt) (1 l  Dt)   p0 (t) m  Dt 
p1 (t) p1 (t Dt) l  Dt p0 (t) (1 m 
Dt)   p1 (t)
Solve the differential equations to obtain (p0
(t), p1 (t)).
28
Step 2 Construct differential equations
l
0
1
m
29
Step 3 Solve differential equations

• There are many different ways to solve
  differential equations
• LaPlace Transformation
• Tools like MatLab or Mathematica

30
Step 4 Find the Probabilities of Working States
m
l
p0
-
l

m
(
)
t
(
t
)


e
l

m
l

m
l
m
-
l

m
p1
(
)
t
(
t
)


e
l

m
l

m
If m 0, the probability at p0 represents the
reliability
m
l
-
l

m
(
)
t
-
l

t
R
(
t
)


e
e
l

m
l

m
31
Step 2 Construct differential equations (Find
the pattern)
32
Step 2 Construct differential equations (Find
the pattern)
In general, assume a STD has n states and is
fully connected. Any state has n incoming and n
outgoing transitions
aij ? 0 is the transition rate from state i to
j. For i, j 1, 2, ..., n, and i ? j.
33
Step 2 Construct differential equations (Find
the pattern)
The probability in state j at t Dt the
probability in state j at t incoming prob
outgoing prob
Math manipulation Divide Dt on both sides, let
Dt ? 0
34
Step 2 Construct differential equations (More
detail of the previous slide)
35
Step 2 Construct differential equations (found
the pattern)
36
Example 1 Apply the Pattern
-3
l
0
m
æ
ö
ç
T

-(2lm)

0
ç

0
è
ø
0
2l
R(t) p1(t) p2(t)
37
Example 2
1
2
3
4
R(t) p1(t) p2(t) p3(t)
38
SUMMARY

• Basic concepts of reliability and reliability
  modeling
• Hardware reliability models
• Software reliability models
• System reliability models consisting of multiple
  components
• Combinatorial models
• Markov models