Testing Basics

1
Testing Basics

• Detection probability

2
Methodology to evaluate testing criterion

• Pick sample program, sample faults, sample
  operational profile, two criteria.
• For each criterion, randomly generate test suites
  that satisfy the criterion.
• Calculate the percentage of the suites that
  detect each of the faults.
• The criterion that has the higher percentage is
  better.

3
Triangle Example
cin gtgt a gtgt b gtgt c type scalene if (a b
a c b c) type isosceles if (a
b a c) type equilateral if (a gt bc
b gt ac c gt ab) typenot a
triangle if (a lt 0 b lt 0 c lt 0)
typebad input coutltlt type
4
Control Flow Graph
Operational profile 3,3,3 abcdegi equi 3,3,4 abce
gi isos 3,3,5 abcegi isos 3,3,6 abcefgi not 3,4
,3 abcegi isos 3,4,4 abcegi isos 3,4,5 acegi sc
al 3,4,6 acegi scal All inputs are equally
likely
5
What are the failure probability for each color
(separately)?
cin gtgt a gtgt b gtgt c type scalene if (a b
a c b c) type isosceles if (a
b a c) type equilateral if (a gt bc
b gt ac c gt ab) typenot a triangle if
(a lt 0 b lt 0 c lt 0) typebad
input coutltlt type
Blue Green Red
6
TTYP probability of detection

• What is the probability of detection with one
  randomly chosen test case?
• What is the probability of detection with two
  randomly chosen test cases?

7
(No Transcript)
8
TTYP per path

• What is the probability of detection with one
  randomly chosen test case per path?
• What is the probability of detection with an
  equal number of randomly chosen test cases?

9
TTYP smaller subdomains

• What might be better smaller subdomains?

10
TTYP - subdomains

• Are paths the best subdomains?
• Would a functional decomposition be better?
• Should we re-define the term subdomain?

11
Comparing Criteria
fault 1 fault 2 fault 3 fault 4
criterion 1 0.08 0.03 0.04 0.85
criterion 2 0.07 0.1 0.05 0.91
criterion 3 0.06 0.12 0.08 0.88
criterion 4 0.05 0.4 0.06 0.89
12
Solving this choice

• What are the assumptions?
• Analogy with dice?

13
TTYP smaller subdomains

• What might be better smaller subdomains?
• Would MCC (multiple condition coverage) be better
  subdomains

14
TTYP2 C0 and C1 coverage

• How do we deal with C0 and C1 coverage since they
  are not subdomain testing methodologies?

15
TTYP3

• How could you estimate the det prob of C0 or C1
  testing?

16
Marble Problem

• Assume that there is a bag of marbles from which
  marbles are drawn with replacement.
• What is the maximum likelihood estimate of p (the
  probability of drawing a purple) marble if you
  draw exactly n purple marbles in a row?
• ? P such that (p)n(1-p) is max or
• ? P such that (p)n .5

17
Evaluating Testing Methods by Delivered
Reliability
For Tuesday, Sep 4 study through section 2.3

• Frankl, Hamlet, Littlewood, Strigini
• IEEE TOSE Aug98