Software Testing Part II

1
Software TestingPart II

• March, 2001

2
Fault-based Testing Methodology(white-box)
Mutation Testing
2
3
Klaidos ivedimas

• Automatinis programos testavimas
• Klaidos ivedimas
• Automatinio testavimo pakartojimas

4
Strong mutation testing

• Called fault-based because its directed at the
  possible faults that can occur in a program
• Provides a measure of test-data adequacy
• The technique begins by generating test data by
  some means it does not really matter how it is
  generated (can be randomly generated)

5
Strong mutation testing (cont)

• Approach generate variations of the original
  program, called mutants
• The original test data is run through both the
  original program and the mutant
• If there is a difference in the output, then the
  mutant is dead

6
Strong mutation testing (cont)

• If the mutant is not killed, then the test data
  is not adequate to reveal the fault and
  distinguish the mutant from the original
• If this is the case, then the test data must be
  augmented to reveal the fault and kill the live
  mutant
• The obvious problem too many simple faults that
  can be introduced, O(n2) for an n line program

7
List of possible faults, Budd (1983)

• constant replacement
• scalar replacement
• scalar variable for constant replacement
• constant for variable
• array reference for constant replacement
• array reference for scalar variable

• constant for array reference replacement
• scalar variable for array reference replace
• array reference for array reference replace
• constant replacement
• data statement alteration

8
List of possible faults, Budd (1983)

• absolute value insert
• unary operator insertion
• statement analysis
• statement deletion
• return statement replace
• goto label replace
• do statement end replace

• comparable array name replacement
• arithmetic operator replacement
• relational operator replacement
• logical connector replacement

list used in Fortran programs
9
How effective is mutation testing?

• Test data that distinguishes all programs
  differing from a correct one by only simple
  errors is so sensitive that it also finds complex
  errors, DeMillo (1978)
• mutation testing makes the competent programmer
  assumption programmers write programs that are
  nearly correct!

10
Example program

• string searching program the program prompts the
  user for a positive integer in the range 1 to 20
  and then for a string of characters of that
  length. The program then prompts for a character
  and returns the position in the string at which
  the character was first found or a message
  indicating that the character was not present in
  the string. The user has the option to search for
  more characters.

11
main() (1 ) char a20, ch, response
y (2) int x, i (3) bool found (4)
cout ltlt Input an integer between 1 and 20
(5) cin gtgt x (6) while ( x lt 1 x gt
20) (7) cout ltlt Input an integer between
1 and 20 (8) cin gtgt x (9)
cout ltlt input ltlt x ltlt characters (10)
for (int i 0 i lt x i) cin gtgt ai (11)
cout ltlt endl
11
12
(12) while ( response y response
Y) (13) cout ltlt input character to
search (14) cin gtgt ch (15)
found false (16) i 0 (17)
while (!found i lt x) (18) if
(ai ch) found true (19) if
(found) cout ltlt ch ltlt at ltlt i ltlt endl (20)
else cout ltlt ch ltlt not in string ltlt
endl (21) cout ltlt search for another
character? y/n (22) cin gtgt
response
string searching program
12
13
mutation testing string example

• Assume we have branch tested the program
• consider lines 15 to 18

Change line (15) to (15) found true
call this mutant 1
(15) found false (16) i
0 (17) while (!found i lt x) (18)
if (ai ch) found true
14
Test data for branch testing
How does the output for the mutant program
compare with the original?
15
RIP

• Since the output of the mutant program differs
  from the output of the original program, mutant
  1 is dead!
• It is important to make only one small change so
  that faults dont cancel each other

16
mutation testing string example

• Assume we have branch tested the program,
• create another mutant

Change line (16) to (16) i 1
(15) found false (16) i
0 (17) while (!found i lt x) (18)
if (ai ch) found true
call this mutant 2
17
Mutant 2 lives!

• The output of the mutant is the same as the
  original program gt mutant 2 is alive
• Analysis of the program reveals that the test
  data is not adequate to reveal the fault add a
  test case to search first position

18
RIP
we add a test case to search first position
mutant 2 shows the inadequacy of the test data
19
Super mutants!

• Sometimes its impossible to kill a mutant
  because no test data can be found that will
  reveal the newly introduced fault
• an absurd example

if we created a mutant by changing the 10 to 11,
we could not kill the newly created mutant!
main() int x 3 int y x3 if (y lt
10) cout ltlt less
20
Suppose the string processing program had
contained a fault!
If we generated a mutant with i 0, our original
test data would not kill the mutant!
(15) found false (16) i
1 (17) while (!found i lt x) (18)
if (ai ch) found true
analysis of the program would reveal (1) the
original test data was inadequte, and (2) the
program has a fault!
21
Strengths of strong mutation testing

• Shows the absence of particular faults contrary
  to Dijkstras comment! By introducing a fault
  into the program and then showing that the test
  data reveals it, we know that the fault cannot be
  there!
• Forces the programmer to scrutinize ( kruopšciai
  ištirti) and analyze the program under test to
  think of test data to expose the newly introduced
  fault, i.e., kill the mutant!

22
Weaknesses of strong mutation testing

• Computationally expensive!
• Huge number of mutants can be generated
• There are systems that can help generate mutant
  programs and test data but much work must be
  done by hand.

23
Weak mutation testing

• Less demanding that strong mutation
• take some construct of the program and create
  some mutants of this construct
• consider if (A B C), possible mutants
• if (A B C)
• if (A B C)
• if ((A B) C)
• etc.

run the original test data and kill the mutants
24
Weak mutation testing (cont)

• Howden (1981) identified five candidate
  constructs
• data access (or variable reference)
• data storage (or variable assignment)
• arithmetic expression
• arithmetic relation
• boolean expression same as multiple condition
  coverage
• Howden developed reliable tests for the five
  constructs

25
Consider data access or variable reference

• Were considering the r-value here.
• Aim is to ensure that the correct variable is
  being accessed.
• consider the variables in string processing
  program, type-by-type
• integer variables are referenced on lines 9, 10,
  17, 19 and 20 make sure we have adequate test
  data to ensure that they are getting correct
  values
• bool variable is referenced on line 15

26
Consider arithmetic relations

• they occur on lines 6 and 17 test data that
  ensures that x has values 0, 1, 2, 1, 20 and 21
  would test x and i

27
Strengths of weak mutation

• Like strong mutation testing, it promotes a
  thorough analysis of the program under test
• computationally cheaper
• not necessary to physically generate the mutants

28
weaknesses of weak mutation

• Unlike strong mutation, it is not reliable for
  the program as a whole adequacy of the data is
  local to the construct being mutated
• Girgis and Woodward (1986) found that weak
  mutation was outperformed by data flow testing
  they also found that different faults were found
  by each technique and promoted a combination of
  testing techniques