Mutation Testing

1
Mutation Testing

• SE 452, Software Verification and Validation
• Mark Ardis, RIT

2
Mutation Motivation

• Path testing exercises the control of a program,
  not the computations along the paths
• Most programs under test are "almost" right

3
Mutation Method

• Pre-process program to generate mutants
• Execute all the mutants and the original program
• If a mutant differs from the original, then it is
  "killed"
• If all mutants are killed, then test set is
  adequate.

4
Example

• Original program
• function square( x integer) integer
• begin
• square x x
• end
• Mutant
• function square2( x integer) integer
• begin
• square2 x x
• end

5
Executing Mutants

• Test set 2 does not kill the mutant
• Test set 2, 4 does kill the mutant, and it
  reveals the flaw in the program
• Original program
• function square( x integer) integer
• begin
• square x x
• end
• Mutant
• function square2( x integer) integer
• begin
• square2 x x
• end

6
Which Mutants?

• Off by one errors (i1, i, i-1)
• Different variable (i, j, k)
• Other likely errors

7
Assumptions ofMutation Testing

• Competent Programmer The perfect program is
  very close to the program under test
• Oracle Hypothesis You know the output for any
  input
• Continuity Every complicated mistake can be
  caught by looking for simple mistakes.

8
Problems with Mutation

• Very expensive
• each test runs the whole program
• many mutants for every program
• Some mutants fail to halt
• May be difficult to detect when a mutant is
  really just an equivalent program
• Not reliable may not try the right mutants