Intro

1
Delta Debugging
Xiangyu Zhang (slides adapted from Tevfik
Bultans )
2
Problem

• In 1999 Bugzilla, the bug database for the
  browser Mozilla, listed more than 370 open bugs
• Each bug in the database describes a scenario
  which caused software to fail
• these scenarios are not simplified
• they may contain a lot of irrelevant information
• a lot of the bug reports could be equivalent
• Overwhelmed with this work Mozilla developers
  sent out a call for volunteers
• Process the bug reports by producing simplified
  bug reports
• Simplifying means turning the bug reports into
  minimal test cases where every part of the input
  would be significant in reproducing the failure

3
An Example Bug Report

• Printing the following file causes Mozilla to
  crash

MULTIPLE SIZE7 AllVALUE"Windows 3.1"Windows 3.1VALUE"Windows 95"Windows 95VALUE"Windows 98"Windows 98VALUE"Windows ME"Windows MEVALUE"Windows 2000"Windows 2000VALUE"Windows NT"Windows NTSystem 7"Mac System 77.5"Mac System 7.57.6.1"Mac System 7.6.18.0"Mac System 8.08.5"Mac System 8.58.6"Mac System 8.69.x"Mac System 9.xX"MacOS XLinuxVALUE"BSDI"BSDIFreeBSDPTION VALUE"NetBSD"NetBSDOpenBSDAIX
Continued in the next page
4
VALUE"BeOS"BeOSHP-UXN VALUE"IRIX"IRIXNeutri
noOpenVMSVALUE"OS/2"OS/2OSF/1N VALUE"Solaris"SolarisVALUE"SunOS"SunOSotherLECT NAME"priority" MULTIPLE SIZE7 VALUE"--"--P1VALUE"P2"P2P3VALUE"P4"P4P5
severity" MULTIPLE SIZE7 VALUE"blocker"blockercr
iticalmajorrmal"normalminorVALUE"trivial"trivialenhancement
5
Delta-Debugging

• It is hard to figure out what the real cause of
  the failure is just by staring at that file
• It would be very helpful in finding the error if
  we can simplify the input file and still generate
  the same failure
• A more desirable bug report looks like this
• Printing an HTML file which consists of
• causes Mozilla to crash.
• The question is Can we automate this?
• Andreas Zeller

6
Overview

• Lets use a smaller bug report as a running
  example
• When Mozilla tries to print the following HTML
  input it crashes
• How do we go about simplifying this input?
• Manually remove parts of the input and see if it
  still causes the program to crash
• For the above example assume that we remove
  characters from the input file

7
Bold parts remain in the input, the rest is
removed
1 F
2
P 3
P 4
P 5
F 6
F 7
P 8
P 9
P 10
F 11
P 12
P 13
P
F means input caused failure P means input did
not cause failure (input passed)
8
14
P 15
P 16
F 17
F 18
F 19
P 20
P 21
P 22
P 23
P 24
P 25
P 26 F
9
Example

• After 26 tries we found that
• Printing an HTML file which consists of
• causes Mozilla to crash.
• Delta debugging technique automates this approach
  of repeated trials for reducing the input.

10
Changes That Cause Failures

• To describe the algorithm we first need to define
  the process
• In general the delta debugging technique deals
  with changeable circumstances
• circumstances whose change may cause a different
  program behavior
• You can think of circumstances (meaning
  changeable circumstances) as all the possible
  behaviors of the environment of the program

11
Circumstances and Failures

• Let E be the set of possible configurations of
  circumstances
• each r ?? E determines a specific program run
• If the environment is fixed the program executes
  deterministically
• rP ?? E corresponds to a run that passes
• rF ?? E corresponds to a run that fails

12
Changes

• We can go from one circumstance to another by
  changes
• A change ? is a mapping ? E ? E which takes one
  circumstance and changes it to another
  circumstance
• changes one program run to another program run by
  changing its environment (input)
• A relevant change ? is a change such that ?(rP)
  rF
• Change ? makes the program fail

13
Decomposing Changes

• A change ? can be decomposed to a number of
  elementary changes ?1, ?2, ..., ?n where ? ?1
  o ?2 o ... o ?n
• where (?i o ?j)(r) ?i(?j(r))
• For example, deleting a part of the input file
  can be decomposed to deleting characters one by
  one from the input file
• another way to say it by composing deleting of
  single characters we can get a change that
  deletes part of the input file

14
Testing

• Given an r ? E we define a function rtest E ?
  P, F, ?
• The function rtest determines if an r ? E causes
  the program to fail (F) or pass (P)
• The output ? means that the result is
  indeterminate
• Maybe the input caused another failure other than
  the one we are trying to capture
• We can compute the function rtest by running the
  program

15
Test Cases

• Given a run rP that does not cause a failure
• cF ?1, ?2, ..., ?n denotes a set of changes
  such that
• rF (?1 o ?2 o ... o ?n )(rP)
• cP is defined as cP ?
• c ? cF is called a test case
• To summarize
• We have a run without failure rP
• We have a set of changes cF ?1, ?2, ..., ?n
  such that
• rF (?1 o ?2 o ... o ?n )(rP) where rF is a run
  with failure
• Each subset c of cF is a test case

16
Testing Test Cases

• Given a test case c, we would like to know if the
  run generated by changing rP by the changes in c
  is a run that causes a failure
• We define a function
• test Powerset(cF) ? P, F, ?
• such that, given c?1, ?2, ..., ?m ? cF
• test(c) rtest((?1 o ?2 o ... o ?n )(rP))
• Note that
• test(cP) test(?) P
• test(cF) test(?1, ?2, ..., ?n) F

17
Minimizing Test Cases

• Now the question is Can we find the minimal test
  case c such that test(c) F?
• A test case c ? cF is called the global minimum
  of cF if
• for all c ? cF , c
• Global minimum is the smallest set of changes
  which will make the program fail
• Finding the global minimum may require us to
  perform exponential number of tests

18
Minimizing Test Cases

• A test case c ? cF is called a local minimum of
  cF if
• for all c ? c , test(c) ? F
• A test case c ? cF is n-minimal if
• for all c ? c , c ? c ? n ? test(c) ? F
• A test case is 1-minimal if
• for all ?i ? c , test(c ?i) ? F

19
Minimization Algorithm

• The delta debugging algorithm finds a 1-minimal
  test case
• It partitions the set cF to ?1, ?2, ... ?n
• ?1, ?2, ... ?n are pairwise disjoint
• cF ?1 ? ?2 ? ... ? ?n
• Define the complement of ?i as ?i cF ? ?i
• Tests each test case defined by the partition and
  their complements
• Reduce the test case if a smaller failure
  inducing set is found
• otherwise refine the partition

20
Each Step of the Minimization Algorithm

• Test each ?1, ?2, ... ?n and each ?1, ?2, ..., ?n
• There are four possible outcomes
• Some ?i causes failure
• Partition ?i to two and continue with ?i as the
  test set
• Some ?i causes failure
• Continue with ?i as the test set with n ? 1
  subsets
• No test causes failure
• Increase granularity by generating a partition
  with 2n subsets
• The granularity can no longer be increased
• Done, found the 1-minimal subset
• In the worst case the algorithm performs cF2
  3cF tests
• For example an n character input requires n23n
  tests in the worst case (highly unlikely to occur
  in practice)

21
Note Changes vs. Minimizing the Input

• Our goal is to find the smallest change ? that
  makes the program fail, i.e, ?(rP) rF
• If we pick rP to be some trivial input (such as
  empty string) for which the program does not
  fail, then finding the smallest fault inducing
  change becomes equivalent to finding the smallest
  input that makes the program fail.

22
Example

• Assume that
• Input string abdefgh causes the program fail
  rF abdefgh
• Program does not fail on empty string rP ?
• If we assume that any string that contains
  causes the program fail, then how will the
  delta-debugging algorithm behave for this case?
• Lets define ?i to mean Include the ith
  character of the failure inducing case in the
  modified input
• Then, for our example, we have
• ?1(?) a, ?2(?) b, ?3(?) , ?4(?) d,
• Note that, the changes are composable
• ?2(?1(?)) ab, ?3(?2(?1(?))) ab,
  ?4(?3(?2(?1(?)))) abd,

23
Example

• Lets run the algorithm
• Initially ?1 ?1, ?2, ?3, ?4 ?2, ?2 ?5,
  ?6, ?7, ?8 ?1
• test(?1) test(?1, ?2, ?3, ?4)
  rtest(?4(?3(?2(?1(?))))) rtest(abd) F
• test(?2) test(?5, ?6, ?7, ?8)
  rtest(?5(?6(?7(?8(?))))) rtest(efgh) P
• This means that we are in the case 1 of the
  algorithm
• Some ?i causes failure
• Partition ?i to two and continue with ?i as the
  test set

24
Example

• We partition ?1 as ?1, ?2 and ?3, ?4
• So now we have ?1 ?1, ?2 ?2, ?2 ?3,
  ?4 ?1
• test(?1) test(?1, ?2) rtest(?2(?1(?)))
  rtest(ab) P
• test(?2) test(?3, ?4) rtest(?4(?3(?)))
  rtest(d) F
• This means that we are again in case 1 of the
  algorithm
• Some ?i causes failure
• Partition ?i to two and continue with ?i as the
  test set

25
Example

• We partition ?1 as ?3 and ?4
• So now we have ?1 ?3 ?2, ?2 ?4 ?1
• test(?1) test(?3) ctest(?3(?)) ctest()
  F
• test(?2) test(?4) ctest(?4(?)) ctest(d)
  P
• We are now in case 4 of the algorithm
• 4. The granularity can no longer be increased
• Done, found the 1-minimal subset
• The result is ?3(?)

26
(No Transcript)
27
Case Studies

• The following C program causes GCC to crash

define SIZE 20 double mult(double z, int
n) int i , j i 0 for (j 0 j j) i i j 1 zi zi
(z01.0) return zn
Continued in the next page
28
void copy(double to, double from, int
count) int n count 7) / 8 switch(count
8) do case 0 to from case
7 to from case 6 to from
case 5 to from case 4 to
from case 3 to from case 2
to from case 1 to from
while ( --n 0) return mult(to, 2) int
main(int argc, char argv) double xSIZE,
ySIZE double px x while (px SIZE) px (px x) (SIZE 1.0)
return copy(y, x, SIZE)
29
Case Studies

• The original input file 755 characters
• Delta debugging algorithm minimizes the input
  file to the following file with 77 characters
• If a single character is removed from this file
  then it does not induce the failure

t(double z,int n)int i,jfor()iij1ziz
i (z00)returnn
30
Isolating Failure Inducing Differences

• Instead of minimizing the input that causes the
  failure we can also try to isolate the
  differences that cause the failure
• Minimization means to make each part of the
  simplified test case relevant removing any part
  makes the failure go away
• Isolation means to find one relevant part of the
  test case removing this particular part makes
  the failure go away
• For example changing the input from
• to
• SELECT NAME"priority" MULTIPLE SIZE7
• makes the failure go away
• This means that inserting the character failure inducing difference
• Delta debugging algorithm can be modified to look
  for minimal failure inducing differences
• Although it is not as popular, it is quite useful
  in some applications.

31
Failure Inducing Differences Example

• Changing the input program for GCC from the one
  on the left to the one on the right removes the
  failure

This input does not cause failure
This input causes failure
define SIZE 20 double mult(double z, int n)
int i , j i 0 for (j 0 j i j 1 zi zi (z01.0)
return zn
define SIZE 20 double mult(double z, int n)
int i , j i 0 for (j 0 j i i j 1 zi zi
(z01.0) return zn
Modified statement is shown in box
32
Discussions

• How to compose a failure report for general
  purpose applications
• Input? (gui, env)
• Output? (how to express?)
• Textual description often fail to provide direct
  help
• How to associate symptoms to code?
• DD on scheduling decisions
• Given a thread schedule for which a concurrent
  program works and another for which the program
  fails, delta debugging algorithm can narrow down
  the differences between two thread schedules and
  find the locations where a thread switch causes
  the program to fail.
• Chipping
• Given two versions of a program such that one
  works correctly and the other one fails, delta
  debugging algorithm can be used to look for
  changes which are responsible for introducing the
  failure
• Fault Localization apply DD to memory state

33
Discussions

• the rtest function.
• a large number of runs required.
• DD Jockey.