Today

1
Today

• More on random testing symbolic constraint
  solving (concolic testing)
• Using summaries to explore fewer paths (SMART)
• While preserving level of branch coverage
• Whitebox fuzz testing (SAGE)
• Start on debugging
• Quick start Zellers delta-debugging algorithm
  and tools

2
SMART Using Summaries in Testing

• Idea complete coverage of all paths is too
  expensive
• Fix limit the number of paths, but preserve the
  level of branch coverage achieved by trying al
  paths
• How? Use function boundaries make the
  exploration compositional
• Compositional Dynamic Test Generation
  (Godefroid) paper is a good look at this kind of
  testing in general

3
The Compositional Approach

• int G 0
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G
2
32
4
How many paths through this program?
8
16
4
The Compositional Approach

• int G 0
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G
Traditional DART/CUTE will have to executethe
program 32 times, and explore 32 paths.
One idea treat each function independently
dont worry about paths that cross
functionboundaries but still solve for all
paths througheach function.
When we reach a return from a function, use
thebasic algorithm to find all paths through
that function.Afterwards, dont care what path
we take through it
5
The Compositional Approach

• int G 0
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G
2
4
2
2
4
4
Now how many paths?
2
Total 4 4 2 10 (vs. 32)
4
6
The Compositional Approach

• int G 0
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G
2
4
2
2
4
Have we lost anything?
Not branch coverage assumingcomplete
exploration is possible,and all constraints are
linear, wepreserve level of branch coverage.
4
2
4
But we could miss bugs
7
The Compositional Approach

• int G 0
• int a4
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G aG 3
2
4
2
2
4
4
2
Could handle by adding explicitbranch for
array bounds check,however.
4
8
The Compositional Approach

• int G 0
• int a4
• int foo (int x, int y) if (x lt y) Gif (x
  10) G
• int bar (int x, int y) if (x lt 50) Gif (y
  gt 8) G

int main () foo(x, y)bar(x, y)if (x y
40) G aG 3
2
4
2
2
4
4
When solving the constraints, SMARTmakes use of
procedure summariesto avoid re-analyzing
functions in fulldetail but these are a static
analysistopic more than a testing topic.
2
4
9
Whitebox Fuzz Testing

• SAGE
• Now well take a quick run through Godefroids
  invited talk at the 2007 Workshop on Random
  Testing

10
Debugging
11
Debugging, in the Trenches
Rasala put his hands on his desk and buried his
face in them. It was just another routine day
down at debugging headquarters.
In the back of Veress mind still lies a small
suspicion that the problem might after all be
noise. And now much to Guyers delight, when
he finds out later on it is Veres himself who
disconnects the I-cache. Then he runs the
program past the point of failure, and everything
works. He puts the I-cache back in and once
again Gollum fails. This doesnt prove the IP is
to blame, but it does tend to eliminate noise as
a suspect, once and for all. . .
from THE CASE OF THE MISSING NAND GATE (Chapter
10 of Kidders The Soul of a New Machine)
12
(The Soul of a New Machine)
Kidders book tells the storyof the development
of a micro-computer in the early 80s. The books
a classic won theAmerican Book Award for
non-fiction. Anyone who cares howcomputers are
made (or howpeople work) should read
it. Chapter 10 is a classic story ofdebugging a
hardware problem.The ideas apply just as well
tosoftware, and this is the bestdescription of
heavy-duty debugIve ever seen.
13
Debugging

• Debugging is really hard -- even with a good
  failing test case in hand
• One of the most time-consuming tasks in software
  development Ball, Eick
• Locating the fault is the most time-consuming
  part of debugging Vesey

14
Debugging

• Takes as much as 50 of development time on some
  projects
• Arguably the most scientific part of computer
  science practice
• Even though its usually done in a totally ad
  hoc, haphazard way!

Debugging is twice as hard as writing the
codein the first place. Therefore, if you write
the codeas cleverly as possible, you are, by
definition, notsmart enough to debug it. -
Brian Kernighan
15
Debugging and Testing

• What are test cases for, anyway?
• Often so we can locate and fix a fault
• Or so we can understand how serious the failure
  is, and triage / flight rule it away
• If we have many bugs, and some may not be
  important enough to merit resources
• Or if there is a reason we cant change the code
  and have to work around the problem
• In either case, we have a debugging task at
  hand must at least understand the failure, even
  if only to triage

16
Scientific Debugging

• Test cases (ones that fail and ones that succeed)
  can be the experiments we perform to verify our
  hypotheses
• The failing test case informs us that there is a
  phenomenon to explain (apple on the head)
• Generate (or examine) more test cases to find out
  more about what is going on in the program

17
Scientific Debugging
18
Testing for Debugging

• Several ways to use test cases in debugging
• Test case minimization (today)
• Shrink the test case so we dont have to look at
  lots of irrelevant or redundant operations
• Fault localization
• Give suggestions about where the fault may be
  (based on test case executions)
• Error explanation
• Give a story of causality
• (A causes B B causes C C causes failure)

attempt to automate part of scientific debugging