Backstop: A Tool for Debugging Runtime Errors

1
Backstop A Tool for Debugging Runtime Errors

• Chris Murphy, Eunhee Kim, Gail Kaiser, Adam
  Cannon
• Columbia University

2
Observation

• Novice/CS1 Java programmers have difficulty
  dealing with uncaught exceptions and the
  resulting stack traces

java.lang.NumberFormatException For input
string "4.5" at java.lang.NumberFormatException
.forInputString (NumberFormatException.java
48) at java.lang.Integer.parseInt(Integer.java
456) at java.lang.Integer.parseInt(Integer.java
497) at Test.main(Test.java124)
3
Problems

• Students are often confused by runtime errors in
  Java
• Stack traces were not developed with the
  introductory-level CS student in mind!
• Few, if any, tools exist to help novice
  programmers interpret and debug runtime errors

4
Our Solution Backstop

• A command-line tool to produce detailed and
  helpful error messages when an uncaught runtime
  error (exception) occurs
• Also provides debugging support by displaying the
  lines of code that are executed as a program
  runs, as well as the values of any variables on
  those lines

5
Hypothesis

• Backstop allows students to understand and debug
  runtime errors more easily than just the Java
  stacktrace
• Backstop is part of Columbia Universitys ongoing
  efforts to develop tools to help CS1 students
  work with Java

6
Overview

• Related Work
• Handling of Runtime Errors
• Usability Study
• Debugging Support
• Conclusions and Future Work

7
Related Work

• Programming environments with easier-to-understand
  compiler errors
• BlueJ, ProfessorJ, DrJava,
• Identify most common compiler errors
• Gauntlet, Expresso, HiC,
• Identifying logical errors
• InSTEP, DEBUG,
• Debugging
• CMeRun

8
Handling of Runtime Errors

• Designed to catch any uncaught exceptions in the
  students program
• Once the exception (or any other Java runtime
  error) is caught, interpret its meaning and
  provide a user-friendly message that also seeks
  to provide some enlightenment as to what caused
  the error

9
Handling of Runtime Errors

• BackstopRuntime wraps the students (compiled)
  Java program
• Uses reflection to invoke the main method
• Catches any uncaught exception and dispatches it
  to the appropriate handler
• StackTraceElements and source code are used to
  produce a friendlier, more helpful error message
  that is appropriate for the type of exception

10

ERROR

An error
occurred at line 124 of Test.java in the
method "main". I'm sorry, the program cannot
continue. The line of code that caused the error
is int myNum Integer.parseInt(arg) It
seems that the code wants to convert a String to
an integer. However, the String "4.5" appears to
be a floating point value, not an integer. Dont
forget, an integer cant contain a decimal point.
You may want to use a different datatype to store
the value, like float. This is called a
NumberFormatException. The full error is as
follows java.lang.NumberFormatException For
input string "4.5" at java.lang.NumberFormatEx
ception.forInputString (NumberFormatException
.java48) at java.lang.Integer.parseInt(Integer
.java456) at java.lang.Integer.parseInt(Intege
r.java497) at Test.main(Test.java124)
Refer to the code, not you
Reinforce what is already known
Make a suggestion
Also use Java terms
11
Usability Study

• Research Question Is the information provided
  by Backstop helpful for understanding runtime
  errors?
• 17 students (8 male, 9 female) who had just
  completed a CS1 summer course
• Students were asked to find/fix a runtime error
  in code they did not write
• After completion, were asked subjective questions
  about the experience

12
Usability Study

• Subject was shown the code for a program that
  reads a String and displays the number of
  occurrences of each letter
• Subject was then told how the algorithm works
• It was then suggested that the code would produce
  an error for some inputs
• Subject was suggested to try a sentence like
  hello world
• Backstop error appeared

13
Problem Code
// loop through and increment the values for each
char for (int i 0 i lt s.length() i)
// get the character at index i char c
s.charAt(i) // increment the value in the
array of occurrences // figure out its index
by subtracting 'a' from it occur(int)(c -
'a')
14

ERROR

An error occurred at
line 30 of StringCount.java in the method "main".
I'm sorry, the program cannot continue. The line
of code that caused the error is
occur(int)(c - 'a') It seems that the code
tried to use an illegal value as an index in an
array. The code was trying to access an element
at index -65 of the array called "occur". The
expression "(int)(c - 'a')" had the value -65
when the error occurred. Remember, you cannot
have a negative index. Be sure that the index is
always positive. This error is called an
ArrayIndexOutOfBoundsException. The full error is
as follows java.lang.ArrayIndexOutOfBoundsExcepti
on -65 at StringCount.main(StringCount.java3
0)
15
Questions

• Was it more useful than the stacktrace?
• All said yes
• Did the output mislead you?
• Two said yes one admitted he didnt read it
• Describe the tone (vs. stack trace)
• Six said it was friendly
• Four said too long
• Two said too pedantic for a general audience
• Three did not notice it

16
Observations

• Most common complaint was that the output is too
  long
• One possibility is giving the option to see more
  details or user-adjusted verbosity levels
• Average time to complete the task with Backstop
  was around 5.8 minutes
• Understanding the cause took 1-2 minutes

17
General Debugging Support

• Display each line of code as it gets executed
• File name
• Line number
• Source code
• Also display the values of any variables

18
Original source code
sum x y y x
Line number printed before code is executed, in
case of exception
Modified source code
System.out.print("\nFib.java27 sum x
y") System.out.print(" (x was "
x)System.out.print(" , y was " y
)System.out.print(")") sum x
y System.out.print(" (sum is now " sum
")") System.out.print("\nFib.java29 y
x")System.out.print(" (x was "
x)System.out.print(")")y x System.out.print("
(y is now " y ")")
Everything on one line to retain original line
number
Verb tense indicates change of variable values
Output at runtime
Fib.java27 sum x y (x was 2 , y was 1)
(sum is now 3) Fib.java29 y x (x was 2) (y
is now 2)
19
StringCount.java27 char c s.charAt(i) (i was
0) (c is now h) StringCount.java30
occur(int)(c - 'a') (occur(int)(c - 'a')
is now 1) StringCount.java27 char c
s.charAt(i) (i was 1) (c is now
e) StringCount.java30 occur(int)(c - 'a')
(occur(int)(c - 'a') is now 1) StringCount.java
27 char c s.charAt(i) (i was 2) (c is now
l) StringCount.java30 occur(int)(c - 'a')
(occur(int)(c - 'a') is now 1) StringCount.java
27 char c s.charAt(i) (i was 3) (c is now
l) StringCount.java30 occur(int)(c - 'a')
(occur(int)(c - 'a') is now 2) StringCount.java
27 char c s.charAt(i) (i was 4) (c is now
o) StringCount.java30 occur(int)(c - 'a')
(occur(int)(c - 'a') is now 1) StringCount.java
27 char c s.charAt(i) (i was 5) (c is now
!) StringCount.java30 occur(int)(c -
'a')

ERROR

An error occurred
at line 30 of StringCount.java in the method
"main". I'm sorry, the program cannot
continue. The line of code that caused the error
is occur(int)(c - 'a') It seems that
the code tried to use an illegal value as an
index to an array. The code was trying to access
an element at index -64 of the array called
"occur". The variable "(int)(c - 'a')" had the
value -64 when the error occured. Remember, you
cannot have a negative index. Be sure that the
index is always positive. This error is called
an ArrayIndexOutOfBoundsException. The full
error is as follows java.lang.ArrayIndexOutOfBoun
dsException -64 at StringCount.main(StringCount.
java30) at sun.reflect.NativeMethodAccessorImpl.
invoke0(Native Method)
20
Questions

• Was it helpful?
• All but one said yes
• Did the output mislead you?
• Five said yes, but four said only at first
• Describe the appearance
• Five said more spacing would make it easier to
  read
• Four said more spacing would make it harder to
  read
• Three said the spacing would make no difference

21
Evaluation

• Backstop helped students identify and fix errors
  in code they did not write
• Backstop automates/mimics the way that many CS1
  students debug a Java program on the command line
• Still need a study involving the use of both
  parts of Backstop to debug a runtime error

22
Conclusion and Future Work

• Our contribution Backstop
• Makes the error messages produced by uncaught
  Java exceptions easier to understand (compared to
  the Java stacktrace)
• Provides friendlier and more useful information
  about how to fix the error
• Automates simple debugging approach
• Future work could include integration with an
  IDE, and further studies to measure Backstops
  effectiveness

23
Backstop A Tool for Debugging Runtime Errors

• Chris Murphy, Eunhee Kim, Gail Kaiser, Adam
  Cannon
• Columbia University
• http//www.cs.columbia.edu/cmurphy/backstop
• cmurphy_at_cs.columbia.edu