Finding and Preventing RunTime Error Handling Mistakes

1
Finding and Preventing Run-Time Error Handling
Mistakes
(subliminal OOPSLA advertisement)

• Wes Weimer
• George Necula
• UC Berkeley

2
Finding and Preventing Run-Time Error Handling
Mistakes
(subliminal OOPSLA advertisement)

• Wes Weimer
• George Necula
• UC Berkeley

3
The Context

• Reliability in programs
• Think Java or C
• But not protocols or specifications
• Run-time errors
• OS resource exhaustion
• Network connectivity problems
• Database access errors
• Disk problems
• Error-handling mistakes
• Forgetting to release a lock when a disk error
  occurs

4
Glossy Summary

• It is difficult to write error-handling code
• We have an analysis for finding mistakes
• Dataflow analysis and fault model
• We found over 800 mistakes in 4 MLOC
• We characterize these mistakes
• What goes wrong in off-the-shelf code?
• Existing PL features are insufficient
• We propose a new feature
• compensation stacks track your obligations
• And back it up with case studies

5
Error-Handling is Significant

• IBM Study up to two-thirds of a program can be
  devoted to error handling Cri87
• Our Survey for Java, between 3 and 46
• Transitively reachable from catch, finally
• Older, bigger programs have more handling
• Mistakes in which applications don't properly
  handle error conditions that occur during normal
  operation are reported as one of the top ten
  causes of Java web app security risks

6
Unfortunately, it often does not work

• Most common exception handlers
• Do Nothing
• Print stack trace, abort program
• Higher level invariants should be restored,
  interface requirements should be respected
• Aside from handling the error, the code should
  clean up after itself
• Why does this not happen?
• It is difficult for programmers to consider all
  execution paths

7
Error Paths 1 Original Code

• ptII3.0.2/ptolemy/actor/gui/JNLPUtilities.java
• Special Thanks Christopher Hylands Brooks
  (verified)

// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input new
BufferedInputStream(sourceURL.openStream())
BufferedOutputStream output new
BufferedOutputStream( new
FileOutputStream(destinationFile)) //
The resource pointed to might be a pdf file,
which // is binary, so we are careful to
read it byte by // byte and not do any
conversions of the bytes. int c
while (( c input.read()) ! -1)
output.write(c)
input.close() output.close() // line
294
8
Error Paths 1 Original Code

• ptII3.0.2/ptolemy/actor/gui/JNLPUtilities.java
• Special Thanks Christopher Hylands Brooks
  (verified)

// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input new
BufferedInputStream(sourceURL.openStream())
BufferedOutputStream output new
BufferedOutputStream( new
FileOutputStream(destinationFile)) //
The resource pointed to might be a pdf file,
which // is binary, so we are careful to
read it byte by // byte and not do any
conversions of the bytes. int c
while (( c input.read()) ! -1)
output.write(c)
input.close() output.close() // line
294
9
Error Paths 2 Try-Finally
// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input BufferedOutputStrea
m output try input new
BufferedInputStream(sourceURL.openStream())
output new BufferedOutputStream(
new FileOutputStream(destinationFile))
// The resource pointed to might be a pdf file,
which // is binary, so we are careful to
read it byte by // byte and not do any
conversions of the bytes. int c while
(( c input.read()) ! -1) output.write(c)
finally input.close()
output.close()
10
Error Paths 3 Try-Finally
// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input BufferedOutputStrea
m output try input new
BufferedInputStream(sourceURL.openStream())
output new BufferedOutputStream(
new FileOutputStream(destinationFile))
// The resource pointed to might be a pdf file,
which // is binary, so we are careful to
read it byte by // byte and not do any
conversions of the bytes. int c while
(( c input.read()) ! -1) output.write(c)
finally input.close()
output.close()
XXX
11
Error Paths 3 Try-Finally
// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input BufferedOutputStrea
m output try input new
BufferedInputStream(sourceURL.openStream())
output new BufferedOutputStream(
new FileOutputStream(destinationFile))
// The resource pointed to might be a pdf file,
which // is binary, so we are careful to
read it byte by // byte and not do any
conversions of the bytes. int c while
(( c input.read()) ! -1) output.write(c)
finally input.close()
output.close()
XXX
12
Error Paths 4 Nested Try-Finally
// Copy sourceURL to destinationFile without
doing any byte conversion. private static void
_binaryCopyURLToFile(URL sourceURL, File
destinationFile) throws IOException
BufferedInputStream input BufferedOutputStrea
m output input new BufferedInputStream(sourc
eURL.openStream()) try output new
BufferedOutputStream( new
FileOutputStream(destinationFile)) try
// The resource pointed to might be a pdf
file, which // is binary, so we are careful
to read it byte by // byte and not do any
conversions of the bytes. int c
while (( c input.read()) ! -1)
output.write(c) finally
output.close() finally
input.close()
13
Where are we?

• Summary and Context
• Error Handling is Hard Many Paths
• Finding Error-Handling Mistakes
• Characterizing Mistakes
• Old New PL Features
• Case Studies
• Grand Finale

14
Defining Error-Handling Mistakes

• Want to show that programs make mistakes when
  run-time errors occur
• What does it mean to make a mistake?
• Safety Policy
• If a socket is opened, it must be closed before
  the program terminates
• If a resource is acquired, it must be released
• We consider four generic resources
• Sockets, files, streams, database locks
• From a survey of catch, finally, finalize
• Program should release them along all paths, even
  those with run-time errors

15
Fault Model

• exceptions and run-time errors are highly
  correlated in Java CDCF03
• Ex Server crashes during call -gt
  UnmarshalException
• Methods come with signatures listing exceptions
  they can throw
• Fault Model
• A called method can terminate normally or raise
  any of its declared exceptions
• A called method for which we have no signature
  can terminate normally or raise any exception for
  which there is an enclosing catch block or that
  is declared to escape the caller

16
Analysis Summary

• Build Control-Flow Graph from source
• Special attention to exceptional control flow
• Methods can raise declared exceptions
• Symbolically execute each method
• Track outstanding resources along paths
• Abstract away data values
• Safety Policy determines when a resource has been
  acquired and when it has been released
• At joins, merge paths with equal resources
• If there is a path to the end of a method that
  has an outstanding resource, note it

17
Analysis Example Buggy Program

• try
• Socket s new Socket()
• s.send(GET index.html)
• s.close()
• finally
• // close should be in finally

18
Analysis Example
start
new socket
send
close
end
19
Analysis Example
start

new socket
send
close
end
20
Analysis Example
start

new socket
socket
send
close
end
21
Analysis Example
start


new socket
socket
send
close
end
22
Analysis Example
start


new socket
socket
send
socket
close
end
23
Analysis Example
start


new socket
socket
send
socket
socket
close
end
24
Analysis Example
start


new socket
socket
send
socket
socket
close

end
25
Analysis Example
start


new socket
socket
send
socket
socket
close


end
26
Analysis Example Path with Mistake
Works for arbitrary type state FSMs as well.
These examples show simple two-state policies.
start


new socket
socket
send
socket
socket
close


end
27
Report Filtering

• Sources of False Positives
• if (sock ! null) sock.close()
• sock_field_we_close_later sock
• return sock
• Filter Error Reports
• Whenever an error path contains one of the above
  forms, we remove a resource of the appropriate
  type
• Removes all false positives in practice
• Introduces false negatives
• (3 in a random sample of 50 eliminated reports,
  suggests 30 false negatives total)

28
Analysis Results
29
Where are we?

• Summary and Context
• Error Handling is Hard Many Paths
• Finding Error-Handling Mistakes
• Characterizing Mistakes
• Old New PL Features
• Case Studies
• Current Work Spec Mining
• Grand Finale

30
Characterizing Mistakes

• Common Protect Some Areas, But Not All
• stafs STAXMonitor class

ObjectInputStream ois null try ois new
ObjectInputStream(/ ... /) // ... catch
(StreamCorruptedException ex) if (ois !
null) ois.close() showErrorDialog(/ ...
/) return false Object obj
ois.readObject() // no try ois.close()
// no finally
31
Characterizing Mistakes (2)

• Model aX() must be followed by cX()
• try-finally not nested (osage)
• try a1() a2() finally c2() c1()
• forget a resource (quartz)
• try a1() a2() finally c1()
• unprotected loops (ohioedge)
• for () a1() work() c1()
• various flags and early release

try a1() f0 if () f1 c1()
// close early work() a2()
finally c2() if (!f) c1() // or close late
32
Characterizing Mistakes (3)

• Complicated flag-work is common
• This example adapted from Browns undo

int f 0 // flag tracks progress try a1()
f 1 work() a2() f 2 work() a3() f
3 work() finally switch (f) // note
fall-through! case 3 try c3() catch
(Exception e) case 2 try c2() catch
(Exception e) case 1 try c1() catch
(Exception e)
33
Characterizing Mistakes (4)

• This approach does not scale well
• Work control flow is duplicated in cleanup code

int f 0 // flag tracks progress try
a1() f 1 work() if () did_a2 true
a2() f 2 work() a3() f 3 work()
finally switch (f) // note fall-through!
case 3 try c3() catch (Exception e)
case 2 if (did_a2) try c2() catch
case 1 try c1() catch (Exception e)

34
Destructors / Finalizers

• Great for stack-allocated objects
• Error-handling contains arbitrary code
• Example adapted from undo, which has 17 unique
  cleanup actions, one 34 lines long

• Called by garbage collector
• Too late! Programmers often use flags
  specifically to free resources early
• No ordering guarantees
• Socket sock new Socket()
• Stream strm new Stream(sock)
• If sock and strm are collected in the same sweep,
  sock.finalize may be called before strm.finalize

35
We Propose compensation stacks

• Store cleanup code in run-time stacks
• First-class objects, can pass them around
• When action succeeds, push cleanup
• action and cleanup can be arbitrary code
• Pop all cleanup code and run it (LIFO)
• when the stack goes out of scope
• or at end-of-scope if there is an exception
• and/or early (when programmer specifies)
• and when the stack is finalized
• Trace of events (loosely)
• a1 a2 a3 a4 a5
• a1 a2 aX cX c2 c1

36
Compensation Stacks

• Generalized destructors
• No made-up objects needed for local cleanup
• Can have multiple stacks
• e.g., one for each request in a webserver
• Annotate important interfaces to take
  compensation stacks
• Cannot make a new socket without putting
  this.close() on a stack of things-to-do
• Everything on a stack will be run on all paths
• Implicit current scope stack can optionally be
  assumed by default

37
Where are we? Were almost done!

• Summary and Context
• Error Handling is Hard Many Paths
• Finding Error-Handling Mistakes
• Characterizing Mistakes
• Old New PL Features
• Case Studies
• Grand Finale

38
Implementation, Case Studies

• Extend Java with such compensation stacks
• Annotate key interfaces (socket, stream, DB)
• Annotate existing programs to use compensation
  stacks
• Both for library resources (easy)
• And for unique cleanup actions
• Add no new error handling
• Ensure that existing handlers are run on all
  paths
• Run programs on third-party workloads
• Mark Brody, George Candea, Tom Martell,

39
Case Study 1 Browns undo

• IMAP/SMTP Proxy (operator time travel)
• Built for reliability
• 35,412 lines of Java, 128 annotation sites
• Contains many unique cleanup actions
• 8- to 34-line code fragments
• As well as many standard close actions
• And up to 5 simultaneous resources in sequence
• Results
• 225 lines shorter (1)
• No performance changes
• Stack overhead dwarfed by I/O overhead

40
Case Study 2 Suns petstore

• Amazon.com lite e-commerce, inventory
• Raises 150 exceptions over 3,900 requests
• Avg Response Time 52.06ms (std dev 100ms)
• 34,608 lines of Java, 123 annotation sites
• Two hours of work
• Standard close cleanup actions
• 3 simultaneous resources database handles
• Results
• 168 lines shorter (0.5)
• 0 such exceptions over 3,900 requests
• Avg Response Time 43.44ms (std dev 77ms)

41
Conclusion

• It is difficult to write error-handling code
• We have an analysis for finding mistakes
• Dataflow analysis and fault model
• We found over 800 mistakes in 4 MLOC
• We characterize these mistakes
• Programmers forget some paths
• Existing PL features are an awkward fit
• We propose a new feature
• compensation stacks track your obligations
• And back it up with case studies

42
Any Questions?
43
False Negatives

• hibernate2 - if, return, if, if, FN1
• jatlite - if, if, if, if, field
• quartz - FN2, if, if, if, FN3
• ejbca - if, if, if, if, if
• hsqldb - if, if, field, if, field
• jboss - field, if, field, if, return
• mckoi-sql - fld, fld, fld, ret, ret
• osage - ret, if, if, ret, if
• portal - ret, ret, ret, ret, ret
• staf - field

44
False Negative 1 (ResultSet)
public Iterator iterate(Object values, Type
types, RowSelection selection, Map
namedParams, SessionImplementor session) throws
HibernateException, SQLException
PreparedStatement st
prepareQueryStatement( getSQLString(), values,
types, selection, false, session )
try
bindNamedParameters(st, namedParams, session)
setMaxRows(st, selection)
ResultSet rs
st.executeQuery()
advance(rs, selection, session)
return new IteratorImpl( rs, session,
getReturnTypes(), getScalarColumnNames() )
catch
(SQLException sqle)
JDBCExceptionReporter.logExceptions(sqle)
closePreparedStatement(st,
selection, session)
throw sqle
45
False Negative 2 (ObjectInputStream)
protected Object getObjectFromBlob(ResultSet rs,
String colName) throws ClassNotFoundExcept
ion, IOException, SQLException
Object obj null Blob blobLocator
rs.getBlob(colName) InputStream
binaryInput null try

if (null ! blobLocator
blobLocator.length() gt 0)
binaryInput blobLocator.getBinaryStream()
catch (Exception ignore)
if (null ! binaryInput)
ObjectInputStream in new ObjectInputStream(bi
naryInput) obj in.readObject()
in.close() return
obj
46
False Negative 3 (ObjectInputStream)
protected Object getObjectFromBlob(ResultSet rs,
String colName) throws ClassNotFoundExcept
ion, IOException, SQLException Object
obj null InputStream binaryInput
rs.getBinaryStream(colName) if
(binaryInput ! null) ObjectInputStream
in new ObjectInputStream(binaryInput)
obj in.readObject() in.close()
return obj
47
What Are Those Benchmarks?
48
Show A Nested Try-Finally
// cayenne-1.0b4/src//dba/oracle/OraclePkGenerato
r.java, line 254 protected List
getExistingSequences(DataNode node) throws
SQLException // check existing sequences
Connection con node.getDataSource().getConnectio
n() try Statement sel
con.createStatement() try ResultSet
rs sel.executeQuery("SELECT LOWER(SEQUENCE_NAME)
FROM ALL_SEQUENCES") try
List sequenceList new ArrayList()
while (rs.next()) sequenceList.add(rs.ge
tString(1)) return sequenceList
finally rs.close() finally
sel.close() finally
con.close()
Most nested try-finally cases are not this easy
to follow.
49
Show Us Another Bug
// com/ohioedge/j2ee/api/org/tool/ejb/LetterTempla
teEJB.java, line 154 private StringBuffer
getColumnStringBufferFromReader()
StringBuffer buf null java.io.Reader reader
null PreparedStatement prepStmt null
ResultSet rs null Connection cn null
try cn ConnectionFactory.getConnection("jd
bc/LetterTemplateDB") StringBuffer qry new
StringBuffer() qry.append(" SELECT
\"letterTemplate\" FROM LetterTemplate ")
qry.append(" WHERE \"letterTemplateID\"
"getLetterTemplateID()) prepStmt
cn.prepareStatement(qry.toString()) rs
prepStmt.executeQuery() while (rs.next())
reader rs.getCharacterStream(1) buf
readIntoStringBuffer(reader)
rs.close()
prepStmt.close() catch (Exception e)
log.error(this.getClass().getName()".getColumnStr
ingBufferFromReader()") e.printStackTrace()
finally try cn.close()
catch (Exception e1) log.error(this.getCla
ss().getName()".getColumnStringBufferFromReader()
") e1.printStackTrace()

return buf
Note error handler that prints a stack trace and
does nothing else ...
50
Show A False Positive
// org/axiondb/tools/Console.java, line 89 try
_conn DriverManager.getConnection(buf.toString(
)) _stmt _conn.createStatement() catch
(SQLException e) cleanUp() throw e //
public void cleanUp() try
_rset.close() catch (Exception e)
try _stmt.close() catch (Exception e)
try _conn.close() catch (Exception e)