Software Reliability Methods

1
Software Reliability Methods

• Sorin Lerner

2
Software reliability methods issues

• What are the issues?

3
Software reliability methods issues

• What is software reliability? How to measure it?
• Bug counts ? Will we ever have bug-free software?
• How many 9s ?
• Service Level Agreements ?
• What is a bug?
• Adherence to specifications
• But what is a specification
• User unhappy is that a bug?
• Different levels of severity

4
Software reliability methods issues

• Cost of the methods for achieving reliability
• Independently develop 5 versions of the software,
  run them all in parallel ) less likely that they
  fail at the same time in the same way. But cost
  is high
• For tools, cost of development of the tools
• Burden on the programmer
• fully automated vs. semi-automated methods
• allow progressive adoption

5
Software reliability methods issues

• Level of guarantee provided by the method
• Hard guarantees, statistical guarantees, no
  formal guarantee
• What if tool is broken trusted computing base
• When is the method used?
• compile-time, link-time, load-time, run-time
• What does the tool see?
• source code, assembly, the whole program or part
  of the program

6
One way of dividing the spectrum
Compiler
010010110100101011011
7
One way of dividing the spectrum
Static techniques
Testing techniques
Run-time techniques
Compiler
010010110100101011011
8
One way of dividing the spectrum
Static techniques
Testing techniques
Run-time techniques
Compiler
010010110100101011011
9
Static Techniques

• Spec
• ????
• ?
• ? \ r
• t ? l

Code satisfies spec?

• Spec says what code should and should not do
• Complete spec specifies all behaviors (hard to
  formalize)
• Incomplete spec only defines some behaviors
• e.g. no null derefs, requests received are
  eventually processed
• Many formalisms exist for specs (Pre/Post
  conditions, FSMs, Temporal Logic, Abstract State
  Machines etc.)

10
Static Techniques
CleanL TSys DSL
DFA WP/SP MC ATP

• Language Design
• Clean language design
• Type Systems
• Domain-specific languages

• Program Analysis
• Dataflow analysis
• WP/SP
• Model checking
• Automated Theorem Proving

Interaction between the two
11
ESC/Java Leino et al PLDI 2002
CleanL TSys DSL
DFA WP/SP MC ATP

• Programmer annotates code with pre- and
  post-conditions, tool verifies that these hold

Automated Theorem Prover
object Foo //_at_ PRE (FORMULA) method
bar(...) ... //_at_ POST
(FORMULA)
)
WP(POST, bar) weakest condition Q such that Q
at entry to bar establishes POST at exist
Compute Weakest Precondition
12
Rhodium Lerner et al POPL 2005
CleanL TSys DSL
DFA WP/SP MC ATP
Compiler
Parser
DSL Opt
DSL Opt
DSL Opt
DSL Opt
DSL Opt
DSL Opt
Code Gen
13
Rhodium Lerner et al POPL 2005
CleanL TSys DSL
DFA WP/SP MC ATP
Compiler
Parser
DSL Opt
DSL Opt
DSL Opt
DSL Opt
DSL Opt
DSL Opt
Code Gen
Checker
Checker
Checker
14
Rhodium Lerner et al POPL 2005
CleanL TSys DSL
DFA WP/SP MC ATP
Rdm Opt
Opt-independent
Checker
Opt-dependent
15
ESP Das et al PLDI 2002
CleanL TSys DSL
DFA WP/SP MC ATP

• Interface usage rules in documentation
• Order of operations, data access
• Resource management
• Incomplete, wordy, not checked

• Violated rules ) crashes
• Failed runtime checks
• Unreliable software

16
ESP Das et al PLDI 2002
CleanL TSys DSL
DFA WP/SP MC ATP
C Program
Rules
ESP
Safe
Not Safe
17
ESP Das et al PLDI 2002
CleanL TSys DSL
DFA WP/SP MC ATP

• ESP is a program analysis that keeps track of
  object state at each program point
• e.g. is file handle open or closed?
• Challenge scale to large programs
• One of scalability issues merge nodes
• Always analyze both sides of merge node )
  exponential (or non-terminating) program analyses
• ESP has a heuristic for handling merges that
• avoids exponential blow-up and runs fast in
  practice
• maintains enough precision to verify programs

18
BLAST Henzinger et al POPL 2000

• Interface usage rules in documentation
• Order of operations, data access
• Resource management
• Incomplete, wordy, not checked

• Violated rules ) crashes
• Failed runtime checks
• Unreliable software

19
BLAST Henzinger et al POPL 2000
C Program
Rules
BLAST
Safe
Error Trace
20
BLAST Henzinger et al POPL 2000
C Program
Rules
BLAST
Safe
Error Trace
21
BLAST Henzinger et al POPL 2000
BLAST
start with a set of predicates
augmented set of predicates
Perform Predicate Abstraction
Refine set of predicates
Rules
C Program
Trace infeasible
No errors found
Analyze trace
error trace found
Trace feasible
Error Trace
Safe
22
BLAST Henzinger et al POPL 2000
CleanL TSys DSL
DFA WP/SP MC ATP
BLAST
start with a set of predicates
augmented set of predicates
Perform Predicate Abstraction
Refine set of predicates
Rules
C Program
Trace infeasible
No errors found
Analyze trace
error trace found
Trace feasible
Error Trace
Safe
23
Type Systems
CleanL TSys DSL
DFA WP/SP MC ATP

• What is a type system?
• A discipline for writing code that can be
  mechanically checked, and can prevent certain
  kinds of run-time errors
• For example, java type system prevents calling
  methods that dont exists, or calling methods
  with parameters of the wrong type

24
Type Systems
CleanL TSys DSL
DFA WP/SP MC ATP

• Type systems can track and provide guarantees
  about many other aspects of computation
• Safe explicit memory management (Crary, Walker
  and Morrisett, POPL 99)
• Execution time bounds (Crary and Weirich, POPL
  00)
• Information flow (Myers, POPL 00)
• Security automata (Walker, POPL 00)

25
Type Systems
CleanL TSys DSL
DFA WP/SP MC ATP

• MultiJava Clifton et al 2000 adds to Java
• multi-methods methods that dispatch symetrically
  on the type of all params, not just the first
• open classes classes
• Adding these features makes modular type checking
  harder, and required innovations on the type
  system side
• Interplay between language design and type systems