SEG 2100 Software Design II

1
Software Quality Testing and Verification II
2
Software Flaws are identified at three levels

• A failure is an unacceptable behaviour exhibited
  by a system
• The frequency of failures measures software
  reliability
• Low failure rate high reliability
• Failures result from violation of a requirement
• A defect is a flaw that contributes to a failure
• It might take several defects to cause one
  failure
• An error is a software decision that leads to a
  defect

3
Eliminating Failures Testing vs Verification

• Testing running the program with a set of
  inputs to gain
• confidence that the software has few defects
• Goal reduce the frequency of failures
• When done after the programming is complete
• Methodology develop test cases run the program
  with
• each test case
• Verification formally proving that the software
  has no defects
• (in this case, the program is said to be
  correct)
• Goal eliminate failures
• When done before, during and after the
  programming is complete
• Methodology write separate specifications for
  the
• code prove that the code and the
  specifications
• are mathematically equivalent

4
Program Correctness

• The correctness of a program is based on a
  specific standard.
• That standard is called a specification.
• int max (int a, int b)
• int m
• if (a gt b)
• m a
• else
• m b
• return m
• E.g., a specification for the above program might
  be that it finds the maximum value of any two
  integers.

5
Formalizing a Specification

• A Formal specification is written as a logical
  expression called an assertion.
• An assertion describes the state of the programs
  variables.
• Two key assertions are the programs precondition
  and its postcondition.
• int max (int a, int b)
• int m
• if (a gt b)
• m a
• else
• m b
• return m
• A domain is a set of values over which a variable
  is well defined.
• The primitive types (int, float, boolean, etc.)
  and standard Java classes (String, Vector,
  HashMap, etc.) provide domains for reasoning
  about programs.

Precondition P whats required for the program
to begin its run.
Postcondition Q whats ensured to be true when
the program finishes
6
Pre- and Postconditions

• Postconditions describes what it will compute.
• For max, a postcondition is Q m max(a, b),
  where max is a mathematical function that
  delivers the larger of two integers.
• Preconditions describe minimum requirements for
  the program to run.
• For max, a and b can be any integers, so the
  precondition is P true.
• Before proving a programs correctness, we first
  write its specifications
• true
• int max (int a, int b)
• int m
• if (a gt b)
• m a
• else
• m b
• return m
• m max(a, b)

7
Another Example computing a factorial (n!)

• n gt 1
• int Factorial (int n)
• int f 1
• int i 1
• while (i lt n)
• i i 1
• f f i
• return f
• f n!
• This example raises two issues. What happens if
• 1. the program has a loop that never terminates?
• 2. The program terminates abnormally (e.g., an
  exception is raised)?

Precondition P
Postcondition Q
8
Defining away the problem partial correctness

• Lets assume that nothing bad ever happens
• programs always terminate after a finite number
  of steps, and
• termination is always normal.
• Then such a program is partially correct if, for
  every set of input values that satisfies
  precondition P, the program computes a result
  that satisfies postcondition Q.
• E.g., Factorial is partially correct if for every
  value of n that satisfies n gt 1, it computes f n!

9
(Partial) Correctness Proofs

• Lets think generally about any program or
  sequence of statements s, whose pre- and
  postconditions are P and Q.
• A Hoare triple is a predicate of the form
• P s Q
• which asserts that execution of statements s,
  beginning in a state that satisfies P, results in
  a state that satisfies Q.
• If we can prove that this Hoare triple is valid,
  (i.e., it is true for all assignments of values
  to variables in P, Q, and s) then the program s
  is said to be correct.
• But how can we construct such a proof?

10
Constructing a Correctness Proof

• If the program is a series of statements
• s1 s2 sn
• We start with the Hoare triple P s1 s2 sn
  Q, and
• use inference rules for programs to
• derive an intermediate triple Pi si Pi1 for
  every statement si in this sequence.
• When done, we also ensure that
• Note This process is similar to a direct proof
  in logic, where we
• use inference rules to
• derive a series of assertions that logically link
  the premises to the conclusion.
• So, what are the inference rules for programs?

11
Inference Rules are tied to program statement
types
12
Formal methods and real programs

• Major question Where do formal methods fit in
  the software process? That is,
• How do we integrate them into object oriented
  programs?
• How do we write pre- and postconditions P and Q
  for
• methods?
• classes?
• systems?
• Once written, how are these used to ensure that
  software is correct?

13
Recall the spiral model for software design
Deployment (v 1.0)
Review
v 1.1
v 1.0
Testing and Verification
Prototype
Requirements
Coding/ Integration
Specifications (use cases)
Design
14
Formal methods in the object oriented design
process

• In OO design, we focus on classes and objects
• Methods and messages are subordinate
• Collections of objects have state, which is the
  set of all active objects and the values of their
  variables at any moment of run time.
• Formal specifications P and Q are therefore
  logical expressions about each objects state.
• Tools for the formal design process
• Specifications Java Modeling Language (JML)
• Design Unified Modeling Language (UML and JML)
• Coding Java and JML
• Verification JML

15
Correctness of OO systems

• Where?
• Method level pre- and post-conditions, loop
  invariants
• Class level class invariant (class state)
• System level intra-class invariants (system
  state)
• When?
• Specification and design phases
• Write specifications for all classes and methods
  (UML/JML)
• Coding phase
• Develop code from the specifications
  (UML/JML/Java)
• Validation phase
• Prove (mathematically) that specifications and
  code are logically equivalent (JML ltgt Java)

16
What is JML? (www.jmlspecs.org)

• History
• Emerged in early 2000s out of ESC/Java2
• Goals
• Integration of formal methods throughout the
  software process
• Formal specification accessible to programmers
• Direct support for design by contract
• Integration with a real language (Java)
• JML allows us to mix specifications directly with
  the Java code
• Preconditions
• Postconditions
• Loop invariants
• Class invariants

17
JML Basics

• JML specifications are special comments in a Java
  program
• //_at_ for one-liners
• /_at_ . _at_/ for
  multiple-liners
• The Hoare triple
• P s1 s2 sn Q
• is written in JML/Java as
• (P and Q are written as Java
• boolean expressions, and use
• parameters, local, and class
• variables as arguments.)

/_at_ requires P ensures Q _at_/ type
method (parameters) local variables
s1 s2 sn
18
JML Language Summary
Note p and e are also good old fashioned Java
boolean expressions, possibly augmented by
JML-specific operations.
19
Heres a simple example, first as a Hoare triple

• n gt 1 P (precondition)
• int Factorial (int n)
• int f 1
• int i 1
• 1 lt i ? i lt n ? f i! R (loop
  invariant)
• while (i lt n)
• i i 1
• f f i
• return f
• f n! Q (postcondition)

20
And again with JML expressions for P, Q, and R
P
Q
R

• /_at_ requires 1 lt n
• ensures \result (\product int i 1lti
  iltn i)
• _at_/
• static int Factorial (int n)
• int f 1
• int i 1
• /_at_ loop_invariant i lt n f (\product int
  j 1 lt j j lt i j)
• _at_/
• while (i lt n)
• i i 1
• f f i
• return f

21
JML-based software tools (1 and 3 are built into
Eclipse)

• 1. Compiling (use jmlc instead of javac)
• Does syntactic and type checking, and byte code
  generation for all JML assertions and Java code
• 2. Static checking (ESC/Java2)
• 3. Runtime assertion checking (use jmlrac
  instead of java)
• Checks truth of precondition P at entry to every
  call
• Checks truth of postcondition Q at exit from
  every call
• Checks truth of loop invariant R before every
  iteration
• Issues a Java Exception when any of these is not
  true
• Note this is not formal verification
• (Checking truth for one instance of a call is
  not the same as checking truth for all instances.
  The latter is proof of correctness!)
• 4. Proof assistance tools (Daikon, LOOP)

22
JML Eclipse Environment
1. Compiling
3. Runtime assertion checking
23
Continuing our JML example, lets wrap Factorial
inside a simple class

• public class myFactorial
• /_at_ requires 1 lt n
• ensures \result (\product int i 1lti
  iltn i)
• _at_/
• static int Factorial (int n)
• public static void main(String args)
• int n Integer.parseInt(args0)
• System.out.println("Factorial of " n " "
  Factorial(n))

24
compile it with jmlc, and run it twice with
jmlrac

• jmlc -Q myFactorial.java
• jmlrac myFactorial 3
• Factorial of 3 6
• jmlrac myFactorial -5
• Exception in thread "main
• org.jmlspecs.jmlrac.runtime.JMLEntryPreconditio
  nError
• by method myFactorial.Factorial regarding
  specifications at
• File "myFactorial.java", line 3,
  character 15 when
• 'n' is -5
• at myFactorial.checkPreFactorialmyFac
  torial(myFactorial.java240)
• at myFactorial.Factorial(myFactorial.ja
  va382)
• at myFactorial.main(myFactorial.java24
  )

normal run
abnormal run (throws a JML exception)
25
JML Exceptions
26
JML helps identify errors

• Example 1 Suppose we change the while loop from
  
• while (i lt n)
• to while (i lt n)
• so that n! will be computed incorrectly.
  
• Heres the result
• jmlrac myFactorial 3
• Exception in thread "main"
• org.jmlspecs.jmlrac.runtime.JMLLoopInvariantErr
  or LOOP INVARIANT
• by method myFactorial.Factorial regarding
  specifications at
• File "myFactorial.java", line 9, character
  24 when
• 'n' is 3
• at myFactorial.internalFactorial(myFactor
  ial.java102)
• at myFactorial.Factorial(myFactorial.java
  575)
• at myFactorial.main(myFactorial.java211)

invariant not satisfied
27
JML Example 2

• Suppose we change the while loop from
• while (i lt n) to
• while (i ltn )
• and also remove the JML loop invariant. Now we
  get
• jmlrac myFactorial 3
• Exception in thread "main"
• org.jmlspecs.jmlrac.runtime.JMLNormalPostcondit
  ionError
• by method myFactorial.Factorial regarding
  specifications at
• File "myFactorial.java", line 4, character 23
  when
• 'n' is 3
• '\result' is 24
• at myFactorial.checkPostFactorialmyFactorial(
  myFactorial.java321)
• at myFactorial.Factorial(myFactorial.java392)
• at myFactorial.main(myFactorial.java24)

postcondition not satisfied
28
JML Example 3

• Disagreement between a JML specification and a
  program may signal an error in the specification.
  
• E.g., if the loop invariant had specified j lt i
  rather than j lt i the following outcome would
  occur
• jmlrac myFactorial 3
• Exception in thread "main"
• org.jmlspecs.jmlrac.runtime.JMLLoopInvariantErr
  or LOOP INVARIANT
• by method myFactorial.Factorial regarding
  specifications at
• File "myFactorial.java", line 9, character
  24 when
• 'n' is 3
• at myFactorial.internalFactorial(myFactor
  ial.java101)
• at myFactorial.Factorial(myFactorial.java
  573)
• at myFactorial.main(myFactorial.java209)

invariant not satisfied
29
But beware JML is no panacea

• jmlrac doesnt trap all errors here are two
  normal runs
• jmlrac myFactorial 21
• Factorial of 21 -1195114496
• jmlrac myFactorial 32
• Factorial of 32 -2147483648
• Recall (1) Java has no ArithmeticOverflow
  exception, but
• (2) Factorial(n) for n gt 12 should give
  a result gt 231-1
• Note jmlrac computes the same wrong result when
  it checks the postcondition as the Factorial
  method computes, so that this error goes
  undetected.
• Conclusion the program and its specifications
  are both wrong.

wrong results!
30
Exception Handling in JML
We can throw Java Exceptions, and then validate
their circumstances in JML whenever they occur.
/_at_ requires P ensures Q
signals (exception) expression _at_/ type method
(parameters) locals s1 s2 sn
(includes throw new exception )
JML executes this
whenever this happens
Two outcomes 1) expression is true and normal
Java exception handling proceeds, or 2)
expression is false and JMLExceptionalPostconditi
onError is reported.
31
JML Example 4 Throwing and Checking Exceptions

• /_at_ requires 1 lt n
• ensures \result (\product int i 1lti
  iltn i)
• signals (ArithmeticException) n gt 12
• _at_/
• static int Factorial (int n)
• if (n gt 12) throw new ArithmeticException(
  )
• else
• jmlrac myFactorial 22
• Exception in thread "main" java.lang.ArithmeticExc
  eption
• at myFactorial.internalFactorial(myFactor
  ial.java9)
• at myFactorial.Factorial(myFactorial.java
  610)
• at myFactorial.main(myFactorial.java213)

Normal Java exception handling occurs, since n gt
12 is true.
32
Additional Points about JML

• 1. We can sometimes avoid a signals clause by
  strengthening the precondition. E.g., for
  Factorial, we could have said
• requires 1 lt n n lt 13
• 2. Specifications are always declarative they
  never affect the state of the program.
• 3. Runtime assertion checking is not proof, but
  it does provide a rigorous framework for
  debugging.
• 4. JML provides a language in which formal
  methods and Java programs can be integrated.
• 5. Theres a lot more to JML. We are
  particularly interested in
• a. class level specifications?
• b. Tools for static checking of specifications?
• c. Tools for proving correctness?

33
Readings (at the course web site)

• Hoare, An axiomatic basis for computer
  programming, Communications of the ACM
  12(10)576-580. ?
• Leino, Hoare-Style Program Verification I?
• Leino, Hoare-Style Program Verification II?
• Gordon, Specification and Verification I ?
• Dwyer et al., Software Specifications
• Leavens and Cheon, Design by Contract with JML?
• Poll, Kiniry, and Cok, Introduction to JML?
• Burdy et al., Overview of JML Tools?
• JML Reference Manual