Formal Methods

1
Formal Methods Quick Tutorial
Ricky W. Butler Oct 22, 2003
http//shemesh.larc.nasa.gov/fm
2
Outline

• Motivation
  (3 minutes)
• Formal Methods by way of example
• Theorem Proving (12
  minutes)
• Model Checking state machines (12 minutes)
• A Few More Thoughts (3
  minutes)

3
There are no stupid questions
4
Motivation
Say, I think I see where we went off. Isnt
eight times seven fifty-six?
5
Software Underpins Everything In Aviation

• Increased Capabilities
• Almost any breakthrough in any field will depend
  on advanced computer systems
• And software is where most of the complexity goes
• Software Costs Dominate
• Development costs (More than half of the
  non-recurring development
• of the
  Boeing 777)
• Integration (only way of testing it
  is all-up simulation)
• Certification (based on process, not
  product)
• Major Safety Concerns
• Reliability (dealing
  with hardware failures)
• Correctness (no faults in
  its own design)
• Man-machine interaction (issue in majority of
  recent crashes)
• Certification (complexity
  forces FAA to rely on DERs)

6
Example System
f(x,B)
x
g(x)
System
B
h(B)
x VAR nat B VAR bool f(x,B) nat
IF x gt 30 AND B THEN xx ELSE 90x ENDIF
g(x) nat IF x lt 30 THEN 2x ELSE 0 ENDIF
h(B) nat IF B THEN 60 ELSE 0 ENDIF
safe_prop THEOREM f(x,B) gt g(x) h(B)
7
Design Verification
In the beginning, there was Simulation

• Build a model
• Feed IT inputs
• Analyze results

8
Verification by Simulation
x VAR nat B VAR bool f(x,B) nat
IF x gt 30 AND B THEN xx ELSE 90x ENDIF
g(x) nat IF x lt 30 THEN 2x ELSE 0 ENDIF
h(B) nat IF B THEN 60 ELSE 0 ENDIF
safe_prop THEOREM f(x,B) gt g(x) h(B)
x B f g
h safe_prop -----------
---------------------------- -------- 0
T 0 0 60
TRUE 10 F 1200 20 0
TRUE 50 T 2500 0
60 TRUE 30 F 3600
60 0 TRUE 30 T
2700 60 60 FALSE
9
Formal Verification
x VAR nat B VAR bool f(x,B) nat
IF x gt 30 AND B THEN xx ELSE 90x ENDIF g(x)
nat IF x lt 30 THEN 2x ELSE 0 ENDIF h(B)
nat IF B THEN 60 ELSE 0 ENDIF
safe_prop THEOREM f(x,B) gt g(x) h(B)
IF x gt 30 AND B THEN xx ELSE 90x ENDIF gt g(x)
h(B)
B
NOT B
IF x gt 30 THEN xx ELSE 90x ENDIF gt g(x) h(B)
90x gt g(x) h(B)
90x gt g(x) 0
90x gt 0
10
Formal Verification (cont.)
x VAR nat B VAR bool f(x,B) nat
IF x gt 30 AND B THEN xx ELSE 90x ENDIF g(x)
nat IF x lt 30 THEN 2x ELSE 0 ENDIF h(B)
nat IF B THEN 60 ELSE 0 ENDIF
B
IF x gt 30 THEN xx ELSE 90x ENDIF gt g(x) h(B)
x gt 30
x lt 30
90x gt g(x) h(B)
xx gt g(x) h(B)
90x gt 2x h(B)
xx gt 0 h(B)
90x gt 2x 60
xx gt 0
90x gt 120x
11
Using a Mechanical Proof Checker (PVS)
PVS SPECIFICATION
OUTPUT FROM PVS THEOREM PROVER
safe_prop ------- 1 FORALL (B bool,
x nat) f(x, B) gt g(x) h(B) Rule? (grind) f
rewrites f(x, B) to IF x gt 30 AND B THEN x x
ELSE 90 x ENDIF g rewrites g(x) to IF x lt 30
THEN 2 x ELSE 0 ENDIF h rewrites h(B) to IF B
THEN 60 ELSE 0 ENDIF Trying repeated
skolemization, instantiation, and
if-lifting, this simplifies to safe_prop
-1 x!1 gt 0 -2 B!1 ------- 1 x!1 gt
30 2 90 x!1 gt 120 x!1 Rule?
demo_incorrect THEORY BEGIN x VAR nat B
VAR bool f(x,B) nat IF x gt 30 AND B THEN
xx ELSE 90x ENDIF
g(x) nat IF x lt 30 THEN 2x ELSE 0 ENDIF
h(B) nat IF B THEN 60 ELSE 0 ENDIF
safe_prop THEOREM f(x,B) gt g(x) h(B) END
demo_incorrect
12
Using a Mechanical Proof Checker (PVS)
CORRECTED PVS SPECIFICATION
OUTPUT FROM PVS THEOREM PROVER
safe_prop ------- 1 FORALL (B bool,
x nat) f(x, B) gt g(x) h(B) Rule? (grind) f
rewrites f(x, B) to IF x gt 30 AND B THEN x x
ELSE 90 x ENDIF g rewrites g(x) to IF x lt 30
THEN 2 x ELSE 0 ENDIF h rewrites h(B) to IF B
THEN 40 ELSE 0 ENDIF Trying repeated
skolemization, instantiation, and
if-lifting, Q.E.D. Run time 0.35 secs. Real
time 3.33 secs.
demo_correct THEORY BEGIN x VAR nat B
VAR bool f(x,B) nat IF x gt 30 AND B THEN
xx ELSE 90x ENDIF
g(x) nat IF x lt 30 THEN 2x ELSE 0 ENDIF
h(B) nat IF B THEN 40 ELSE 0 ENDIF
safe_prop LEMMA f(x,B) gt g(x) h(B) END
demo_correct
13
The Key Difference

• Simulation/Testing only explores a small part of
  the state space
• Formal Verification explores the entire state
  space

The only way you can assure yourself that there
are no safety-relevant bugs hidden in some dark
corner of your software is to explore the entire
state space.
14
Can We Automate Verification?
Verified!
Program
The Big V
or
Safety Props
Buggy!
Counter Example
15
Kurt Goedel (1906-1978)
Answer NO!
16
Why Cant We Automate Verification?

• Algebra (in general) is undecidable
• (i.e. There is no terminating algorithm that can
  determine whether a formula is true)
• But some sub-theories are decidable (e.g.
  Presburger arithmetic) develop automated
  decision procedures

• While loops in a program lead to induction proofs
• (Discovery of the induction invariant involves
  creativity ?n P(n) requires P(n) ? Q1(n) ? ?
  Qj(n))

• Modeling the environment that the program
  interacts with often involves continuous
  mathematics (e.g. calculus, trig)
  hybrid models

• Finite-state models can be automatically analyzed
  via model-checking (execution times can be
  exorbitant)

17
Proof Checkers
But we do use Proof Checkers! (aka Theorem
Provers) -- because we make too many
mistakes Why then do you call it a theorem
prover if it can only check?
Because it does help, it just cant do the whole
job.
18
State Machine Analysis
State (x nat, y nat) Events ex, ey, em ,
es
19
State Machine Analysis
(Proving Invariants)
ex
ex
ex
0, 0
10,0
20,0
30,0
es
ey
ey
ey
ey
ex
ex
ex
10,10
20,10
0,10
30,10
em
em
es
0,20
PROVE For all reachable states (x,y) from
(0,0) x ? A and
y ? B
20
State Machine Analysis
(Proving Invariants)

• Theorem proving
• deduction
• human directed
• Model checking
• exhaustive search
• automatic
• state space must be finite

21
State Machine Analysis
(Proving Invariants)
ex
ex
ex
0, 0
10,0
20,0
30,0
es
ey
ey
ey
ey
ex
ex
ex
10,10
20,10
0,10
30,10
em
em
es
0,20
PROVE for all reachable states (x,y) from
(0,0) x ? A and
y ? B
Counter-example A 30, B 20
ex
ex
ex
es
(0,0) ----gt (10,0) ----gt (20,0) ----gt (30,0)
----gt (0,30)
22
State Machine Analysis
(Proving Invariants)
INVARIANT For all reachable states (x,y) from
(0,0) that x ?
max(A,B) and y ? max(A,B)
Proof Approach Assume invariant holds in
current state. Show each transition preserves
the invariant. By induction true for all
reachable states.
23
SMV Version of State Machine
MODULE main VAR x 0..50 y 0..50
event ex,ey,em ASSIGN init(x) 0
init(y) 0 next(x) case event ex x
lt AA - 10 x 10 event
ey y lt BB - 10 x event
em x gt y y gt 0 x - y
event em y gt x x gt 0 x
1 y
esac next(y) case event ex x lt AA - 10
y event ey y lt BB - 10
y 10 event em x gt y
y gt 0 y event em y gt x
x gt 0 y - x 1
x esac
24
SMV Execution Times
Execution Times (secs)
Range of State Space Variables
25
(No Transcript)
26
Output From SMV Model Checker
DEFINE AA 40 BB 50 invariant_1
AG (x lt AA y lt BB) invariant_2 AG (x lt
max_AB y lt max_AB) -- specification
invariant_1 is false -- as demonstrated by the
following execution sequence state 1.1 x 0 y
0 event em state 1.2 event ey state
1.3 y 10 state 1.4 y 20
state 1.5 y 30 state 1.6 y 40 state
1.7 y 50 event em state 1.8 x 50 y
0
-- specification invariant_2 is true
27
Different perspectives on the Joys of Theorem
Proving
Ms. Miller, tell the verification team I will be
a little late with that proof.
Just two more lemmas and the proof of the KB3D
CDR algorithm will be complete!
28
What Does It Look Like In a Theorem Prover?
x
Y
System
n
Z
x VAR posreal positive real number n
VAR posnat positive natural number Y
IF n gt 15 THEN x ELSE n x ENDIF
Z IF x ? 10 AND n ? 10 THEN 2 x / n
ELSE 10 / x ENDIF
safe_prop THEOREM Y Z lt 200
29
FM in development process
size
proofs
106 states
deeper, conceptual
Requirements
1010 states
High-level design
shallow, thousands of cases
10100 states
Low-level design
10100 100states (i.e. 10 million lines code)
Code
Executable
Programming language such as C have complex,
ill-defined semantics
30
Formal Methods Does Not Bring Perfection!
intended functionality

• peer review
• simulation
• testing

Requirements
proof
High Level Design
proof
Detailed Design
proof
Implementation
Are the component models accurate?

• peer review
• simulation
• testing

physical devices
31
Warning some of our team members are a little
sensitive
You guys are both my witnessesHe insinuated that
ZFC set theory is superior to type theory!
32
The Research Challenge

• Formal Methods are not a silver bullet
• Must be applied early in life cycle---not a band
  aid
• Involves a significant learning curve
• Each new application requires the development of
  new models and analysis techniques
• Cost of using formal methods still high for some
  applications.
• The software engineering community is dominated
  by the process management viewpoint (i.e., that
  careful management of software development
  process is the key to quality)
• Historically, disasters must occur before major
  changes are made.

33
EXTRA SLIDES
34
What Does It Look Like In a Theorem Prover?
sys_lem -1 Y IF n gt 15 THEN x ELSE n x
ENDIF -2 Z IF x lt 10 AND n lt 10 THEN 2 x
/ n ELSE 10 / x ENDIF ------- 1 Y Z lt
200 Rule? (case "n gt 15") Case splitting on
n gt 15, this yields 2 subgoals
35
What Does It Look Like In a Theorem Prover?
sys_lem.1 -1 n gt 15 -2 Y IF n gt 15 THEN
x ELSE n x ENDIF -3 Z IF x lt 10 AND n lt
10 THEN 2 x / n ELSE 10 / x ENDIF
------- 1 Y Z lt 200 Rule?
(assert) Simplifying, rewriting, and recording
with decision procedures, This completes the
proof of sys_lem.1
What Happened? Well, assuming n gt 15 we have
Y x Z 10 / x
36
What Does It Look Like In a Theorem Prover?
sys_lem.2 -1 Y IF n gt 15 THEN x ELSE n x
ENDIF -2 Z IF x lt 10 AND n lt 10 THEN 2 x
/ n ELSE 10 / x ENDIF -3 n lt 15
------- 1 Y Z lt 200 Rule?
(assert) Simplifying, rewriting, and recording
with decision procedures,
give us the other subgoal
37
What Does It Look Like In a Theorem Prover?
sys_lem.2 -1 Y n x -2 Z IF x lt
10 AND n lt 10 THEN 2 x / n ELSE 10 / x
ENDIF -3 n lt15 ------- 1 Y Z lt
200 Rule? (replace -1) (hide -1) sys_lem.2 -1
Z IF x lt 10 AND n lt 10 THEN 2 x / n ELSE
10 / x ENDIF -2 n lt15 ------- 1 nx Z
lt 200
38
What Does It Look Like In a Theorem Prover?
Rule? (case "x lt 10 AND n lt 10") Case
splitting on x lt 10 AND n lt 10, this
yields 2 subgoals sys_lem.2.1 -1 x lt
10 AND n lt 10 -2 Z IF x lt 10 AND n lt 10
THEN 2 x / n ELSE 10 / x ENDIF -3 n lt15
------- 1 n x Z lt 200 Rule? (replace
-1) Replacing using formula -1, this simplifies
to
39
What Does It Look Like In a Theorem Prover?
sys_lem.2.1 -1 Z 2 x / n -2 n lt
10 -3 x lt 10 ------- 1 n x Z lt
200 Rule? (replace -1) (hide -1) -1 n lt
10 -2 x lt 10 ------- 1 n x (2 x
/ n) lt 200 Rule? (assert)
hit it with the hammer
40
What Does It Look Like In a Theorem Prover?
sys_lem.2.1 -1 n lt 15 -2 x lt 10
------- 1 n x (2 x / n) lt 200 Rule?
(mult-ineq -2 -2) -1 x x lt 10 10 -2
n lt 15 -3 x lt 10 ------- 1 n x
(2 x / n) lt 200 Rule? (assert) Simplifying,
rewriting, and recording with decision
procedures, This completes the proof of
sys_lem.2.1.
But nothing happens!
hit it with the hammer again
41
What Does It Look Like In a Theorem Prover?
-1 Z IF x lt 10 AND n lt 10 THEN 2 x / n
ELSE 10 / x ENDIF -2 NOT (x lt 10 AND n lt
10) -3 n lt 15 ------- 1 n x Z lt
200 Rule? (replace -1) (hide -1) -1 x gt10 OR
n gt 10 -2 n lt 15 ------- 1 n x 10
/ x lt 200 Rule? (assert)
This completes the proof of sys_lem2.2. This
completes the proof of sys_lem.2. Q.E.D.
42
Different Domains --gt
Different Verification Challenges
x
c(x,y)
System
y
Need to formally reason about
c(x,y) y /Sin2(x) Cos2(x)
trigonometry

c(x,y)
series
calculus
c(x,y) d/dx (x3y y5)/x