Coverage Metrics in Formal Verification

1
Coverage Metricsin Formal Verification
Hana Chockler
Hebrew University
2
Plan of the Lecture

• Coverage in simulation-based verification
• Motivation
• Coverage metrics
• Other directions of research
• Summary

3
Motivation
Fix the model/specification
NO
YES
Should I really believe this?
Did I check what I meant to check? Did I check
everything that I meant to check?
4
Coverage in Simulation-Based Verification

• Coverage metrics are heuristic measures of
  comprehensiveness of a given test.
• Different coverage metrics
• Code-based.
• Circuit structure-based.
• Functionality-based.
• etc.
• High coverage indicates that fewer bugs remained.

In model-checking we may visit all states. Does
it mean that coverage is always 100 ?
Measures the percentage of code executed by the
test.
Measures the percentage of gates visited during
the test.
Measures the percentage of functionality checked
by the test.
5
Motivating Example
System
Specification
The system satisfies the specification
This problem can be discovered by studying the
effect of local mutations of the system on the
satisfaction of the specification.
every request is eventually granted
In temporal logic
What is wrong with the system?
6
Motivating Example (cont.)
System
This problem can be discovered by studying the
effect of local mutations of the system on the
simulation relation with the design.
Specification
The system is simulated by the design
every request is eventually granted
Simulation of the system by the specification
As a high-level design
7
Previous Work

• Y.Hoskote, T.Kam, P.-H.Ho, X.Zhao.
• Coverage estimation for symbolic model
  checking.
• For subset of ACTL.
• Observability transformation applied to the
  specification, implying syntax-dependent
  algorithm.
• Linear in the specification and the system.
• Implementation revealed a bug in a priority
  buffer!

• S.Katz, D.Geist, O.Grumberg.
  Have I written enough
  properties?
• For ACTL safety formulas.
• Checking bisimulation between the system and the
  reduced tableau of the specification.
• Exponential in the (entire) specification.

8
Our ContributionDefinitions of
coverageChockler, Kupferman, Vardi 01

• Distinction between input, output, and control
  signals
• Coverage cannot be measured with respect to input
  signals, since we cannot affect their values.
• Changing the value of a control signal affects
  both the label and the transition relation.
• Changing the value of an output signal affects
  only the labeling of a state.

00
Request for coffee
Request for tea
10
01
9
Definitions of coverage CKV01

• State-based versus logic-based coverage
• In state-based coverage we study the effect of
  changing the value of one signal in one state.
• In logic-based coverage we study the effect of
  fixing the value of one signal everywhere.

Control signals are x and y
State-based coverage
Logic-based coverage
10
Definitions of coverage Chockler,Kupferman,Kursh
an,Vardi 01

• Different types of mutations with respect to
  unwinding the structure to an infinite tree
• Node-based coverage the value of a signal is
  changed in one occurrence of a state in the
  infinite tree.
• Structure-based coverage the value of a signal
  is changed in all the occurrences of the same
  state in the tree.
• Tree-based coverage the value of a signal is
  changed in a subset of the occurrences of a state
  in the infinite tree.

Satisfies Eventually q
Does not satisfy Always q or always not(q)
Does not satisfy Eventually q
Satisfies Always q or always not(q)
unwinding
q
q
11
Definitions of coverageChockler, Kupferman 02

• Coverage with respect to simulation
• A specification is given as a high-level design.
• A correct implementation is simulated by the
  design.
• A mutation is covered if it is no longer
  simulated by the design.

Implementation
Specification
12
Naive Algorithms forCoverage Computation
Checks each mutation separately.
number of mutations number of states X number
of signals

• Complexity
• For CTL specifications quadratic in the size of
  the structure and linear in the size of
  specification
• For LTL specifications quadratic in the size of
  the structure and exponential in the size of
  specification
• For simulation cubic in the size of state space

13
Improving Average Complexity
Mutations differ from each other only slightly
much of the model-checking or simulation can
be done once

• Incomplete labeling function permits to label the
  states with variables.
• Different assignments to the variables represent
  different mutations.
• A part of model-checking (or simulation) can be
  performed without assigning the variables.
• The rest can be performed in steps, each time
  assigning half of the variables.

14
Improving Average Complexity Applications CKV01

• Coverage of CTL specifications
• Automata-theoretic approach to CTL model-checking
  KVW94 represents a structure with the formula
  as an AND-OR graph, which looks like a Boolean
  circuit.
• Model-checking is equivalent to shrinkage of the
  graph with respect to the values of the leaves.
• Mutations differ from each other by the values of
  the leaves.
• Assignment to half of the leaves shrinks the
  circuit in average by the factor of 2.
• The complexity is O(n log n) on the average.

15
Improving Average ComplexityExample
Structure K
q
F
T
F
pq
p
T
T
16
Improving Average Complexity Applications CK02

• Coverage in simulation
• Enumerative algorithm of HHK95 starts with the
  maximal relation (with respect to the labeling
  function) and reduces it in each step.
• The complexity of simulation is O(n m ), where n
  is the size of the state space and m is the size
  of the transition relation.
• In the same way we can start with the maximal
  relation with respect to the incomplete labeling
  function.
• We never perform the same work twice!
• The average complexity of simulation by steps
  is
• O(n m log n).

17
Symbolic Algorithms for Coverage

• CTL specifications CKV01
• Compute the set of pairs of states ltw,vgt such
  that w satisfies the specification in the mutant
  where the value of the observable signal is
  flipped in v.

4n OBDD variables

• Simulation CK02
• Compute the set of triples ltw,v,wgt such that w
  simulates w in the mutant where the value of the
  observable signal is flipped in v.

6n OBDD variables

18
LTL CKKV01

• Node Coverage in FormalCheck
• Can be done by introducing a new variable that
  non-deterministically sets the step where we
  check the mutant instead of the original
  structure.

19
Recommended Workflowwith Coverage Check
Fix the model/specification
NO
YES
20
Other directions of researchVacuity
Chockler,Kupferman
Specification every request is eventually granted
satisfies the specification!
System S
Where is the problem?
The specification is satisfied in S vacuously
Best known complexity complexity of model
checking X size of the specification
Previous work BBER, KV
21
Other directions of researchResponsibility
Chockler,Halpern,Kupferman
Motivating example Coverage of existential
properties
In both systems all successors are uncovered
Specification There exists a successor labeled
with q
22
Other directions of researchProperty Testing
The idea Goldreich,Goldwasser,Ron 98
distinguish between good and very bad instances
with high probability by sampling a constant
number of bits from the input.
having the given property
Might be useful for checking the outputs of
random simulator (Intel).
Has a constant complexity!
Previous work Alon,Krivelevich,Newman,Szegedy
99 property testing algorithm for regular
languages.
23
Property testing insteadof model checking?
Fix the model/specification
counterexample
NO
There is surely a bug
YES
Does not always mean that the system is correct!
24
Property testing .
. instead of model checking?
Fix the model/specification
counterexample
NO
YES
Usually the systems contain many bugs
25
Work in progressChecking branchingtemporal
properties on treesChockler,Kupferman,Shpilka
Purely existential properties are trivially
testable, since they have no bad trees.
It is enough to change a constant number of paths
in a tree to make it satisfy an existential
property
Universal properties are testable with a constant
number of queries.
26
Summary

• Positive answer of model checker does not mean
  that the system is correct.
• The specification can cover only part of the
  system.
• We described efficient algorithms for coverage
  computation for CTL, LTL, and simulation.
• We presented verification methodology that
  contains property testing algorithms before
  (instead of) model checking and vacuity and
  coverage check after model checking.

27
Future Work

• Coverage algorithms for different types of
  modifications code coverage, circuit structure
  coverage, branch coverage, FSM coverage
• Coverage algorithms for other temporal logics.
• Coverage algorithms for SAT solvers.
• Implementation of coverage algorithms.
• Useful presentation of coverage information.