Serdar Tasiran, Tayfun Elmas Ko University, Istanbul, Turkey

1
VYRD VerifYing Concurrent Programs by Runtime
Refinement-violation Detection

• Serdar Tasiran, Tayfun Elmas Koç University,
  Istanbul, Turkey
• Shaz QadeerMicrosoft Research, Redmond, WA

2
Motivation
Verifying ConcurrentData Structures

• Widely-used software systems are built on
  concurrent data structures
• File systems, databases, internet services
• Standard Java and C class libraries
• Intricate synchronization mechanisms to improve
  performance
• Prone to concurrency errors
• Concurrency errors
• Data loss/corruption
• Difficult to detect, reproduce through testing

3
Our Approach
Runtime Checking of Refinement

• Refinement
• For each execution of Impl
• there exists an equivalent, atomic execution of
  Spec
• Use refinement as correctness criterion
• More thorough than assertions
• More observability than pure testing
• Runtime verification Check refinement using
  execution traces
• Can handle industrial-scale programs
• Intermediate between testing exhaustive
  verification

4
The VYRD Tool
Test harness
Impl
Write to log
...
Returnsuccess
Call Insert(3)
Unlock A0
A0.elt3
Call LookUp(3)
Returnsuccess
Unlock A1
A1.elt4
Returnsuccess
read A0
Return true
A0.eltnull
Unlock A0
Call Insert(4)
Call Delete(3)
Read from log
Execute logged actions
Run methods inwitness ordering
Replay Mechanism
Implreplay
Spec
Refinement Checker
traceImpl
traceSpec
5
Outline

• Example
• Refinement
• Testing
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience

6
Multiset
Implementation LookUp

• Multiset data structure
• M 2, 3, 3, 3, 9, 8, 8, 5
• Why this example?
• Mimics patterns we encounteredin real systems

LookUp (x) for i 1 to n acquire(Ai) if
(Ai.contentx Ai.valid)
release(Ai) return true else
release(Ai) return false
7
Multiset
Implementation FindSlot

• FindSlot Helper routinefor InsertPair
• For space allocation
• Does not set valid field
• x not in multiset yet

FindSlot (x) for i 1 to n acquire(Ai) if
(Ai.contentnull) Ai.content x
release(Ai) return i else
release(Ai) return 0
8
Multiset
Implementation InsertPair

• Refinement violation if only one of x, y
  inserted
• Allows exceptional termination
• Example Multiset array of size 2
• 2 concurrent InsertPairs
• both find slots for xs
• both fail to find slots for ys
• Not possible in atomic execution
• First InsertPair must succeed
• This execution of Multiset is not buggy
• But there is no equivalent linear or atomic
  execution of Impl
• Not reducible, not linearizablewithout separate
  Spec

9
Multiset
Specification

• Spec state
• M set of integers
• Each method
• Atomic deterministic state update/observation
• Given
• current state, arguments and
• method return value (if one exists)
• specifies new Spec state

10
Outline

• Example
• Refinement
• Testing
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience

11
Multiset
Testing

• Dont know which happened first
• Insert(3) or Delete(3) ?
• Should 3 be in the multiset at the end?
• Must accept both possibilities as correct
• Common practice
• Run long multi-threaded test
• Perform sanity checks on final state

12
Multiset
I/O Refinement
13
The Vyrd Approach
Refinement asCorrectness Criterion

• Refinement
• For each execution of the implementation (Impl)
• there exists an equivalent, atomic execution of
  Spec
• Linearizability, atomicity/reducibility
• For each execution of Impl
• there exists an equivalent atomic execution of
  Impl
• Refinement was used as correctness criterion for
  verifying Boxwood, the Scan file system
• Atomicity/reducibility, linearizability declare
  these systems incorrect
• Reducibility, linearizability sometimes too
  restrictive
• Refinement as correctness criterion
• More extensive checking than assertions
• More observability than pure testing
• More about this in a moment

14
I/O-refinement
Selecting Commit Actions

• Commit points Determine witness ordering, drive
  Spec
• Hints to refinement checking tool
• No formal procedure
• Intuitively, where new data structure state
  becomes visible to other threads
• Straightforward for programmer to select
• For each method
• Designate lines in source code
• Multiple lines annotated as commit
• May be conditional on program state, return value
• For each method execution, only one action should
  be marked as the commit action

release(Ai) // commit
15
Outline

• Example
• Refinement
• Testing
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience

16
View-refinement
Need for more observability
T1 InsertPair(5,7)
T2 InsertPair(6,8)
Read A0.elt null
FINDSLOT (x) // Buggy for i ? 1 to n if
(Ai.content null) acquire(Ai)
Ai.content x release(Ai) return
i return 0
Read A0.elt null
Read A1.elt null
0
1
2
3
?
?
?
?
elt
valid
F
F
F
F
elt
5
7
?
?
valid
F
F
F
F
5
7
?
?
elt
Overwrites 5!
valid
T
T
F
F
LookUp(5)true, LookUp(7)true
0
1
2
3
6
7
?
?
elt
valid
T
T
F
F
Read A2.elt null
6
7
8
?
elt
valid
6
7
8
?
elt
valid
LookUp(6)true, LookUp(8)true
LookUp(5)false
17
View-refinement
I/O-refinementmay miss errors
T1 InsertPair(5,7)
T2 InsertPair(6,8)
Read A0.elt null

• If observer methods dont get interleavedin the
  right place,I/O refinement maymiss errors
• Source of bug too far in the past when I/O
  refinement violation happens

Read A0.elt null
Read A1.elt null
0
1
2
3
?
?
?
?
elt
valid
F
F
F
F
elt
5
7
?
?
valid
F
F
F
F
5
7
?
?
elt
Overwrites 5!
valid
T
T
F
F
LookUp(5)true, LookUp(7)true
0
1
2
3
6
7
?
?
elt
valid
T
T
F
F
Read A2.elt null
6
7
8
?
elt
valid
6
7
8
?
elt
valid
LookUp(6)true, LookUp(8)true
18
View-refinement
More Observability

• I/O-refinement may miss errors
• View-refinement
• I/O-refinement correspondence between states
  of Impl and Spec at commit points
• Catches state discrepancy at the next commit
  point
• Early warnings for possible I/O refinement
  violations

19
View-refinement
View Variables

• State correspondence
• Hypothetical view variables must match at
  commit points
• view variable
• Value of variable is abstract data structure
  state
• Updated atomically once by each method
• viewImpl state information for Impl
• For A1..n
• Extract contentif validtrue
• viewSpec state information for Spec
• Elements of the multiset
• viewSpec ? M (nothing to abstract)
• Other Specs may have state to be abstracted

20
View-refinement
View Variables for Multiset

• viewImpl Computed using abstraction function
• View is a canonical representation
• Canonizes state for viewExact match not required

viewImpl1, 3, 5, 6
21
View-refinement
Checking Refinement
22
View-refinement
Catching FindSlot Bug
T1 InsertPair(5,7)
T2 InsertPair(6,8)
Read A0.elt null
FINDSLOT (x) // Buggy for i ? 1 to n if
(Ai.content null) acquire(Ai)
Ai.content x release(Ai) return
i return 0
Read A0.elt null
Read A1.elt null
0
1
2
3
?
?
?
?
elt
valid
F
F
F
F
elt
5
7
?
?
valid
F
F
F
F
5
7
?
?
elt
Overwrites 5!
valid
T
T
F
F
LookUp(5)true, LookUp(7)true
0
1
2
3
6
7
?
?
elt
valid
T
T
F
F
Read A2.elt null
6
7
8
?
elt
valid
6
7
8
?
elt
valid
LookUp(6)true, LookUp(8)true
23
View-refinement
Catching FindSlot Bug
viewSpec Ø
5, 7
5, 6, 7, 8
viewImpl Ø
5, 7
6, 7, 8
InsertP(5,7)Returns success
InsertP(6,8)Returns success
Call InsertP(5,7)
Call InsertP(6,8)
Read A0.elt
A0.content5
Read A0.elt
A0.validtrue
A0.validtrue
Read A0.elt
A1.content7
A1.validtrue
A0.content6
A2.content8
A0.validtrue
A2.validtrue
Read A2.elt
Implementation
24
Outline

• Example
• Refinement
• Testing
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience

25
The VYRD Tool

• Instrument Impl
• in order to log actionsin the order they happen
• Commit actions annotated by user
• Write abstraction function

Test harness
Impl
Write to log

• Enablesonline/offline checking

...
Returnsuccess
Call Insert(3)
Unlock A0
A0.elt3
Call LookUp(3)
Returnsuccess
Unlock A1
A1.elt4
Returnsuccess
read A0
Return true
A0.eltnull
Unlock A0
Call Insert(4)
Call Delete(3)
Read from log
Execute logged actions
Run methods inwitness ordering
Replay Mechanism
Implreplay
Spec
Refinement Checker
traceImpl
traceSpec
26
The VYRD Tool
Atomized Impl as Spec

• Spec atomized version of Impl
• Fully synchronized methods
• Use single global lock
• Separates checking concurrency errors from
  sequential verification
• Slight modification
• Return value from Impl method additional argument
  to Spec methods
• More permissive than Impl
• Can handle failure return values
• Exact state match at commit pointsnot required
• Match view variables only
• Different from commit atomicity

27
Outline

• Example
• Refinement
• Testing
• I/O-refinement
• View-refinement
• The VYRD tool
• Experience

28
Experience
The Boxwood Project
29
Experience
Experimental Results

• Scalable method Caught bugs in industrial-scale
  designs
• Boxwood (30K LOC)
• Scan Filesystem (Windows NT)
• Java Libraries with known bugs
• Moderate annotation effort
• Several lines for each method
• I/O-refinement
• Low logging and verification overhead
• BLinkTree Logging 17 over testing, refinement
  check 27
• View-refinement
• BLinkTree Logging 20 over testing, refinement
  check 137
• More effective in catching errors
• Boxwood Cache of random method calls until
  error caught
• View-refinement 26
• I/O-refinement 539

30
Experience
Concurrency Bug in Cache

• Very similar to bug found in Scan file system
• Had not been caught by developers
• Current version does not contain bug
• Bug manifestation
• Cache entry is correct
• Permanent storage has corrupted data
• Cause of bug Concurrent execution of Write and
  Flush on the same entry
• Write to a dirty entry not locked properly
• Flush writes corrupted data to Chunk Manager
• Marks entry clean
• Hard to catch through testing
• As long as Reads hit in Cache, return value
  correct
• Caught through testing only if
• Cache fills, clean entry in Cache is evicted
• No Writes to entry in the meantime
• Entry read after eviction
• Very unlikely

31
Conclusions

• Runtime refinement checking
• Powerful technique with reasonable computational
  cost
• Effective for complex industrial-scale software
• Key novelty Improves observability of testing
• Future work
• Improving coverage/controllability
• Reducing manual instrumentation
• by limited use of model checking
• Tayfun Elmas, Serdar TasiranVyrdMC Driving
  Runtime Refinement Checking with Model Checkers
  Fifth Workshop on Runtime Verification
  (RV'05).The University of Edinburgh, Scotland,
  UK. July 12, 2005.