Experiences with Formal Specifications of FaultTolerant File Systems - PowerPoint PPT Presentation

1 / 36
About This Presentation
Title:

Experiences with Formal Specifications of FaultTolerant File Systems

Description:

Experiences with Formal Specifications of Fault-Tolerant File Systems ... [Chkliaev, et.al., 00], [Crow, et.al., 98], [Joshi, et.al., 03], [Houston, et.al., 91] ... – PowerPoint PPT presentation

Number of Views:58
Avg rating:3.0/5.0
Slides: 37
Provided by: ROX75
Category:

less

Transcript and Presenter's Notes

Title: Experiences with Formal Specifications of FaultTolerant File Systems


1
Experiences with Formal Specifications of
Fault-Tolerant File Systems
  • Roxana Geambasu (University of Washington)
  • Andrew Birrell (Microsoft Research)
  • John MacCormick (Dickinson College)

2
Fault-Tolerant File Systems (FTFSs)
  • FTFSs are crucial components in todays
    datacenters
  • They underlie most of what we do on the Web
  • Dependability correctness of FTFSs are
    paramount

Web services
Google Earth
Google Analytics
Amazon services
Google File System (GFS)
Niobe
Dynamo
2
3
FTFSs Are Extremely Complex
  • Contain sophisticated protocols for
  • replica consistency,
  • recovery (replica addition to compensate for
    failures),
  • reconfiguration (replica removal due to failure),
  • load balancing, etc.
  • Hence, FTFS protocols and implementation are hard
    to get right

4
Formal Methods (FM)
  • Formal methods have been used extensively to
    increase trust in complex systems
  • Formal specification languages are unambiguous
  • Model checking and formal proofs are reliable
  • However, FTFS designers still rely solely on
    prose and intuitive reasoning
  • Prose may be ambiguous, inaccurate
  • Intuitive reasoning may be faulty

5
FTFS Design and Analysis Challenges
  • Without formal methods, it is hard to
  • Understand FTFS behavior and semantics
  • Intuitive reasoning is hard and error-prone
  • Explore alternative designs
  • Alternative designs may affect semantics in
    complex ways
  • Compare various FTFSs
  • Prose is ambiguous and code bases are huge (tens
    of thousands of lines of code)

6
Goal Convince FTFS Builders to Use FM
  • Previous studies showed how and for what purposes
    to use FM for many classes of systems, e.g.
  • Local/distributed FSs, processor caches, TCP
    congestion
  • Our work
  • Shows how and for what purposes to use FM for
    another specific class of important systems
  • fault-tolerant file systems
  • Identifies convenient ways in which FM help in
    understanding, designing comparing FTFSs

7
Our Experience
  • We wrote TLA specifications for three protocols
  • Chain replication (Cornell University)
  • Niobe (Microsoft)
  • GFS (Google)
  • Our experience shows that FM help solve FTFS
    challenges
  • Comparing system mechanisms tradeoffs
  • Understanding and proving semantics
  • Exploring alternative designs

7
8
Outline
  • Specification effort
  • Experiences with formal specifications for FTFS
  • Comparing system mechanisms
  • Understanding and proving semantics
  • Exploring alternative designs
  • Conclusions

8
9
Specification Effort
  • Question How hard is it to build specifications?
  • Answer Moderately precise specifications are
    reasonably easy to produce

10
Outline
  • Specification effort
  • Experiences with formal specifications for FTFS
  • Comparing system mechanisms
  • Understanding and proving semantics
  • Exploring alternative designs
  • Conclusions

10
11
1. Comparing System Mechanisms
  • Case study GFS vs. Niobe
  • From prose, they seemed very different systems
  • GFS trades some consistency for throughput
  • Niobe designed for strong consistency
  • Our TLA specifications highlight significant
    mechanism overlap and also key differences

11
12
Capturing Similarities Differences
  • More than half of the TLA code-base is common
  • Specifications are small due to TLA
    expressiveness
  • Compare their total sizes to the tens of
    thousands of LOC of the systems implementations

single-master, primary-secondary
replication
Common
(291 lines)
Niobe
GFS
(287 lines)
(189 lines)
13
Differences Stand Out Clearly in TLA
  • Example Write completion in GFS and Niobe

w
w
1
4
w
w
ACK
ACK
3
2
14
Differences Stand Out Clearly in TLA
  • Example Write completion in GFS and Niobe

w
w
w
w
1
4
1
4
w
w
Group reconfiguration
w
w
ACK
ACK
ACK
ACK
3
2
3
2
15
Understanding Tradeoffs
  • Example Write completion in GFS and Niobe

Tradeoff
Smaller latency, but writes may leave group
inconsistent
A write never leaves replica group in
inconsistent state
16
Lesson Formalism Helps in Comparison
  • Formal specifications distill key differences and
    similarities between systems
  • Understanding the key differences enables us to
    understand tradeoffs

17
Outline
  • Specification effort
  • Experiences with formal specifications for FTFS
  • Comparing system mechanisms
  • Understanding and proving semantics
  • Exploring alternative designs
  • Conclusions

17
18
2. Understanding FTFS Consistency
  • Hard to prove consistency models for FTFSs
  • For weakly consistent systems, it can be even
    harder
  • Solution use refinement mapping
  • Reduce system to a really simple model
  • Prove the correctness of the reduction
  • Reason about the SimpleStore
  • For convenience, we use model-checking instead of
    full manual proofs at Step 2

SimpleStore
consistency model
consistency model
reduction
System
19
SimpleStores for the Three FTFSs
  • SimpleStores capture only client-visible
    behaviors and abstract out all protocol
    mechanisms
  • SimpleStores are easy to reason about

Chain_SS
Niobe_SS
GFS_SS
reduction
reduction
reduction
Chain
Blue
Niobe
Chain
GFS
20
Chains Consistency Semantics
Chain_SS
linearizable
linearizable
Proof is straightforward (half a page)
reduction
Chain
  • Using convenient methods, we gained reliable
    insight into Chains consistency model

20
21
Niobes Consistency Semantics
Chain_SS
Niobe_SS
GFS_SS
linearizable
linearizable
reduction
reduction
??
Chain
Niobe
GFS
linearizable
linearizable
  • Similar experience as with Chain
  • Thus, formal methods help in verifying standard
    consistency models for strongly-consistent FTFSs

21
22
GFS Consistency Semantics
  • Formal methods proved helpful in several ways
  • An interesting conclusion (details in the paper)
  • Using refinement mappings, we were able to show
    that, under a small set of assumptions, GFS has
    regular-register semantic

GFS_SS
well-defined intermediate-level consistency model
regular register
reduction
GFSassumptions
regular register
23
Lesson Formalism Helps Understand Semantics
  • Refinement mappings help in understanding
    reliably verifying consistency models of FTFS
  • They are useful for both strongly consistent and
    weakly consistent FTFSs

24
Outline
  • Specification effort
  • Experiences with formal specifications for FTFS
  • Comparing system mechanisms
  • Understanding and proving semantics
  • Exploring alternative designs
  • Conclusions

24
25
3. Exploring Alternative Designs
  • Exploring alternative designs is much easier
    using our framework (TLA specs, SimpleStores,
    reductions)

System SimpleStore
reduction
System model
25
26
Case-Study Changing Niobes Design
  • Currently, Niobes clients read from primary only
  • Reading from any replica may improve throughput
  • Design question
  • What happens to Niobe if it adopts read-any
    policy?

Niobe_SS
GFS_SS
linearizable
regular register
Chain
GFSassumption
Nioberead-any
?
regular register
regular register
27
Conclusions
  • FTFSs are extremely important in todays Web
  • We showed how formal methods can help improve our
    understanding and trust in FTFSs
  • Lessons from our experience with three FTFSs
  • Writing formal specifications is relatively easy
  • Formal methods enable
  • Insightful comparison of mechanisms tradeoffs
  • Reliable verification of consistency properties
  • Convenient investigation of alternative designs

28
Appendix
28
29
Related Work
  • FM are extensively used to reason about software
    Bickford, et.al., 96 and hardware Shimizu,
    et.al., 02
  • However, FTFS builders have not adopted them yet
  • By sharing our experience, we hope to convince
    FTFS builders of the utility of specifying their
    systems formally
  • Using FM to improve understanding and trust in
    systems
  • Previous works apply FM to various classes of
    systems
  • Chkliaev, et.al., 00, Crow, et.al., 98,
    Joshi, et.al., 03, Houston, et.al., 91
  • The closest works are those looking at
    distributed FS (AFS, Coda) Sivathanu, et.al.,
    05, Wing, et.al., 97, Yang, et.al., 04
  • We show how to apply them in the specific context
    of FTFS
  • Reducing complex systems to simple ones in order
    to reason about semantics has been used before
    Joshi, et.al., 03
  • We apply this method to FTFSs

30
GFS Assumptions
  • If
  • A write never crosses chunk boundaries
  • GFS client library offers chunk-level operations
  • A write never goes to a stale replica
  • Implement this assumption using a lease mechanism
  • Then

GFS_SS
regular register
reduction
GFSassumptions
regular register
31
Standard Consistency Models
  • Linearizability (Atomic register semantic)
  • Any client-visible history H generated by the
    system is equivalent to a legal sequential
    interleaving S
  • The sequential interleaving S preserves the
    real-time ordering of operations from H
  • Serializability
  • Any client-visible history H generated by the
    system is equivalent to a legal sequential
    interleaving S
  • Regular register semantic
  • Read not concurrent with any write returns most
    recently written value
  • Read concurrent with some writes returns either
    the value of the in-process writes or the most
    recently written value
  • Safe register semantic
  • Read not concurrent with any write returns most
    recently written value
  • Read concurrent with some writes can return
    anything

32
Summary of Contributions
  • Identified a new important class of extremely
    complex systems FTFSs
  • Showed three aspects of FTFS design analysis
    for which FM prove especially valuable
  • Mechanism comparison, semantics understanding,
    and design space exploration
  • Showed how to apply specific FMs to FTFSs
  • Showed how to construct SimpleStores and what can
    be learned from them
  • SimpleStores are reusable between systems
  • We believe that our study, tailored toward FTFSs,
    can be more relevant to FTFS designers than more
    general studies

32
33
Lessons from Our Experience
  • Building high-level specifications for FTFS is
    relatively easy
  • It is also remarkably useful for understanding
    system
  • The exercise of writing specifications exposes
    similarities in seemingly dissimilar systems
    (GFS, Niobe)
  • Formal specifications also distill the key design
    differences
  • Specifications enable convenient verifications of
    consistency for both strongly and weakly
    consistent systems
  • Niobe and Chain are both linearizable
  • GFS can be upgraded to regular register via a
    clear set of assumptions
  • GFS design to read from any replica heavily
    influences its consistency
  • Intuition can fail often times
  • Niobe seemed to be reducible to Chain_SS, but
    actually was not

34
Chain SimpleStore
Responses
Requests
writes
reads
writes
reads
read channel
r1
r2
r3
read()
SerialDB
w6
commit(w5)
w7
w5
write channel
drop(w7)
Chain_SS
34
35
The Temporal Logic of Actions (TLA)
  • Formalism that combines a temporal logic with a
    logic of actions
  • Especially designed for specification of
    distributed asynchronous systems
  • TLA specifications model the system as a state
    machine
  • Define system variables (state)
  • Model actions that the system can take as state
    transitions

35
36
Understanding Tradeoffs
Smaller write latency, but writes may leave
group inconsistent
A write never leaves replica group in
inconsistent state
Error
Old value
4
1
4
read
read
1
3
2
3
2
Write a Comment
User Comments (0)
About PowerShow.com