FaultScalable Byzantine FaultTolerant Services

1
Fault-Scalable Byzantine Fault-Tolerant Services

• Abd-El-Malek, Ganger, et. al.
• Carnegie Mellon University
• SOSP 2005

2
Motivation

• Byzantine Fault-tolerance
• Fault-Scalable Service
• Performance degrades gradually if more server
  faults are tolerated
• Agreement-based approach fail due to
  server-server broadcast
• Asynchronous communication model

3
Overview

• Introduction
• Efficiency and Scalability
• Query/Update (Q/U) Protocol
• System Model
• Working
• Classifications and Constraints
• Implementation Details
• Evaluation

4
Salient Features of Q/U Protocol

• Efficient, optimistic, quorum-based protocol
• Operations queries and updates
• Tolerates Byzantine-faulty clients servers
• Preferred Quorum per object
• Single Phase Updates in normal execution
• 36 decrease in performance vs 83 for BFT from
  1-5 faults
• Cost 5b1 servers for b faulty servers

5
Efficiency Scalability

• Efficiency
• No prepare phase in failure atomicity, but need
  to repair inconsistent objects
• No locks in concurrency atomicity, but
  possibility of livelock
• preferred quorum, consistent object version lead
  to single phase operations

• Throughput-Scalability
• Comparison with quorum protocols in database
  systems
• Two-phase, synchronous, scalability
• Q/U protocol allows fine-grained objects
• Advantages reduces contention, concurrent
  updates

6
Q/U Protocol System Model

• Asynchronous Timing Model
• Point-to-Point channels between servers/clients
  with symmetric key infrastructure between each
  pair
• Channels are unreliable drop, duplicate, resend
  messages
• Clients/Servers may be Byzantine faulty
• Server
• Has persistent storage durable through a crash
  recovery
• Hybrid failure model combining Byzantine with
  crash-recovery failures
• benign correct/follows its specification except
  for crash-recovery
• malevolent out-of-specification non-crash
  behavior
• faulty crashes and (does not recover/ takes
  infinite time to recover) or is malevolent
• U Universal, T Faulty, B Malevolent, Q
  quorum

7
Terminology

• queries and updates, methods take arguments,
  return answers, request a quorum, server receives
  requests, if accepts it invokes a method on its
  local object version
• Update method results in new object version
• Server has object version alongwith its logical
  timestamp in version history called replica
  history (returned in response to request)
• Client stores servers replica histories in
  object history set (OHS)
• OHS represents client partial observation of
  global system state
• Timestamps in OHS are candidates
• Candidates are classified to determine
• which object version should be used
• Candidates ? object version

8
Client Side

• Non-optimistic execution Clients perform repair
  phase
• Clients operation is said to condition on its
  Object History Set (OHS)
• Client sends OHS, method to invoke, argument to
  method to servers
• Client Server classify candidate in
  conditioned-on OHS gt object version to be
  operated on
• Threshold quorums classify based on order of a
  candidate
• If all replicas have same version then candidate
  is complete
• Single phase operations are dependent on
• Clients cache OHS (concurrency-free)
• Remains current as long as no other client
  updates the object
• Clients maintain locality of preferred quorum
  (failure-free)
• Object version will be latest

9
Server Side

• Server validates integrity of OHS
• Client sends authenticators corresponding to
  every server
• Authenticators are lists of HMACs
• Server checks whether OHS is current
• Compares local replica history to OHS
• If server has higher timestamp OHS is not current
• Server performs operation
• If update then new object version
• Server sends response back to client indicating
  success, including the answer, replica history,
  and its authenticator
• Client returns success if it receives quorum of
  responses

10
Tolerating failures

• If server crashes, some quorum become unavailable
• Clients use probing (finding a live quorum) to
  collect quorum of responses
• Timestamps contain clientID, operation, OHS
• Operation can complete successfully if client
  sends same operation to all servers in quorum
• Identical object versions at servers
• Authenticators make it impossible to forge
  replica histories of benign servers

11
Concurrency and Repair

• Concurrent updates may lead to different object
  versions gt repair consisting of barrier and copy
  operations
• Barrier
• Clients safely advance logical time during
  contention
• Prevents other operations from completing
• Copy
• Copies latest object version prior to barrier
  forward in logical time
• Classification of OHS is necessary to decide need
  for repair
• Clients and servers both perform classification
• Different clients could be copying different
  repairable candidates forward (livelock)
• Client backoff

12
Non-preferred quorum access

• If non-preferred quorum is accessed
• Servers may not have updated object version
• Server object-sync to retrieve latest object
  version
• sync-server (one who wants latest object) gets
  object from host-server (one who has latest
  object)
• Info on host-servers from OHS
• Responses from atleast b1 host-servers is
  required

13
Classification and Constraints

• Candidate is classified as complete, repairable
  or incomplete
• If complete global information then candidate
  could be established accepted by all benign
  servers in some quorum
• repairable sets
• potential candidate accepted at all benign
  servers in some repairable set

14
Implementation details

• Cached OHS not current
• Server rejects operation and returns its current
  replica history
• Server performs operation and sends its answer
  client classifies OHS after quorum of responses
• Quorum access strategy and probing
• If not preferred quorum then probably object-sync
  
• Deterministic function to map object ID to
  preferred quorum - (ID)mod(n)
• Dynamic mapping for load balancing
• Inline repair
• Repair candidate in place by performing the
  operation (if no contention)
• Use case - when non-preferred quorum is accessed
  for first time
• Handling repeated requests at server
• Store some history of answers
• Retry and Backoff
• Update-update concurrency leads to backoff
  (random exponential)
• Update-query concurrency leads to inline repair
  (no backoff)

15
Implementation details (contd)

• Object syncing
• One server sends entire object and others send
  hash
• Authenticators
• HMAC vs digital signatures (large n)
• Compact timestamps
• Hash of operation and OHS
• Compact replica histories
• Prune replica histories and delete old object
  versions (tradeoff)
• Malevolent components
• Clients not following backoff DOS attack
  (server could rate limit clients)
• Clients may send updates to subset of quorum
  (lazy verification paper)
• Server can return invalid authenticator
  (techniques within code)

16
Correctness

• Operations are strictly serializable
• conditioned-on chain set of object versions
  found by traversing back in logical time to lt0,0gt
• In a benign execution the Q/U protocol is
  obstruction-free

17
Evaluation
18
Evaluation (contd)
19
Questions!!