Consistency and Replication

1
Consistency and Replication

• Chapter 6

Part I Consistency Models
2
Reasons for Replication

• Reliability
• Mask failures
• Mask corrupted data
• Performance
• Scalability (size and geographical)
• Examples
• Web caching
• Horizontal server distribution
• Object distribution

3
Example Object Replication (1)

• Organization of a distributed remote object
  shared by two different clients.

4
Example Object Replication (2)

• A remote object capable of handling concurrent
  invocations on its own.
• A remote object for which an object adapter is
  required to handle concurrent invocations

5
Example Object Replication (3)

• A distributed system for replication-aware
  distributed objects.
• A distributed system responsible for replica
  management

6
Cost of Replication

• Replicas must be kept consistent
• Dilemma
• Replicate data for better performance
• Modification on one copy triggers modifications
  on all other replicas
• Propagating each modification to each replica can
  degrade performance
• When and how the modifications are made
  consistency model
• Weak versus strong consistency model

?
7
Consistency Issues Access/Update Ratio
Lost Updates
User accesses to the page

Updates to the Web page
time
8
Consistency Models

• The general organization of a logical data store,
  physically distributed and replicated across
  multiple processes.

9
Framework for Consistency Partial and Total
Orders

• Let S be a set, and R ? S ? S
• R is anti-reflexive if ?x ? S, (x,x) ? R
• R is transitive if ?x, y, z ? S, if (x,y) ? R and
  (y,z) ? R then (x,z) ? R
• A PO is an anti-reflexive, transitive relation
• A PO is denoted by (S,R)
• xRy means (x,y) ? R
• A TO is a PO (S,R) such that ?x, y? S x ? y,
  either xRy or yRx

10
Framework for Consistency Operations and Data
Items

• Operations are either writes or reads (other
  operations are possible)
• A write is denoted wp(x)v
• A read is denoted rp(x)v
• A read-write data item is the set of all
  sequences lto1, o2, ongt such that
• Each oi is either a read or a write
• Each read returns the same value written by the
  most recent preceding write in the sequence

11
Framework for Consistency Operations and
Processes

• Each operation can be decomposed into two
  components
• Invocation and response
• wp(x)v invocation wp(x)v response empty
• rp(x)v invocation rp(x)? response v
• A process is a sequence of operation invocations
• A process computation is a sequence of operations
  obtained by augmenting each invocation in the
  process by its response

12
Framework for Consistency Multiprocess Systems

• A (multiprocess) system (P,D) is a set of
  processes, P, and a set of data items, D, such
  that all operation invocations of processes in P
  are applied to items in D
• A (multiporcess) system (P,D) computation is a
  collection of process computations one for each
  process in P

13
Framework for Consistency Example
Program p x y
Program q y x
System (P,D) P p,q D x,y
Process p r(y)v? w(x)v?
Process q r(x)v? w(y)v?
System (P,D) Computation p r(y)5 w(x)5 q r(x)0
w(y)0
Process p Comp r(y)5 w(x)5
Process q Comp r(x)0 w(y)0
14
Framework for Consistency Program Order

• Define program order, dnoted (O, ltpo), by o1ltpo
  o2 iff o2 follows o1 in ps computation

Program p x y
Program q y x

• rp(y)5 ltpo wp(x)5
• rq(x)0 ltpo wq(y)0
• All of program order for the exmple

Process p r(y)v? w(x)v?
Process q r(x)v? w(y)v?
Process p Comp r(y)5 w(x)5
Process q Comp r(x)0 w(y)0
15
Framework for Consistency Consistency Models

• A consistency model is a set of constraints on
  system computations
• A system computation of (P,D) satisfies a
  consistency model CM if the computation meets all
  the constraints in CM
• For two consistency models CM1 and CM2 CM1 is
  stronger than CM2 if the constraints of CM1 imply
  those of CM2
• CM2 is weaker than CM1

16
Framework for Consistency Validity

• Given a set of operations O
• Ow indicates all the write operations in O
• Or indicates all the read operations in O
• Op is the subset of O containing ps operations,
  for some process p
• Ox is the subset of O containing operations on
  x, for some data item p
• Let (O,lt) be a total order of O
• (O,lt) is valid if for each data item x, the
  subsequence (Ox,lt) is valid for x.

17
Framework for Consistency Valid Total Orders
Computation p w(x)5 r(y)5 q r(x)0 w(y)5 r(x)5
x and y are initially 0
Valid Total Order rq(x)0 wq(y)5 wp(x)5 rq(x)5
rp(y)5
Invalid Total Order wp(x)5 rq(x)0 wq(y)5 rq(x)5
rp(y)5
18
Sequential Consistency (SC) Lamport

• the result of any execution is the same as if
  the operations of all the processes were executed
  in some sequential order, and the operations of
  each indvidual process appear in this sequece in
  the order specified by its program
• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies SC if there is a valid total order
  (O,lt) such that (O,ltpo) ? (O,lt)

19
SC Intuition

process
process
process
FIFO Channels
Switch (e.g. bus, token)
All Data Items ( the set D)
20
Sequential Consistency Examples
C1 satisfies SC (O,lt) ltwp(x)1, rq(x)1, wq(x)2,
rp(x)2gt (O,ltpo) (wp(x)1, rp(x)2), (rq(x)1,
wq(x)2)
C2 does not satisfy SC (O, ltpo) (wp(x)1,
rp(x)2), (wq(x)2, rq(x)1) ltwp(x)1, rq(x)1,
wq(x)2, rp(x)2gt (violates PO) ltwp(x)1, wq(x)2,
rp(x)2, rq(x)1gt (is not valid) Cycle wp(x)1 ?
wq(x)2 wq(x)2 ? wp(x)1
Exercise Does C3 satisfy SC? (x and y are
initially 0)
21
Coherence Goodman

• SC per data item
• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies Coherence if for each x ? D there is a
  valid total order (Ox,ltx) such that (Ox,ltpo) ?
  (Ox,ltx)

22
Coherence Intuition

process
process
process
FIFO Channels

One Data Item
One Data Item
One Data Item
23
Coherence Examples
C1 satisfies Coherence (Ox,ltx) ltwp(x)1,
rq(x)1, wq(x)2, rp(x)2gt
C2 does not satisfy Coherence
C3 satisfies Coherence but not SC
Does C4 satisfy Coherence? SC?
24
SC versus Coherence

• If Computation C satisfies SC, then it satisfies
  Coherence
• Proof exercise
• If a Computation C satisfies Coherence, then it
  does not necessarily satisfy SC
• Proof Computation C3 is a counter example

All Computations satisfying consistency model CM
C(CM)
C(Coherence)
C(SC)
25
Pipelined Random Access Machine (P-RAM) Lipton
Sandberg

• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies Coherence if for each p ? P there is a
  valid total order (Op ? Ow,ltp) such that (Op ?
  Ow,ltpo) ? (Op ? Ow,ltp)

26
P-RAM Intuition
FIFO Channels
27
P-RAM Examples
C1 satisfies P-RAM (also SC and Coherence) (Op ?
Ow,ltp) ltwp(x)1, wq(x)2, rp(x)2gt (Oq ? Ow,ltq)
ltwp(x)1, rq(x)1, wq(x)2gt
C2 satisfies P-RAM but not Coherence
C3 satisfies Coherence but not SC nor P-RAM
Does C4 satisfy P-RAM?
Does C5 satisfy Coherence? P-RAM? SC?
28
SC versus P-RAM

• If Computation C satisfies SC, then it satisfies
  P-RAM
• Proof exercise
• If a Computation C satisfies P-RAM, then it does
  not necessarily satisfy SC
• Proof Computation C4 is a counter example

C(P-RAM)
C(SC)
29
Coherence versus P-RAM

• If Computation C satisfies Coherence, then it
  does not necessarily satisfy P-RAM
• Proof Computation C5 is a counter example
• If a Computation C satisfies P-RAM, then it does
  not necessarily satisfy Coherence
• Proof Computation C2 is a counter example
• There are computations that satisfy both
  Coherence and P-RAM, but not SC
• Proof find a computation C

C satisfies P-RAM and Coherence, but not SC
C(P-RAM)
C(Coherence)
C(SC)
30
Causal Consistency (CC) Ahamad et al.

• Define the write-before-read order, (O,ltwbr), by
  o1 ltwbr o2, if o1 is w(x)v and o2 is r(x)v for
  some x and v.
• Define the Causal order order (O,ltco) ((O,ltwbr)
  ? (O,ltpo))
• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies CC if for each p ? P there is a valid
  total order (Op ? Ow,ltp) such that (Op ?
  Ow,ltco) ? (Op ? Ow,ltp)

31
CC Intuition
p q s
w(x)0
w(x)1
r(x)1
w(y)2
r(y)2
r(x)0

• Allowed in P-RAM
• If s sees wq(y)2 which was performed after
  rq(x)1, which sees
• wp(x)1 performed after wp(x)0, it must be the
  case that rs(x)0 also
• sees wp(x)1

32
CC Examples (1)
C1 satisfies CC (also SC, Coherence, P-RAM ),
exercise
C2 satisfies CC and P-RAM but not Coherence
C7 satisfies CC, P-RAM, and Coherence but not SC
C4 satisfies CC, P-RAM, and Coherence, but not
SC.
C5 satisfies Coherence, but not CC, P-RAM, or SC?
33
CC Examples (2)
C6 satisfies P-RAM, Coherence, but not CC
(neither SC)

• (O,ltwbr) (wp(x)3,rs(x)3), (wp(x)1,rq(x)1),
  (wq(y)1,rs(y)1)
• Since wp(x)3 ltpo wp(x)1 and wp(x)1 ltwbr rq(x)1,
  then wp(x)3 ltco wp(x)1 ltco rq(x)1
• But rq(x)1 ltpo wq(y)1 and wq(y)1 ltwbr rs(y)1,
  then
• wp(x)3 ltco wp(x)1 ltco rq(x)1 ltco rs(y)1
• Finally, rs(y)1 ltpo rs(x)3, therefore
• wp(x)3 ltco wp(x)1 ltco rq(x)1 ltco rs(y)1 ltco
  rs(x)3, which is invalid

34
Comparison of read-write models

• If Computation C satisfies CC, then it satisfies
  P-RAM
• Proof follows from CC definition
• If a Computation C satisfies P-RAM, then it does
  not necessarily satisfy CC
• Proof Computation C6 is a counter example
• Coherence and CC are incomparable (exercise)
• SC is stronger than CC (exercise)

CC
C(P-RAM)
C(Coherence)
C(SC)
35
Synchronization Operations

• In addition to reads and writes, introduce
  synchp() operation
• Os denotes the subset of O containing synch
  operations
• Define the weak program order order, (O,ltwpo), by
  o1 ltwpo o2 if o1 ltpo o2 and
• o1 and o2 are on the same data item,
• o1 or o2 is a synchronization operation, or
• There is o st o1 ltwpo o and o ltwpo o2

36
Weak Consistency (WC) Dubios et al.

• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies WC if for each p ? P there is a valid
  total order (Op ? Ow,ltp) such that
• (Op ? Ow ? Os,ltwpo) ? (Op ? Ow ? Os,ltp)
• ?q ? P, (Os,ltp) (Os,ltq)

37
WC Intuition

process
process
process
S1 S3 S2
S1 S3 S2
S1 S3 S2

S1
S3
Reads and writes
Synchronization Points
S2
38
WC Example

• All of p, q, and m must agree on a total order
  of synch operations consistent with program
  order for example
• ltsq(), sp(), sm(), sq()gt
• (Op ? Ow, ltp) lt wm(x)5, sq(), wp(x)3, sp(),
  wq(y)1, sm(), sq() gt
• (Oq ? Ow, ltq)
• lt rq(x)0, wm(x)5, wp(x)3, sq(), sp(),
  wq(y)1, sm(), sq(), rq(x)3gt
• (Om ? Ow, ltm)
• lt wm(x)5, sq(), wp(x)3, sp(), wq(y)1,
  rm(y)1, sm(), sq(), rm(x)3 gt
• Exercise construct a computation that does not
  satisfy WC

39
More Synchronization Operations

• In addition to reads and writes, introduce
  relp(l) and acqp(l) operation (Os)
• relp(l) p releases lock l
• acqp(l) p acquires lock l
• Define the acquire-release order order, (O,ltaro),
  by o1 ltaro o2 if o1 ltpo o2 and
• o1 and o2 are on the same data item,
• o1 is acquire and o2 is a read or write,
• o1 is a read or write and o2 is a release, or
• There is o st o1 ltwpo o and o ltwpo o2

40
Release Consistency (RC) Gharachorloo et al.

• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies RCsc if for each p ? P there is a valid
  total order (Op ? Ow ? Os,ltp) such that
• (Op ? Ow ? Os,ltaro) ? (Op ? Ow ? Os,ltp)
• ?q ? P, (Os,ltp) (Os,ltq)
• (Os,ltpo) ? (Os,ltp) SC

41
RC Intuition

process
process
process
A2
R2
A1

Critical Section
R1
A3
R3
42
Timed Operations

• When there is a global time in the system,
  invocation and responses of operations are time
  stamped
• Define the time-order order, (O,ltto), by o1 ltto
  o2 iff invocation(o2).ts lt response(o1).ts

43
Linearizability (Lin) Herlihy Wing

• Let O be the set of all the operations of a
  computation C of a system (P,D). Then, C
  satisfies Lin if there is a valid total order
  (O,lt) such that
• (O,ltpo) ? (O,lt)
• (O,ltto) ? (O,lt)

44
Linearizability versus SC
response
w(x)1
w(x)2
r(x)3
Invocation
r(x)2
w(x)3
time
Linearizable
w(x)1
w(x)2
r(x)3
p
r(x)2
w(x)3
q
time
SC but not Linearizable
45
Lazy Consistency Models

• When updates are scarce
• When updates are not conflicting
• Examples DNS and WWW
• Eventual Consistency (EC) Lazy propagation of
  updates to all replicas
• If no updates take place for a long time, all
  replicas will become consistent
• Cheap to implement
• If a client always accesses the same replica, EC
  is trivial

46
Eventual Consistency

• Read-any/write any replication scheme with a
  mobile client