Epidemic Techniques

1
Epidemic Techniques

• Algorithms and Implementations

2
Agenda

• Consistency issues
• Epidemic algorithms
• Astrolabe
• Conclusion

3
Databases replicated at many sites need to
maintain consistency

• Relaxed consistency problem
• Database is changed at one site
• Change must propagate to all other sites
• All copies must eventually agree
• Copies should be mostly current
• Important factors
• Propagation time
• Network traffic (ideally proportional to Size of
  the update X Number of servers)

4
Epidemic algorithms help spread updates and
maintain consistency

• Epidemic terminology
• A site with an update it is willing to share is
  infective
• A site which has yet to receive an update is
  susceptible
• A site with an update it is no longer willing to
  share is removed

Susceptible
Infective
Removed
5
Agenda

• Consistency issues
• Epidemic algorithms
• Astrolabe
• Conclusion

6
Xerox has three algorithms to create database
consistency

• Direct mail
• Updates are mailed from originating site to all
  other sites
• Anti-entropy (epidemic)
• All sites regularly chose other sites and
  exchange database contents
• Rumor mongering (epidemic)
• Updates become hot rumors which are
  periodically sent to other sites until most sites
  contacted are infective

7
Direct mail is almost, but not completely reliable

• Queues
• Updates are queued to prevent delays
• Queue located in stable storage
• Failures
• Queues overflow
• Destinations inaccessible for long periods of
  time
• Source lacks accurate knowledge of all other
  sites
• Traffic
• n messages per update
• Each message traverses all links from source to
  destination
• Traffic proportional to the number of sites X
  distance between sites

Server 1
Update
Queue
Mail

Server n
8
Anti-entropy is reliable but costly

• Site A choose site B at random
• The databases are compared
• Pull A gets database from B
• Push A sends database to B
• Push-pull A and B exchange databases
• When used as backup pull or push-pull is
  preferable

Pull
A
B
Push
A
B
Push-pull
A
B
9
Checksums can be used with anti-entropy to
improve performance

• Comparing databases is expensive
• A recent update list can be kept
• Recent updates are exchanged
• Updates applied
• Checksums of database contents exchanged
• Databases compared only if checksums disagree

10
Rumor mongering is less costly but can be
inconsistent

• n individuals initially susceptible
• Rumor planted making A infective
• A contacts others at random to share the rumor
• Everyone who hears the rumor becomes infective
• When A unnecessarily contacts someone A will
  become inactive (removed) with probability 1/k
• Increasing k insures almost everyone will hear
  the rumor

A
11
Some variations on rumor mongering exist

• Blind vs. Feedback
• Feedback can tell when a recipient has already
  heard a rumor
• Blind stops spreading the rumor with probability
  1/k regardless of whether recipient has already
  heard the rumor
• Counter vs. Coin
• Coin loses interest with probability 1/k
• Counter loses interest after k unnecessary
  contacts
• Simulations indicate that counter and feedback
  used in combination have the least delay

12
Pull performs better when updates are frequent

• Push vs. pull
• Up until now, have assumed that updates are
  pushed
• When a database has a high rate of rumor
  injection
• Pull more likely to find non-empty rumor lists
• When database is mostly quiescent
• Push will cease to introduce traffic
• Choice is based on the rate of updates
• Connection limits help push but hinder pull

13
Rumor mongering a better choice when using
anti-entropy as backup
Direct mail
Rumor mongering

• What happens when anti-entropy detects
  inconsistency?
• Nothing. Anti-entropy makes the databases
  consistent
• Ok when only a few sites were missed
• Update redistributed
• Better in the event of a complete failure
• Worst case distribution reached half the sites

14
Deletion is more complicated than simply removing
a file

• In anti-entropy and rumor mongering an absent
  file will be replaced by an old version
• Solution
• File replaced with death certificate
• Death certificates spread removing old copies of
  deleted items
• When and how do death certificates get deleted?

File A
Death Certificate A
15
Death certificates become dormant but can be
resurrected

• Death certificates are stamped with two
  timestamps T1 and T2
• When T1 is reached, most servers delete the
  certificate
• Servers on death certificates retention site
  list keep a dormant copy
• Dormant copies discarded when T1T2 is reached
• Dormant death certificates are resurrected if an
  obsolete copy of the data is encountered

Dormant certificate kept on A, B, and D
All certificates deleted
Death certificate
X
T1- 100 T2- 200 Retention List- A, B, D
T1- 100 T2- 200 Retention List- A, B, D
T1- 100 T2- 200 Retention List- A, B, D
Current time 100
Current time 200
Current time 1200
16
Timestamps tricky when reactivating a death
certificate

• Setting the timestamp forward to current clock
  value reactivates the death certificate
• Problem legitimate updates made between death
  certificate and current time will be erased
  erroneously
• An activation timestamp must be added to prevent
  the deletion of changes more current than the
  death certificate

Dormant certificate kept on A, B, and D
Certificate reactivated
Death certificate
Activation time 100 T1- 100 T2-
200 Retention List- A, B, D
Activation time 100 T1- 300 T2-
400 Retention List- A, B, D
Activation time 100 T1- 100 T2-
200 Retention List- A, B, D
Current time 100
Current time 200
Current time 1200
17
Distance between nodes can effect traffic overhead

• Updates cost less to send when the source and
  destination are close
• Assume a worst case linear network
• Nearest neighbor selection results in high
  convergence time
• Links per cycle would be O(1)
• O(n) cycles would be needed
• Uniform random connections result in high traffic
  overhead
• Average connection time of O(n)
• Convergence O(log n)
• Traffic per link per cycle is O(n)
• Nonuniform distribution reduces traffic and has
  acceptable convergence time

1
2
n

18
Spatial distribution can improve traffic in
anti-entropy

• Each site builds a list of sites sorted by
  distance
• An anti-entropy exchange partner is selected from
  the list according to some function f(i) i-a
• Spatial distribution significantly reduces
  traffic on critical links
• Convergence time is not significantly worse with
  a higher spatial distribution

19
Push and pull rumors more sensitive to spatial
distribution

• There is a high probability that S and T will
  chose each other
• If update introduced at S or T, will be pushed to
  the other
• Rumor will eventually die without reaching all
  other nodes

U1
S
U2
T

Um
20
Xerox chose to implement randomized anti-entropy
algorithm

• Anti-entropy guarantees consistency
• Well chosen spatial distribution algorithm
  reduced link traffic by factor of 4 and critical
  link traffic by 30
• Xerox experienced improvement in consistency and
  network traffic overhead with implementation

21
Agenda

• Consistency issues
• Epidemic algorithms
• Astrolabe
• Conclusion

22
Astrolabe provides fast, dynamic mgmt of large
stores of information

• DNS
• A directory service
• Organizes machines into domains
• Associates attributes with each domain
• Designed to map domain names to IP addresses and
  mail servers
• Changes rare
• Updates are slow to propagate

• Astrolabe
• An information management service
• Organizes resources into a hierarchy of zones,
  like domains
• Attributes associated with each zone
• Zones not bound to specific servers
• Attributes can be very dynamic
• Updates propagate quickly

23
Astrolabe can be used in p2p systems to cache
large objects

• Problem
• Infeasible to keep large objects on a central
  database and copy on every access
• Load time and network load too high
• Solution
• Store copies on different hosts
• Use Astrolabe to find a nearby, fresh copy

A
A
A
24
Astrolabe strives to satisfy four basic principles

• Scalability through hierarchy
• Maintains consistent overhead
• Flexibility through mobile code
• SQL queries allow different applications to
  communicate
• Robustness through a randomized peer-to-peer
  protocol
• Communicate by running a process on each host
• Epidemic protocol used
• Security through certificates
• Digital signatures used to allow or deny access
  to data, operations, etc.

25
Zone hierarchy makes Astrolabe scalable

• A zone is
• A host or a set of non-overlapping zones (no
  hosts in common)
• Tree structure
• Leaves are hosts
• Each zone (except root) has a local zone
  identifier
• Each zone has an attribute list (MIB)
• Attributes are generated by aggregation
  functions, summary of childrens attributes
• Leaf zones have writable virtual child zones used
  to populate attributes for that zone

MIB
Zone
Host
26
Aggregate functions are used to query the tree

• Aggregate functions summarize and are bounded in
  size
• Aggregate functions are programmable
• Code embedded in time-stamped aggregate function
  certificates (AFCs)
• AFCs stored as attributes in MIBs
• For every zone an agent is in, it scans hosts
  looking for childrens attributes, then
  aggregates results
• Zones learn about other zones through gossip
  protocol
• Applications invoke Astrolabe through calls to
  library

AFC
27
Agents on each host maintain a database of the
zone hierarchy

• Astrolabe agent runs on each host
• Each agent stores a subset of MIBs in the
  Astrolabe tree
• A copy of root MIB
• A copy of all MIBs of the roots children
• For each level a list of child zones (and
  attributes) is kept along with which child
  represents its own zone

Asia
Europe
self
USA
self
Cornell
MIT
pc1
pc2
self
pc3
pc4
self
system
inventory
monitor
28
Gossiping is an epidemic protocol used to
propagate information

• Periodically, an agent selects a zone in which to
  gossip
• Agent picks some child at random (other than its
  own) within that zone to gossip with
• Agent sends chosen child the id, rep, and issued
  attributes of all MIBs of all children at that
  level and up to the root
• Recipient can then tell which entries are out of
  date
• Updates are passed back and forth
• Note timestamps can be compared only if the
  attribute is issued by the same rep

29
Astrolabe allows members to be added or removed

• Member removal
• Each MIB knows which rep (agent) created it and
  when it was last updated
• When an agent has not seen an update for some
  zone from a rep for time Tfail, the MIB is
  removed
• When the last MIB for a zone is removed, the zone
  is also removed

• Member integration
• IP multicast sets up initial contact
• When two trees join, each tree multicasts a
  gossip message at a fixed rate
• Broadcasting gossip on local LAN is also used
• Astrolabe agents maintain a set of relatives who
  should be contacted on occasion

30
Certificates are used to guarantee security

• Each zone is allowed to override the security
  requirements of his parent zone
• Control zone creation, gossip rate, failure
  detection time-outs, introducing new AFCs, etc.
• Each zone has a Certificate Authority (CA) which
  issues certificates for that zone
• Zone certificate binds zoneID to its public key
• MIB certificate gossiped with zone certificate
  to propagate data between hosts
• Aggregate function certificate (AFC) contains
  code and other info for aggregation functions.
  Agent will only install AFCs issued by ancestor
  zones or by one of their clients.
• Client certificate authenticates a client.
  Astrolabe agents do not maintain a client
  database for scalability. If an ancestor signs
  the client certificate with its CA key, the
  client is trusted.

31
An AFC is introduced into the system through the
virtual children

• AFCs can be introduced by adding an attribute to
  the virtual child zone
• The agent will automatically evaluate the
  attribute
• AFCs can propagate by copying into the parent
  MIBs until they reach the root
• Adoption is used to propagate back down the tree
• Agents scan ancestor for new attributes
• New AFCs automatically copied
• For garbage collection, an expiration time can be
  specified

32
An AFC must meet certain security requirements to
propagate

• AFC must be signed by ancestor zone, or a client
  of that zone
• A client must have permission to propagate
• AFC cannot have expired
• The name of the AFC attribute and the category
  attribute must match
• Prevents a malicious client from introducing an
  AFC for a purpose other than advertised

33
Experiments demonstrate Astrolabes scalability

• Branch factor increases
• A higher branching factor leads to larger
  messages and more traffic
• Astrolabe remains scalable even with a high
  branch factor
• Loss rates
• A higher loss rate does not seriously affect
  scalability
• Due to the randomization algorithm

34
Agenda

• Consistency issues
• Epidemic algorithms
• Astrolabe
• Conclusion

35
Conclusion

• Scalability through hierarchy
• Zones enable scalability
• Flexibility through mobile code
• AFCs can be generated by one agent and the
  propagated throughout to learn the attributes on
  a variety of hosts
• Robustness through a randomized p2p protocol
• Zones select other zones at random and propagate
  MIB of least common ancestor
• Guarantees changes will eventually reach the
  entire system
• Security through certificates
• Certificates authenticate every level of
  communication
• Conclusion Astrolabe is a scalable, robust
  system which allows changes to propagate quickly
  and guarantees eventual consistency

36
Backup
37
Astrolabe improves upon several previous systems
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)