Resource Allocation in OpenHash: a Public DHT Service

1
Resource Allocation in OpenHash a Public DHT
Service

• Sean Rhea
• with Brad Karp, Sylvia Ratnasamy,
• Scott Shenker, Ion Stoica, and Harlan Yu

2
Introduction

• In building OceanStore, worked on Tapestry
• Found Tapestry problem harder than expected
• Main problem handling churn
• Built Bamboo to be churn-resilient from start
• Working by 6/2003
• Rejected from NSDI in 9/2003
• Released code in 12/2003, about 10 groups using
• Accepted to USENIX, 6/2004

3
Introduction (cont.)

• Intended to Bamboo to be general, reusable
• Supports Common API for DHTs
• Tens of DHT applications proposed in literature
• Still very few in common use--why?
• One possible barrier deployment
• Need access to machines, not everyone on PL
• Must monitor, restart individual processes
• Takes about an hour/day minimum right now

4
Simple DHT Applications

• Many uses of DHTs very simple just put/get
• Dont use Common API Dabek et al.
• No routing, no upcalls, etc.
• Examples
• Dynamic DNS
• FreeDB
• In general use DHT as highly available cache or
  rendezvous service
• Should be able to share a single DHT deployment

5
Sophisticated DHT Applications

• Other functionality of DHTs is lookup
• Map identifiers to application nodes efficiently
• Used by most sophisticated applications
• i3, OceanStore, SplitStream
• Can implement lookup on put/get
• Algorithm called ReDiR, IPTPS paper this year
• Sophisticated applications could also share a
  single DHT deployment

6
OpenHash a Public DHT Service

• Idea Public DHT to amortize deployment effort
• Very low barrier to entry for simple applications
• Amortize bandwidth cost for sophisticated apps
• Challenges
• Economics
• Security
• Resource Allocation

7
Overview

• Introduction
• OpenHash interface, assumptions
• Resource allocation
• Goals/Problem formalization
• Rate-limiting puts
• Fair sharing
• Discussion

8
OpenHash Interface/Assumptions

• Want to keep things simple for clients
• Remember goal low barrier to entry
• Simple put/get
• put (key, value, time-to-live)
• get (key)
• Service contract
• Puts accepted/rejected immediately (not queued)
• Once accepted, put values available for whole TTL
• Predictable, zero-effort availability for clients
• After that, will be thrown out by DHT
• Easy garbage collection, also valuable for some
  apps

9
Resource Allocation Introduction

• Problem disk space is limited
• If service popular, may exhaust
• Malicious clients might exhaust on purpose
• Rough goal every client gets fair share of store
• Ideally, algorithm should be work-conserving
• Example
• Three clients A, B, and C 10 GB of total space
• A and B want 1 GB each, C wants 20 GB
• A and B should get 1 GB each C should get 8 GB

10
Problem Simplification

• For now, shares calculated per-DHT node
• Global fair sharing saved for future work
• Clients that balance puts wont notice a problem
• Most DHT applications already balance puts
• Apps that can choose their keys can do even
  better
• Side benefit encourages balancing puts
• Mitigates need for load balancing in DHT
• Let the users handle load balancing
• Easier for us to implement!

11
Problem Formalization (First Try)

• C - total available storage
• si - storage desired by client i, S ? si
• sfair - fair share such that
• C ? min(si, sfair)
• gi - storage granted to client i, G - ? gi
• Goals
• Fairness ? i gi min(si, sfair)
• Utilization G min(C, S)

12
Problem Formulation (Second Try)

• Previous version didnt account for time
• Can only remove stored values as TTLs expire
• As such, can only adapt so quickly
• Before accepting one put, another must expire
• Add goal always accept puts at rate ? R
• Prefer puts from underrepresented clients
• Intuition R bounds time it takes to correct
  unfairness
• New questions
• How to guarantee space frees up at rate ? R?
• How to divide R among clients?

13
Overview

• Introduction
• OpenHash interface, assumptions
• Resource allocation
• Goals/Problem formalization
• Rate-limiting puts
• Fair sharing
• Discussion

14
Accepting At Rate ? R

• S(t) - total data stored at time t
• A(t1, t2) - data added to system in t1, t2)
• D(t1, t2) - data freed in t1, t2)
• For adaptivity, need A(t, t?t) ? R ? ?t
• Capacity limit S(t) A(t, t?t) - D(t, t?t) ?
  C
• Rearrange C D(t, t?t) - S(t) ? A(t, t?t)
• Combined with top eqn C D(t, t?t) - S(t) ? R
  ? ?t
• Rearrange D(t, t?t) ? R ? ?t - C S(t)
• Result can accept any put that wont make us
  violate this equation at any point in the future

15
Implementing Rate Limiting

• Before accepting put, must check D(t, t?t)
• Can we check this efficiently?
• Easy, assuming all puts have same TTL
• Can implement using a virtual pipe
• Pipe is TTL long, total capacity C
• New puts go into pipe, expire on exit
• Can easily show pipe is optimal for this case
• With varying TTLs, problem harder
• Puts with short TTLs expire in middle of pipe
• Bin-packing problem on new puts find latest spot
  in pipe that satisfies desired size and TTL

16
Overview

• Introduction
• OpenHash interface, assumptions
• Resource allocation
• Goals/Problem formalization
• Rate-limiting puts
• Fair sharing
• Discussion

17
Choosing Puts for Fair Sharing

• Assume can accept new puts at rate ? R
• How do we divide it up between clients?
• Unlike fair queuing, two competing goals
• Want to make decisions (put/reject) quickly
• In FQ, may queue for a long time before fowarding
• Suffer consequences of decisions for full TTL
• In FQ, only interested in fairness over short
  window
• But one big advantage long history
• Remember all puts whose TTLs havent expired

18
The Rate-Based Approach

• Accept based on recent put rates
• Already storing all puts, so also store rates
• (Could estimate these as in Approx. Fair Drop.)
• Basically, fair share the input rate R
• Pros
• Easy to implement
• If all clients put at uniform rates, gives fair
  stores
• Cons
• To get fair share, must put at uniform rate
• What about bursty clients (avg. rate ltlt max.
  rate)?

19
The Storage-Based Approach

• Accept puts based on amount of storage used
• Keep counters of storage used by each client
• Prefer new puts from clients with less data on
  disk
• Pros
• Also easy to implement
• Gives fair stores regardless of uniformity of
  client put rates
• Cons
• Over-represented clients block on
  under-represented ones
• Could be very disruptive as new clients enter
  system

20
The Commitment-Based Approach

• Base fairness around commitments
• How many bytes stored for how much more time
• New bytes entail more future commitment than old
• Pros
• Better at bursts than rate-based approach
• Better at not blocking over-represented clients
  than storage-based approach
• Cons
• Hard to think about in detail, hard to implement?

21
Related Work

• Various fair queuing techniques
• Standard FQ
• Approximate Fair Dropping
• CSFQ
• Other DHT work
• Palimpsest
• Other networking work
• Internet backplane

22
Discussion

• What is the optimal rate limiting algorithm?
• How close to our various schemes come to it?
• Whats the right model for sharing?
• Rate-based approach?
• Storage-based approach?
• Commitment-based approach?
• Some hybrid?
• Lottery Scheduling?
• What other models make sense?
• Palimpsest?