Peertopeer computing research: a fad

1
Peer-to-peer computing researcha fad?

• Frans Kaashoek
• kaashoek_at_lcs.mit.edu
• Joint work with H. Balakrishnan, P. Druschel ,
  J. Hellerstein , D. Karger, R. Karp, J.
  Kubiatowicz, B. Liskov, D. Mazières, R. Morris,
  S. Shenker, and I. Stoica

2
What is a P2P system?
Node
Node
Node
Internet
Node
Node

• A distributed system architecture
• No centralized control
• Nodes are symmetric in function
• Larger number of unreliable nodes
• Enabled by technology improvements

3
P2P an exciting social development

• Internet users cooperating to share, for example,
  music files
• Napster, Gnutella, Morpheus, KaZaA, etc.
• Lots of attention from the popular press
• The ultimate form of democracy on the Internet
• The ultimate threat to copy-right protection on
  the Internet

4
How to build critical services?

• Many critical services use Internet
• Hospitals, government agencies, etc.
• These services need to be robust
• Node and communication failures
• Load fluctuations (e.g., flash crowds)
• Attacks (including DDoS)

5
The promise of P2P computing

• Reliability no central point of failure
• Many replicas
• Geographic distribution
• High capacity through parallelism
• Many disks
• Many network connections
• Many CPUs
• Automatic configuration
• Useful in public and proprietary settings

6
Traditional distributed computingclient/server
Server
Client
Client
Internet
Client
Client

• Successful architecture, and will continue to be
  so
• Tremendous engineering necessary to make server
  farms scalable and robust

7
Application-level overlays
Site 3
Site 2
N
N
N
ISP1
ISP2
Site 1
N
N
ISP3

• One per application
• Nodes are decentralized
• NOC is centralized

Site 4
N
P2P systems are overlay networks without central
control
8
Distributed hash table (DHT)
(File sharing)
Distributed application
data
get (key)
put(key, data)
(DHash)
Distributed hash table
lookup(key)
node IP address
(Chord)
Lookup service

• Application may be distributed over many nodes
• DHT distributes data storage over many nodes

9
A DHT has a good interface

• Put(key, value) and get(key) ? value
• Simple interface!
• API supports a wide range of applications
• DHT imposes no structure/meaning on keys
• Key/value pairs are persistent and global
• Can store keys in other DHT values
• And thus build complex data structures

10
A DHT makes a good shared infrastructure

• Many applications can share one DHT service
• Much as applications share the Internet
• Eases deployment of new applications
• Pools resources from many participants
• Efficient due to statistical multiplexing
• Fault-tolerant due to geographic distribution

11
Many recent DHT-based projects

• File sharing CFS, OceanStore, PAST, Ivy,
• Web cache Squirrel, ..
• Backup store Pastiche
• Censor-resistant stores Eternity, FreeNet,..
• DB query and indexing Hellerstein,
• Event notification Scribe
• Naming systems ChordDNS, Twine, ..
• Communication primitives I3,

Common thread data is location-independent
12
CFS Cooperative file sharing
File system
block
get (key)
put (key, block)
Distributed hash tables
.
node
node
node

• DHT is a robust block store
• Client of DHT implements file system
• Read-only CFS, PAST
• Read-write OceanStore, Ivy

13
File representationself-authenticating data
File System key995
431SHA-1
144 SHA-1
901 SHA-1

995 key901 key732 Signature
key431 key795
a.txt ID144

(i-node block)

(data)
(root block)
(directory blocks)

• Key SHA-1(content block)
• File and file systems form Merkle hash trees

14
DHT distributes blocks by hashing IDs
Block 732
Block 705
Node B
995 key901 key732 Signature
247 key407 key992 key705 Signature
Node A
Internet
Block 407
Node C
Node D
Block 901
Block 992

• DHT replicates blocks for fault tolerance
• DHT caches popular blocks for load balance

15
Historical web archiver

• Goal make and archive a daily check point of the
  Web
• Estimates
• Web is about 57 Tbyte, compressed HTMLimg
• New data per day 580 Gbyte
• 128 Tbyte per year with 5 replicas
• Design
• 12,810 nodes 100 Gbyte disk each and 61 Kbit/s
  per node

16
Implementation using DHT
Crawler
Client
put(sha-1(URL), page)
get (URL)
Distributed hash tables
.
node
node
node

• DHT usage
• Crawler distributes crawling and storage load by
  hash(URL)
• Client retrieve Web pages by hash(URL)
• DHT replicates data for fault tolerance

17
Backup store

• Goal backup on other users machines
• Observations
• Many user machines are not backed up
• Backup requires significant manual effort
• Many machines have lots of spare disk space
• Using DHT
• Merkle tree to validate integrity of data
• Administrative and financial costs are less for
  all participants
• Backups are robust (automatic off-site backups)
• Blocks are stored once, if key sha1(data)

18
DHT implementation challenges

• Scalable lookup
• Balance load (flash crowds)
• Handling failures
• Coping with systems in flux
• Network-awareness for performance
• Robustness with untrusted participants
• Programming abstraction
• Heterogeneity
• Anonymity
• Indexing
• Goal simple, provably-good algorithms

this talk
19
1. The lookup problem
N2
N1
N3
Internet
Put (Keysha-1(data), Valuedata)
?
Client
Publisher
Get(keysha-1(data))
N6
N4
N5

• Get() is a lookup followed by check
• Put() is a lookup followed by a store

20
Centralized lookup (Napster)
N2
N1
SetLoc(title, N4)
N3
Client
DB
N4
Publisher_at_
Lookup(title)
Keytitle Valuefile data
N8
N9
N7
N6
Simple, but O(N) state and a single point of
failure
21
Flooded queries (Gnutella)
N2
N1
Lookup(title)
N3
Client
N4
Publisher_at_
Keytitle ValueMP3 data
N6
N8
N7
N9
Robust, but worst case O(N) messages per lookup
22
Algorithms based on routing

• Map keys to nodes in a load-balanced way
• Hash keys and nodes into a string of digit
• Assign key to closest node

• Forward a lookup for a key to a closer node

• Join insert node in ring

Examples CAN, Chord, Kademlia, Pastry, Tapestry,
Viceroy, .
23
Chords routing table fingers
½
¼
1/8
1/16
1/32
1/64
1/128
N80
24
Lookups take O(log(N)) hops
N5
N10
N110
K19
N20
N99
N32
Lookup(K19)
N80
N60

• Lookup route to closest predecessor

25
CAN exploit d dimensions

• Each node is assigned a zone
• Nodes are identified by zone boundaries
• Join chose random point, split its zone

26
Routing in 2-dimensions
(0,1)
(0.5,0.5, 1, 1)


(0,0.5, 0.5, 1)
(0.5,0.25, 0.75, 0.5)



(0.75,0, 1, 0.5)
(0,0, 0.5, 0.5)
(0,0)
(1,0)

• Routing is navigating a d-dimensional ID space
• Route to closest neighbor in direction of
  destination
• Routing table contains O(d) neighbors
• Number of hops is O(dN1/d)

27
2. Balance load
N5
K19
N10
N110
K19
N20
N99
N32
Lookup(K19)
N80
N60

• Hash function balances keys over nodes
• For popular keys, cache along the path

28
Why Caching Works Well
N20

• Only O(log N) nodes have fingers pointing to N20
• This limits the single-block load on N20

29
3. Handling failures redundancy
N5
N10
N110
N20
N99
N32
N40
N80
N60

• Each node knows IP addresses of next r nodes
• Each key is replicated at next r nodes

30
Lookups find replicas
N5
N10
N110
3.
N20
1.
2.
N99
K19
N40
4.
N50
N80
N60
N68
Lookup(K19)

• Tradeoff between latency and bandwidth Kademlia

31
4. Systems in flux

• Lookup takes log(N) hops
• If system is stable
• But, system is never stable!
• What we desire are theorems of the type
• In the almost-ideal state, .log(N)
• System maintains almost-ideal state as nodes join
  and fail

32
Half-life Liben-Nowell 2002
N new nodes join
N nodes
N/2 old nodes leave

• Doubling time time for N joins
• Halfing time time for N/2 old nodes to fail
• Half life MIN(doubling-time, halfing-time)

33
Applying half life

• For any node u in any P2P networks
• If u wishes to stay connected with high
  probability,
• then, on average, u must be notified about ?(log
  N) new nodes per half life
• And so on,

34
5. Optimize routing to reduce latency
N20
N40
N41
N80

• Nodes close on ring, but far away in Internet
• Goal put nodes in routing table that result in
  few hops and low latency

35
close metric impacts choice of nearby nodes
N06
USA
N105
USA
K104
Far east
N32
N103
Europe
N60
USA

• Chords numerical close and table restrict choice
• Prefix-based allows for choice
• Kademlias offers choice in nodes and places
  nodes in absolute order close (a,b) XOR(a, b)

36
Neighbor set
N06
USA
USA
N105
K104
N32
N103
Far east
Europe
N60
USA

• From k nodes, insert nearest node with
  appropriate prefix in routing table
• Assumption triangle inequality holds

37
Finding k near neighbors

• Ping random nodes
• Swap neighbor sets with neighbors
• Combine with random pings to explore
• Provably-good algorithm to find nearby neighbors
  based on sampling Karger and Ruhl 02

38
6. Malicious participants

• Attacker denies service
• Flood DHT with data
• Attacker returns incorrect data detectable
• Self-authenticating data
• Attacker denies data exists liveness
• Bad node is responsible, but says no
• Bad node supplies incorrect routing info
• Bad nodes make a bad ring, and good node joins it

Basic approach use redundancy
39
Sybil attack Douceur 02
N5

• Attacker creates multiple identities
• Attacker controls enough nodes to foil the
  redundancy

N10
N110
N20
N99
N32
N40
N80
N60

• Need a way to control creation of node IDs

40
One solution secure node IDs

• Every node has a public key
• Certificate authority signs public key of good
  nodes
• Every node signs and verifies messages
• Quotas per publisher

41
Another solutionexploit practical byzantine
protocols
N06
N105
N
N
N
N32
N103
N
N60

• A core set of servers is pre-configured with keys
  and perform admission control
• The servers achieve consensus with a practical
  byzantine recovery protocol Castro and Liskov
  99 and 00
• The servers serialize updates OceanStore or
  assign secure node Ids Configuration service

42
A more decentralized solutionweak secure node
IDs

• ID SHA-1 (IP-address node)
• Assumption attacker controls limited IP
  addresses
• Before using a node, challenge it to verify its ID

43
Using weak secure node IDS

• Detect malicious nodes
• Define verifiable system properties
• Each node has a successor
• Data is stored at its successor
• Allow querier to observe lookup progress
• Each hop should bring the query closer
• Cross check routing tables with random queries
• Recovery assume limited number of bad nodes
• Quota per node ID

44
Philosophical questions

• How decentralized should systems be?
• Gnutella versus content distribution network
• Have a bit of both? (e.g., OceanStore)
• Why does the distributed systems community have
  more problems with decentralized systems than the
  networking community?
• A distributed system is a system in which a
  computer you dont know about renders your own
  computer unusable
• Internet (BGP, NetNews)

45
What are we doing at MIT?

• Building a system based on Chord
• Applications CFS, Herodotus, Melody, Backup
  store,Ivy,
• Collaborate with other institutions
• P2P workshop, PlanetLab
• Big ITR
• Building a large-scale testbed
• RON, PlanetLab

46
Summary

• Once we have DHTs, building large-scale,
  distributed applications is easy
• Single, shared infrastructure for many
  applications
• Robust in the face of failures and attacks
• Scalable to large number of servers
• Self configuring across administrative domains
• Easy to program
• Lets build DHTs . stay tuned .
• http//project-iris.net

47
7. Programming abstraction

• Blocks versus files
• Database queries (join, etc.)
• Mutable data (writers)
• Atomicity of DHT operations