CS176 Spring 2006 Structured Overlay Applications II

1
CS176 Spring 2006Structured Overlay Applications
II

• Administrivia
• Reminder homework 1 due Friday night,
  midnightTurn in as Zip file via email to
  krishnap_at_cs.ucsb.edu
• Think about homework 2 (coming soon)
• Think about project topicsProject will take more
  time, need to overlap w/ hw2
• Today
• Project Ideas
• Another detailed example of structured overlay
  application MulticastCase study Scribe
• Reminder
• All lecture material is fair game for the
  Midtermespecially material available in the
  assigned reading

2
Project Requirements

• Substantial in scope
• Produce a concrete software artifact
• Preferably useful to others as building block
• Evaluation
• Project specification (30 of project)
• What is it supposed to do?
• What are the pieces, which ones will be done?
• How well is it supposed to perform?
• Does it accomplish what you set out to do (70 of
  project)
• Does it work?
• Is it feature/component complete?
• Is it well documented and usable/extensible by
  others?
• How does it perform?
• Best grades will be for inventive, yet highly
  usable projects

3
Project Ideas

• Enhancements to Chimera
• Build a high-performance DHT layer on Chimera
  (XL)
• Integrating Chimera with network sensors (XL)
• Implement a working version of Scribe on Chimera
• Improving Chimera throughput through code
  optimization
• Implementing Chimera Applications
• A fully decentralized resilient chat client
• A Peer-to-Peer Visualization library (XL)
• Take logs from Chimera, synch by timestamp,
  visualize
• Automated tools for a persistent Chimera
• Implement tools to maintain Chimera liveness on
  PL
• A benchmark for structured overlay networks

4
Review the P2P APIs?

• Additional services on top of Chimera
• What are the APIs for overlay applications?
• Distributed storage on Chimera (DHT)Treat
  machines like buckets in a hash tablehash (file)
  5, so store it on machine 5
• Distributed yellow pages (DOLR)Store your file
  or your service whereverJust tell everyone about
  location (file) ME

5
Internet Media Delivery

• Last time, we talked about distributed storage
• OceanStore
• Many similar applications disk backup, large
  content distribution etc.
• Another class of applications media delivery
• Streaming video on the Internet
• VS fashion shows
• Broadcast events live, or delayed
• Video on demand (VoD) systems

6
The Challenge

• Limited bandwidth at the server
• No single server has enough b/w
• Large potential audience for video stream
• The general approach
• Organize viewers into multicast tree
• Original server acts as root of tree
• Each parent copies stream data to children
• Application-level Multicast
• IP-multicast is difficult to deploy, (changes to
  routers etc)
• ALM is easier, the listeners participate

7
Scribe API

• Scribe is a scalable application-level multicast
  infrastructure built on top of Pastry
• create (credentials, group-id)
• create a group with the group-id
• join (credentials, group-id, message-handler)
• join a group with group-id.
• Published messages for the group are passed to
  the message handler
• leave (credentials, group-id)
• leave a group with group-id
• multicast (credentials, group-id, message)
• publish the message within the group with
  group-id
• credentials are used throughout for access
  control.

8
Scribe System

• A Scribe node
• May create a group
• May join a group
• May be the root of a multicast tree
• May act as a multicast source
• Scribe messages
• CREATE
• JOIN
• LEAVE
• MULTICAST
• publish a message to the group

9
Scribe Groups

• A Scribe group per session
• Unique group-id
• Multicast tree to disseminate messages
• Root of tree rendezvous point
• GroupID
• Is the hash of the groups textual name
  concatenated with its creators name
• Create Group
• Send a CREATE message with the group-id as the
  key
• Pastry delivers message to root (key)
• This node becomes the rendezvous point
• deliver method checks and stores credentials and
  also updates the list of groups

10
Join Group

• Join Group
• Send JOIN message with group-id as key
• Pastry routes to rendezvous point
• If intermediate node is forwarder
• Add the node as its child
• If intermediate node is not a forwarder
• Creates child table for the group, and adds the
  node
• Sends a JOIN towards the rendezvous point.
• Terminates JOIN message from the child

new node
new node
root
11
Multicast Message

• Multicast a message to the group
• Scribe node sends MULTICAST message to the
  rendezvous point
• A node caches the IP address of the rendezvous
  point so that it does not need Pastry for
  subsequent messages
• Single multicast tree for each group
• Access control for a message is performed at the
  rendezvous point

member
sender
root
member
12
Leave Group

• Leave Group
• Scribe node records locally that it left the
  group
• If the node has no children in its table, it
  sends a LEAVE message to its parent
• The message travels recursively up the multicast
  tree
• The message stops at a node which has children
  after removing the departing node
• What if my parent leaves?

root
13
Multicast Tree Repair I

• Broken link detection and repair
• Non-leaf nodes send heartbeat message to children
• Multicast messages serve as implicit heartbeat
• If child does not receive heartbeat message
• assumes that the parent has failed
• finds a new route by sending a JOIN message to
  the group-id, thus finding a new parent and
  repairing the multicast tree

root
14
Multicast Tree Repair II

• Rendezvous point failure
• The state associated with a rendezvous point is
  replicated across k closest nodes
• When the root fails, the children detect the
  failure and send a JOIN message which gets routed
  to a new node-id numerically closest to the
  group-id
• Fault detection and recovery is local and
  accomplished by sending minimal messages

15
Bottleneck Remover

• All nodes may not have equal capacity in terms of
  computational power and bandwidth
• Under high load conditions, the lower capacity
  nodes become bottlenecks
• Solution Offload children to other nodes
• Choose the group that uses the most resources
• Choose a child of this group that is farthest
  away
• Ask the child to join its sibling which is
  closest in terms of delay
• This gives an improved performance
• Increases link stress for joining