Zurich, Switzerland

1
SP3 3L Overlay Services3M Content
distribution and storage Danny Bickson, HUJI

• Zurich, Switzerland
• March 5-6, 2007

2
Partners, Effort Output WP3L

• EPFL, TUC, UCL
• Deliverables D.3l.2 (Second report on
  higher-level services on top of overlay
  networks)
• 6 papers at various publication stages
  (submitted, under revision, accepted)
• Implementations DHTrie, MAPS, R-Join, P-Grid,
  P2PS, P2PKit

3
Services and Overlay Systems

• Main goals
• Develop services for large-scale networks by
  building on top of DHTs
• Develop a service-oriented architecture based on
  DHTs and components

4
WP3l Contribution to Evergrow Goals
Information Retrieval and Filtering Protocols
DHTrie and MAPS Continuous DHT-based multi-way
join algorithm RJoin
Efficient belief propagation algorithms for
Bayesian Networks stored on DHTs A generic
belief-porpagation service implementation in
P2PKit
Large Scale Distributed Application Development
Using P2PS and P2PKit
5
Overview

• Connections with last year
• The DHTrie protocols
• Information Retrieval and Filtering on top of
  DHTs
• Continuous relational query processing on top of
  DHTs
• Algorithms for two-way joins
• The development of DHT P2PS and service-oriented
  infrastructure P2PKit
• Main new contributions (reported this year)
• The DHTrie protocols
• Identify a latency vs. network-traffic tradeoff
  in DHTrie and propose a tunable resource
  publication algorithm for its resolution.
• Develop MAPS an approximate information
  filtering service
• Continuous relational query processing on top of
  DHTs
• Algorithms for multi-way joins
• The development of DHT P2PS and service-oriented
  infrastructure P2PKit
• Extend the P2PS implementation with atomic
  join/leave algorithms
• Develop efficient BP algorithms on top of P-Grid
  and P2PS.

6
DHTrie and MAPS The Setting

7
Network traffic vs. latency tradeoff in DHTrie

8
MAPS Approximate Information Filtering

• Predict publication behavior of peers based on
  time-series analysis
• High recall is achieved although we are
  monitoring only a few (most promising) peers
• Joint work with DELIS project

9
P2PS and P2PKit

• Improvement of join/leave algorithms for ring
  maintenance of P2PS making the system more robust
  and fault tolerant
• Generic Belief Propagation Service developed with
  P2PKit on top of P2PS
• K-partition of the network for data indexing
  using Belief Propagation, providing load-balanced
  data retrieval

10
Example collaboration
Graph coloring work (SP4 Aston)
Input from SP3
Network storage of file parts (SP4 Aston)
Data Indexing (SP3 HUJI EPFL)
BP of monitoring churn (SP3 HUJI UCL)
BP Service (SP3 UCL TUC)
11
Belief Propagation on P-Grid

• Bayesian network on P-Grid
• Spring relaxation
• physics-inspired approach for CS algorithms.
• Correlated data are placed close for efficiency
• Minimum energy configuration
• Variable clustering
• Reduced communication cost
• Trade-off with load-balance
• Investigated networks
• Trees, scale-free, random

12
Future plans

• The protocols DHTrie and MAPS
• Continue analysis and experimental evaluation
• Develop prototype. Demo in the case of a digital
  library scenario.
• Continuous relational query processing on top of
  DHTs
• Continue analysis and experimental evaluation
• The development of DHT P2PS and service-oriented
  infrastructure P2PKit
• Develop self-management services in P2PKit
• Efficient BP algorithms
• Implement BP on top of P-Grid. Evaluate. Demo in
  the case of a distributed software development
  scenario.

13
Partners, Effort Output WP3M

• Main topic Content distribution and storage
• HUJI, SICS, KTH, Aston
• Deliverables D.3l.M (Final report on storage and
  content distribution)
• 7 papers at various publication stages
  (submitted, under revision, accepted)
• Implementations Julia, MyriadStore

14
Contribution to Evergrow Goals
Julia content distribution network (bulk data and
streaming) Epidemic algorithms for data
replication
Myriad Store P2P storage Data replication using BP
15
Storage, Content distributionand P2P analysis

• Main goals
• Ultimate AKAMAI content distribution
  (bulk-data and streaming media) using Julia
• Ultimate Raid distributed storage using
  MyriadStore
• Common approach utilize complex system
  algorithms for supporting future networks
• Belief propagation, epidemic algorithms

16
Overview

• Work in progress (reported last year)
• The Julia content delivery network
• Extending the support for real time media
  streaming
• Gossip (epidemic) algorithms for data replication
• Myriad Store P2P storage
• New work (reported this year)
• Clustering for sensor networks using BP
• Using network coding for data dissemination
• Real time monitoring using BP
• Rating data items in social networks

17
Performance of P2P streaming media

• 1000 nodes, GT-ITM topology, streaming file with
  500 segments
• RDP delay experienced relative to the optimal
  transmission time

18
Performance of P2P rating algorithm

• 1M nodes, MSNMessenger social network topology,
  15M edges
• Y-scale indicates convergence (logarithmic scale)

19
2006/2007 Themes
BP for data correlation, localization, IR and
IF Global properties estimation Analysis of P2P
networks for improving reliability and fast
recovery
Message passing techniques The Julia content
delivery network Gossip algorithms Data
replication using BP algorithm
Self-managing replicated storage and
backup Distributed backup on top of DKS
20
Future plans - tomorrow

• Work package 3m utilizing complex systems
  techniques to optimize content distribution

21
Collaborations

• Internal
• Several meetings (Jan 06, Feb 06, June 06, Dec.
  06, Feb 07)
• Cross project
• Cooperation with SP1 DIMES for creating clients
  for Linux, MacOS and mobile platforms, ETOMIC for
  programmable NICs.
• Support for the EverLab cluster SP2.
• Ongoing discussions with SP4 regarding message
  passing algorithms
• Other projects
• Participated in PlanetLab meetings
• Meetings with the industry IBM / Motorola/
  Microsoft
• Other EU projects Selfman EU project, Grid4All
  EU project, CoreGRID network of excellence, DELIS
  MPI

22
Collaborations

• Cross project
• Using the EverLab cluster (SP2) for experiments
• Ongoing discussions with SP4 regarding message
  passing algorithms
• Other projects
• TUC - MPI Saarbrucken (IP DELIS) joint
  development of MAPS (ADPUC 2006 paper) and
  comparison with DHTrie (submitted paper)
• Participated in PlanetLab meetings
• Other EU projects Selfman, Grid4All, CoreGRID,
  OntoGrid