CoBlitz: A Scalable Largefile Transfer Service COS 461

1
CoBlitz A Scalable Large-file Transfer Service
(COS 461)

• KyoungSoo Park
• Princeton University

2
Large-file Distribution

• Increasing demand for large files
• Movies or software release
• On-line movie/ downloads
• Linux distribution
• Files are 100MB tens of GB
• One-to-many downloads
• How to serve large files to many clients?
• Content Distribution Network(CDN)?
• Peer-to-peer system?

3
What CDNs Are Optimized For
Most Web files are small (1KB 100KB)
4
Why Not Web CDNs?

• Whole file caching in participating proxy
• Optimized for 10KB objects
• 2GB 200,000 x 10KB
• Memory pressure
• Working sets do not fit in memory
• Disk access is 1000 times slower
• Waste of resources
• More servers needed
• Provisioning is a must

5
Peer-to-Peer?

• BitTorrent takes up 30 Internet BW

1. Download a torrent file 2. Contact the
tracker 3. Enter the swarm network 4. Chunk
exchange policy - Rarest chunk first or
random - Tit-for-tat incentive to upload
- Optimistic unchoking 5. Validate the checksums
up
down
peers
torrent
tracker
Benefit extremely good use of resources!
6
Peer-to-Peer?

• Custom software
• Deployment is a must
• Configurations needed
• Companies may want managed service
• Handles flash crowds
• Handles long-lived objects
• Performance problem
• Hard to guarantee the service quality
• Others are discussed later

7
What Wed Like Is
Large-file service with No custom client No
custom server No prepositioning No rehosting No
manual provisoning
8
CoBlitz Scalable Large-file CDN

• Reducing the problem to small-file CDN
• Split large-files into chunks
• Distribute chunks at proxies
• Aggregate memory/cache
• HTTP needs no deployment
• Benefits
• Faster than BitTorrent by 55-86 (500)
• One copy from origin serves 43-55 nodes
• Incremental build on existing CDNs

9
How It Works
Only reverse proxy(CDN) caches the chunks!
CDN Redirector Reverse Proxy
DNS
chunk1
chunk2
CDN
CDN
chunk3
Agent
CDN
Client
CDN
Client
Agent
CDN
CDN
chunk4
chunk5
10
Smart Agent

• Preserves HTTP semantics
• Parallel chunk requests

CDN
sliding window of chunks
CDN
done
HTTP
Client
CDN
CDN
waiting
waiting
done
CDN
done
no action
waiting
no action
waiting
no action
waiting
Agent
11
Chunk Indexing Consistent Hashing
Problem How to find the node responsible for a
specific chunk?
Static hashing f(x) some_f(x) n But n is
dynamic for servers - node can go down -
new node can join
N-1 0
X1
X3
CDN node (proxy)
Consistent Hashing F(x) some_F(x) N (N is
a large but fixed number) Find a live node k,
where F(k) F(URL) is minimum
Xk Chunk request
X2
12
Operation Challenges

• Provides public service over 2.5 years
• http//coblitz.codeen.org3125/URL
• Challenges
• Scalability robustness
• Peering set difference
• Load to the origin server

13
Unilateral Peering

• Independent proximity-aware peering
• Pick n close nodes around me
• Cf. BitTorrent picks n nodes randomly
• Motivation
• Partial network connectivity
• Internet2, CANARIE nodes
• Routing disruption
• Isolated nodes
• Benefits
• No synchronized maintenance problem
• Improve both scalability robustness

14
Peering Set Difference

• No perfect clustering by design
• Assumption
• Close nodes shares common peers

15
Peering Set Difference

• Highly variable App-level RTTs
• 10 x times variance than ICMP
• High rate of change in peer set
• Close nodes share less than 50
• Low cache hit
• Low memory utility
• Excessive load to the origin

16
Peering Set Difference

• How to fix?
• Avg RTT ? min RTT
• Increase of samples
• Increase of peers
• Hysteresis
• Close nodes share more than 90

17
Reducing Origin Load
Origin server

• Still have peering set difference
• Critical in traffic to origin
• Proximity-based routing
• Converge exponentially fast
• 3-15 do one more hop
• Implicit overlay tree
• Result
• Origin load reduction by 5x

Rerun hashing
18
Scale Experiments

• Use all live PlanetLab nodes as clients
• 380400 live nodes at any time
• Simultaneous fetch of 50MB file
• Test scenarios
• Direct
• BitTorrent Total/Core
• CoBlitz uncached/cached/staggered
• Out-of-order numbers in paper

19
Throughput Distribution
1
0.9
BT-Core
0.8
Out-of-order staggered
0.7
0.6
Fraction of Nodes 0.5
Direct
0.4
BT
-
total
0.3
BT
-
core
In
-
order uncached
0.2
In
-
order staggered
0.1
In
-
order cached
0
0
2000
4000
6000
8000
10000
Throughput(Kbps)
20
Downloading Times
21
Why Is BitTorrent Slow?

• In the experiments
• No locality randomly choose peers
• Chunk indexing extra communication
• Trackerless BitTorrent Kademlia DHT
• In practice
• Upload capacity of typical peers is low
• 10 to a few 100 Kbps for cable/DSL users
• Tit for tat may not be fair
• A few high-capacity uploaders help the most
• BitTyrantNSDI07

22
Synchronized Workload Congestion
Origin Server
23
Addressing Congestion

• Proximity-based multi-hop routing
• Overlay tree for each chunk
• Dynamic chunk-window resizing
• Increase by 1/log(x), (where x is win size) if
  chunk finishes
• Decrease by 1 if retry kills the first chunk

24
Number of Failures
25
Performance After Flash Crowds
26
Data Reuse
7 fetches for 400 nodes, 98 cache hit
27
Real-world Usage

• 1-2 Terabytes/day
• Fedora Core official mirror
• US-East/West, England, Germany, Korea, Japan
• CiteSeer repository (50,000 links)
• University Channel (podcast/video)
• Public lecture distribution by PU OIT
• Popular game patch distribution
• PlanetLab researchers
• Stork(U of Arizona) 10 others

28
Fedora Core 6 Release

• October 24th, 2006
• Peak Throughput 1.44Gbps

Release point 10am
1 G
Origin Server 30-40Mbps
29
On Fedora Core Mirror List

• Many people complained about I/O
• Performing peak 500Mbps out of 2Gbps
• 2 Sun x4200 w/Dual Operons, 2G mem
• 2.5 TB Sata-based SAN
• All ISOs in disk cache or in-memoy FS
• CoBlitz uses 100MB mem per node
• Many PL node disks are IDEs
• Most nodes are BW capped at 10Mpbs

30
Conclusion

• Scalable large-file transfer service
• Evolution under real traffic
• Up and running 24/7 for over 2.5 years
• Unilateral peering, multi-hop routing, window
  size adjustment
• Better performance than P2P
• Better throughput, download time
• Far less origin traffic

31
Thank you!

• More information
• http//codeen.cs.princeton.edu/coblitz/
• How to use
• http//coblitz.codeen.org3125/URL

Some content restrictions apply See Web site
for details Contact me if you want full access!