Richard HughesJones

1
MB-NG ReviewHigh Performance Network
Demonstration 21 April 2004

• Richard Hughes-Jones
• The University of Manchester, UK

2
It works ?So whats the Problem with TCP

• TCP has 2 phases Slowstart Congestion
  Avoidance
• AIMD and High Bandwidth Long Distance networks
• Poor performance of TCP in high bandwidth wide
  area networks is due
• in part to the TCP congestion control algorithm -
  cwnd congestion window
• For each ack in a RTT without loss
• cwnd -gt cwnd a / cwnd - Additive Increase,
  a1
• For each window experiencing loss
• cwnd -gt cwnd b (cwnd) - Multiplicative
  Decrease, b ½
• Time to recover from 1 packet loss 100 ms rtt

3
Investigation of new TCP Stacks

• High Speed TCP
• a and b vary depending on current cwnd using a
  table
• a increases more rapidly with larger cwnd
  returns to the optimal cwnd size sooner for the
  network path
• b decreases less aggressively and, as a
  consequence, so does the cwnd. The effect is that
  there is not such a decrease in throughput.
• Scalable TCP
• a and b are fixed adjustments for the increase
  and decrease of cwnd
• a 1/100 the increase is greater than TCP Reno
• b 1/8 the decrease on loss is less than TCP
  Reno
• Scalable over any link speed.
• Fast TCP
• Uses round trip time as well as packet loss to
  indicate congestion with rapid convergence to
  fair equilibrium for throughput.
• HSTCP-LP High Speed (Low Priority) backs off if
  rtt increases
• BiC-TCP Additive increase large cwnd binary
  search small cwnd
• H-TCP after congestion standard then switch to
  high performance
• ???

4
Comparison of TCP Stacks

• TCP Response Function
• Throughput vs Loss Rate steeper faster
  recovery
• Drop packets in kernel

MB-NG rtt 6ms
DataTAG rtt 120 ms
5
Multi-Gigabit flows at SC2003 BW Challenge

• Three Server systems with 10 GigEthernet NICs
• Used the DataTAG altAIMD stack 9000 byte MTU
• Send mem-mem iperf TCP streams From SLAC/FNAL
  booth in Phoenix to
• Chicago Starlight
• rtt 65 ms
• window 60 MB
• Phoenix CPU 2.2 GHz
• 3.1 Gbit hstcp I1.6
• Amsterdam SARA
• rtt 175 ms
• window 200 MB
• Phoenix CPU 2.2 GHz
• 4.35 Gbit hstcp I6.9
• New TCP stacks are very Stable
• Both used Abilene to Chicago

6
Transfer Applications Throughput 1

• 2Gbyte file transferred RAID0 disks Manc UCL
• GridFTP
• See alternate 600/800 Mbit and zero

• Apache web server curl-based client
• See steady 720 Mbit

7
Transfer Applications Throughput 2

• 2Gbyte file transferred RAID5 - 4disks Manc RAL
• bbcp
• Mean 710 Mbit/s

Mean 710

• GridFTP
• See many zeros

Mean 620
8
Topology of the MB NG Network
Manchester Domain
UKERNA DevelopmentNetwork
UCL Domain
Boundary Router Cisco 7609
Boundary Router Cisco 7609
Edge Router Cisco 7609
RAL Domain
Key Gigabit Ethernet 2.5 Gbit POS Access MPLS
Admin. Domains
Boundary Router Cisco 7609
9
High Throughput Demo
Manchester
London
Dual Zeon 2.2 GHz
Dual Zeon 2.2 GHz
Cisco GSR
Cisco GSR
Cisco 7609
Cisco 7609
1 GEth
1 GEth
2.5 Gbit SDH MB-NG Core
10
Standard to HS-TCP

• No loss, but output queue filled by sender

11
HS-TCP to Scalable

• No loss, but output queue filled by sender

12
Standard, HS-TCP, Scalable

• Drop 1 in 25,000

13
Standard Reno TCP

• Drop 1 in 106

14
Focus on Helping Real Users Throughput CERN
-SARA

• Using the GÉANT Backup Link
• 1 GByte disk-disk transfers
• Blue is the Data
• Red is the TCP ACKs
• Standard TCP
• Average Throughput 167 Mbit/s
• Users see 5 - 50 Mbit/s!

• High-Speed TCP
• Average Throughput 345 Mbit/s

• Scalable TCP
• Average Throughput 340 Mbit/s
• Technology link to EU Projects DataGrid DataTAG
  GÉANT

15
BaBar Case Study Host, PCI RAID Controller
Performance

• RAID0 (striped) RAID5 (stripped with
  redundancy)
• 3Ware 7506 Parallel 66 MHz 3Ware 7505 Parallel
  33 MHz
• 3Ware 8506 Serial ATA 66 MHz ICP Serial ATA
  33/66 MHz
• Tested on Dual 2.2 GHz Xeon Supermicro P4DP8-G2
  motherboard
• Disk Maxtor 160GB 7200rpm 8MB Cache
• Read ahead kernel tuning /proc/sys/vm/max-readahe
  ad

Disk Memory Read Speeds
Memory - Disk Write Speeds
16
Topology of the MB NG Network
Manchester Domain
UKERNA DevelopmentNetwork
UCL Domain
Boundary Router Cisco 7609
Boundary Router Cisco 7609
Edge Router Cisco 7609
RAL Domain
Key Gigabit Ethernet 2.5 Gbit POS Access MPLS
Admin. Domains
Boundary Router Cisco 7609
17
BaBar Data Throughput on MBNG kit

• RAID5 - 4disks RAL - Manc
• Includes small files Kbytes
• bbftp 1 stream with compression

With bb diag
bbftp 6 streams

• bbftp 1 stream no compression
• 10 2 G byte files each peak is a 20 G byte
  transfer

bbftp 1 stream Files 1 Mbyte
18
Helping Real UsersRadio Astronomy VLBIPoC with
NRNs GEANT 1024 Mbit/s 24 on 7 NOW
19
VLBI Project Throughput Jitter 1-way Delay

• 1472 byte Packets man -gt JIVE
• FWHM 22 µs (B2B 3 µs )

• 1472 byte Packets Manchester -gt Dwingeloo JIVE

• 1-way Delay note the packet loss (points with
  zero 1 way delay)

20
Case Study ATLAS LHC

• Tests streaming built Events from Level3 Trigger
  to remote compute farm in real time
• 500 Mbit to 1 Gbit CERN Man
• Investigation of use of new high performance TCPs
  
• Testing concepts in the ATLAS Offline Computing
  model
• More Mesh than Star
• CERN Tier0 to Tier 1s
• Tier 2s to all Tier 1s
• Tests planned over production networks
• Lancaster-Manchester NNW SuperJANET4
• Lancaster-Manchester to CERN

21
(No Transcript)
22
Scalable TCP DataTAG

• Drop 1 in 106

23
HS-TCP DataTAG

• Drop 1 in 106

24
Standard Reno TCP DataTAG

• Drop 1 in 106 Transition highspeed to Standard
  TCP _at_ 520s

25
Summary

• Multi-Gigabit transfers are possible and stable
• Demonstrated that new TCP stacks help performance
  
• DataTAG has made major contributions to
  understanding of high-speed networking
• There has been significant technology transfer
  between DataTAG and other projects
• Now reaching out to real users.
• But still much research to do
• Achieve performance Protocol vs implementation
  issues
• Stability / Sharing issues
• Optical transports hybrid networks

26
10 Gigabit Tuning PCI-X

• 16080 byte packets every 200 µs
• Intel PRO/10GbE LR Adapter
• PCI-X bus occupancy vs mmrbc
• Measured times
• Times based on PCI-X times from the logic
  analyser
• Expected throughput 7 Gbit/s

27
DataTAG Testbed
28
BaBar Case Study Disk Performance

• BaBar Disk Server
• Tyan Tiger S2466N motherboard
• 1 64bit 66 MHz PCI bus
• Athlon MP2000 CPU
• AMD-760 MPX chipset
• 3Ware 7500-8 RAID5
• 8 200Gb Maxtor IDE 7200rpm disks
• Note the VM parameterreadahead max
• Disk to memory (read)Max throughput 1.2 Gbit/s
  150 MBytes/s)
• Memory to disk (write)Max throughput 400 Mbit/s
  50 MBytes/s)not as fast as Raid0

29
RAID Controller Performance
Write Speed
Read Speed
RAID 0
RAID 5
30
BaBar Serial ATA Raid Controllers RAID5

• 3Ware 66 MHz PCI

• ICP 66 MHz PCI

31
VLBI Project Packet Loss Distribution

• Measure the time between lost packets in the time
  series of packets sent.
• Lost 1410 in 0.6s
• Is it a Poisson process?
• Assume Poisson is stationary
  ?(t) ?
• Use Prob. Density Function P(t) ?
  e-?t
• Mean ? 2360 / s426 µs
• Plot log slope -0.0028expect -0.0024
• Could be additional process involved

32
The performance of the end host / disks BaBar
Case Study RAID BW PCI Activity

• 3Ware 7500-8 RAID5 parallel EIDE
• 3Ware forces PCI bus to 33 MHz
• BaBar Tyan to MB-NG SuperMicroNetwork mem-mem
  619 Mbit/s
• Disk disk throughput bbcp 40-45 Mbytes/s (320
  360 Mbit/s)
• PCI bus effectively full!
• User throughput 250 Mbit/s

Read from RAID5 Disks
Write to RAID5 Disks