vlbi_udp Throughput Performance and Stability.

1
vlbi_udpThroughput Performance and Stability.
Richard Hughes-Jones The University of
Manchester www.hep.man.ac.uk/rich/ then
Talks
2
vlbi_udp Flow Diagram
Vlbi_control
vlbi_send
vlbi_recv
Control thread
Control thread
TCP Control
Disk
Memory
Disk
Memory
Input thread
Output thread
Ring buffer
UDP Data
Send thread
Receive thread
From Simon Casey
3
History or the story so far

• iGrid2002 monolithic code
• Convert to use pthreads
• control
• Data input
• Data output
• Code branch for Simons file transfer tests
• Work on vlbi_recv
• Output thread polled for data in the ring buffer
  burned CPU
• Input thread signals output thread when there is
  work to do else wait on semaphore packet loss
  at high rate, variable thoughput
• Output thread uses sched_yield() when no work to
  do CPU used
• Network performance in progress

4
Instability 2 streams into JIVE, March 2006
From Simon Casey
5
Early B2B UDP Tests

• Kernel 2.6.6 (May06)
• vlbi_recv semaphore
• Wait 15 us signal 200 events
• Unstable throughput
• 815 Mbit/s
• Packet loss 0.05
• Lost at top of IP stack
• Inter-packet time
• Processing time
• mean 0.1166 sigma 0.3493
• CPU load

• L1 0 0 0 100 0 0 0
• L2 0 0 0 99 0 0 0
• L3 0 0 0 99 0 0 0
• L4 2 0 43 53 0 1 18

6
Confirm with B2B UDP Tests

• Kernel 2.6.6 (Aug06)
• vlbi_recv semaphore
• Wait 15 us signal 100 events
• Throughput a bit unstable
• 815 Mbit/s
• Packet loss 0.05
• Lost at top of IP stack
• CPU number packet loss
• Have loss when using same CPU as that servicing
  interrupts
• Confirmed by many udpmon tests !

7
B2B UDP Tests different kernel

• Kernel 2.6.9
• vlbi_recv semaphore
• Wait 12 us signal 200 events
• Stable throughput
• 999 Mbit/s variation less than 1 Mbit/s
• No packet loss
• Inter-packet time
• Processing time
• mean 0.1206 sigma 0.3274
• CPU load

• Cpu0 0.0 us, 0.0 sy, 0.0 ni, 99.3 id,
  0.3 wa, 0.3 hi, 0.0 si
• Cpu1 0.3 us, 0.0 sy, 0.0 ni, 99.3 id,
  0.3 wa, 0.0 hi, 0.0 si
• Cpu2 0.3 us, 0.0 sy, 0.0 ni, 99.7 id,
  0.0 wa, 0.0 hi, 0.0 si
• Cpu3 6.0 us, 17.0 sy, 0.0 ni, 40.3 id,
  0.0 wa, 1.0 hi, 35.7 si

8
B2B UDP Tests

• Kernel 2.6.9
• vlbi_recv sched_yield()
• Wait 20 us
• Stable throughput
• 612.5 Mbit/s variation less than 1 Mbit/s
• No packet loss
• Inter-packet time
• Processing time
• mean 0.1005 sigma 0.1438
• CPU load

• Cpu0 3.7 us, 8.0 sy, 0.0 ni, 88.4 id,
  0.0 wa, 0.0 hi, 0.0 si
• Cpu1 1.0 us, 4.0 sy, 0.0 ni, 95.0 id,
  0.0 wa, 0.0 hi, 0.0 si
• Cpu2 0.0 us, 0.0 sy, 0.0 ni, 100.0 id,
  0.0 wa, 0.0 hi, 0.0 si
• Cpu3 11.6 us, 22.9 sy, 0.0 ni, 41.9 id,
  0.0 wa, 1.0 hi, 22.6 si

9
B2B UDP Tests

• Kernel 2.6.9
• vlbi_recv sched_yield()
• Wait 12 us
• Stable throughput
• 999 Mbit/s variation less than 1 Mbit/s
• No packet loss
• Inter-packet time
• Processing time
• mean 0.1005 sigma 0.1438
• CPU load

• Cpu0 0.0 us, 0.0 sy, 0.0 ni, 99.7 id,
  0.3 wa, 0.0 hi, 0.0 si
• Cpu1 11.3 us, 88.7 sy, 0.0 ni, 0.0 id,
  0.0 wa, 0.0 hi, 0.0 si
• Cpu2 0.3 us, 0.0 sy, 0.0 ni, 99.3 id,
  0.3 wa, 0.0 hi, 0.0 si
• Cpu3 9.3 us, 15.6 sy, 0.0 ni, 37.5 id,
  0.0 wa, 1.3 hi, 36.2 si

10
Next Steps

• Fix the timing precision again!
• Test with latest 2.6.16 and 2.6.17 kernels
• Why does shed_yield() take more CPU time? Is it a
  problem or not?
• Test on long-distance Networks
• Production (packet switched)
• UKLight
• Test at 4-8 Gbit

11

• Any Questions?

12
More Information Some URLs 1

• UKLight web site http//www.uklight.ac.uk
• MB-NG project web site http//www.mb-ng.net/
• DataTAG project web site http//www.datatag.org/
• UDPmon / TCPmon kit writeup http//www.hep.man
  .ac.uk/rich/net
• Motherboard and NIC Tests
• http//www.hep.man.ac.uk/rich/net/nic/GigEth_te
  sts_Boston.ppt http//datatag.web.cern.ch/datata
  g/pfldnet2003/
• Performance of 1 and 10 Gigabit Ethernet Cards
  with Server Quality Motherboards FGCS Special
  issue 2004
• http// www.hep.man.ac.uk/rich/
• TCP tuning information may be found
  athttp//www.ncne.nlanr.net/documentation/faq/pe
  rformance.html http//www.psc.edu/networking/p
  erf_tune.html
• TCP stack comparisonsEvaluation of Advanced
  TCP Stacks on Fast Long-Distance Production
  Networks Journal of Grid Computing 2004
• PFLDnet http//www.ens-lyon.fr/LIP/RESO/pfldnet200
  5/
• Dante PERT http//www.geant2.net/server/show/nav.0
  0d00h002

13
More Information Some URLs 2

• Lectures, tutorials etc. on TCP/IP
• www.nv.cc.va.us/home/joney/tcp_ip.htm
• www.cs.pdx.edu/jrb/tcpip.lectures.html
• www.raleigh.ibm.com/cgi-bin/bookmgr/BOOKS/EZ306200
  /CCONTENTS
• www.cisco.com/univercd/cc/td/doc/product/iaabu/cen
  tri4/user/scf4ap1.htm
• www.cis.ohio-state.edu/htbin/rfc/rfc1180.html
• www.jbmelectronics.com/tcp.htm
• Encylopaedia
• http//www.freesoft.org/CIE/index.htm
• TCP/IP Resources
• www.private.org.il/tcpip_rl.html
• Understanding IP addresses
• http//www.3com.com/solutions/en_US/ncs/501302.htm
  l
• Configuring TCP (RFC 1122)
• ftp//nic.merit.edu/internet/documents/rfc/rfc1122
  .txt
• Assigned protocols, ports etc (RFC 1010)
• http//www.es.net/pub/rfcs/rfc1010.txt
  /etc/protocols

14

• Backup Slides

15
VLBI Network Topology
16
Geant2 Launch event, June 2005