ATLAS Networking

1
ATLAS Networking T2UK
Richard Hughes-Jones The University of
Manchester www.hep.man.ac.uk/rich/ then
Talks
2
Remote Computing Farms

• Discussion at CERN to establish a work-plan for
  2006
• Valuable for Monitoring and Calibration
• MOU Alberta CERN Krakow Manchester
• New Network Topology with all links carried by
  GÉANT and NRNs
• Planned Investigations
• Characterise the new network links and end host
  performance
• Toolsiperf udpmon thrulay yatm
• Measure the ATLAS request-response behaviour
• Tools tcpmon, web100 tcpdump
• Setup the WAN emulator with the measured
  conditions
• Compare network and ATLAS traffic observations
• Install and test ATLAS application gateway (as
  used at the pit)
• Test deployment of Online TDAQ HLT releases
• Measure performance of Online TDAQ HLT releases
• Consider how to link Real-Time T/DAQ to remote
  Grid farms
• First draft of Work Plan document circulated

3
Network Operation Performance

• Analysis of Fault Tolerance in ATLAS T/DAQ
  Networks
• Document the action of the switches
• Fate of the packets
• Effect on T/DAQ applications
• Networks Considered
• Front End (DataFlow) Network
• BackEnd Network
• Controls Network (Run control, services, some
  monitoring)
• Consider questions like
• Failure of a link between the ROS and the ROS
  Concentrator Switch
• Draft Document being discussed
• Performance tests discussed
• The PCI-e 4 1GE PEG4 NIC Silicom.
• Simple and trunking Throughput
• ROS SuperMicro Motherboard
• 6 PCI, 1 4 lane PCI-e, one 3.4 GHz Xeon (dual
  socket)

4
Network Monitoring in ATLAS T/DAQ

• Levels of Monitoring
• SNMP Statistics MRTG, RRD, YATM higher sample
  rate
• Traffic patterns, bytes, packets NOT dropped
  packets
• Network test programs udpmon, iperf
• Throughput loss 1-way delay rtt
• Standalone ATLAS test programs speaking the TDAQ
  application protocol.
• Richard
• ATLAS test programs speaking the TDAQ application
  protocol using TDAQ APIs
• Stefan
• Monitoring by the TDAQ application itself
• Integration of Message Passing Libraries
• DataFLow (Reiner) and EF (Mario) main difference
  in substantiation of buffers
• Integrate over common thin shim over the socket
  calls
• Idea to put monitoring into (common) message
  passing layer
• What can be observed?
• Question of keeping state Application would be
  the best place !

5
Related Work RAID, ATLAS Grid

• RAID0 and RAID5 tests
• 4th Year MPhys project last semester
• Throughput and CPU load
• Different RAID parameters
• Number of disks
• Stripe size
• User read / write size
• Different file systems
• Ext2 ext3 XSF
• Sequential File Write, Read
• Sequential File Write, Read with continuous
  background read or write
• Status
• Need to check some results document
• Independent RAID controller tests planned.

6
ESLEA ATLAS Grid on UKLight

• Demonstration of benefits of Dedicated links
• 1 Gbit Lightpath Lancaster-Manchester
• Disk 2 Disk Transfers
• Storage Element with SRM using distributed disk
  pools dCache xrootd

7
Check out the end host bbftp

• What is the end-host doing with your application
  protocol?
• Transatlantic bbftp over TCP/IP
• Look at the PCI-X buses
• 3Ware 9000 controller RAID0
• 1 Gbit Ethernet link
• 2.4 GHz dual Xeon
• 660 Mbit/s

8

• Any Questions?

9

• Backup Slides

10
TCP Stacks CPU Load

• Real User problem!
• End host TCP flow at 960 Mbit/s with rtt 1 ms
  falls to 770 Mbit/s when rtt 15 ms

11
A Few Items for Discussion

• Achievable Throughput
• Sharing link Capacity (OK what is sharing?)
• Convergence time
• Responsiveness
• rtt fairness (OK what is fairness?)
• mtu fairness
• TCP friendliness
• Link utilisation (by this flow or all flows)
• Stability of Achievable Throughput
• Burst behaviour
• Packet loss behaviour
• Packet re-ordering behaviour
• Topology maybe some simple setups
• Background or cross traffic - how realistic is
  needed? what protocol mix?
• Reverse traffic
• Impact on the end host CPU load, bus
  utilisation, Offload
• Methodology simulation, emulation and Real
  links ALL help

12
More Information Some URLs 1

• UKLight web site http//www.uklight.ac.uk
• MB-NG project web site http//www.mb-ng.net/
• DataTAG project web site http//www.datatag.org/
• UDPmon / TCPmon kit writeup http//www.hep.man
  .ac.uk/rich/net
• Motherboard and NIC Tests
• http//www.hep.man.ac.uk/rich/net/nic/GigEth_te
  sts_Boston.ppt http//datatag.web.cern.ch/datata
  g/pfldnet2003/
• Performance of 1 and 10 Gigabit Ethernet Cards
  with Server Quality Motherboards FGCS Special
  issue 2004
• http// www.hep.man.ac.uk/rich/
• TCP tuning information may be found
  athttp//www.ncne.nlanr.net/documentation/faq/pe
  rformance.html http//www.psc.edu/networking/p
  erf_tune.html
• TCP stack comparisonsEvaluation of Advanced
  TCP Stacks on Fast Long-Distance Production
  Networks Journal of Grid Computing 2004
• PFLDnet http//www.ens-lyon.fr/LIP/RESO/pfldnet200
  5/
• Dante PERT http//www.geant2.net/server/show/nav.0
  0d00h002