eVLBI Network Implementations

1
eVLBI Network Implementations

• Chris Phillips
• ATNF CSIRO

Chris.Phillips_at_csiro.au
2
Networking layers

• FTP/Telnet/HTTP/SMTP
• TCP UDP RTP
• IP/ARP/RARP
• Ethernet, PPP, ATM, SONET
• Fiber, Coax, twister pair

• Application
• Transport
• Network
• Data Link
• Physical

3
Networking Layers, cont
Application
Transport
Network
Link
4
IP

• Basis of internet
• 32 bit network addressing
• Unreliable, connectionless datagram
• No guarantees IP datagram will successfully get
  to its destination
• May arrive out of order (different routing)
• Checksum over header, but not data

5
UDP

• Built on top of IP
• Connectionless, unreliable
• Packets may arrive out of order, duplicated or
  not at all
• Basically IP with port addresses
• Includes packet checksum
• Used for DNS, TFTP, BOOTP

6
TCP

• Built on top of IP
• Connection oriented
• Reliable end-to-end delivery
• Guaranteed in-order delivery
• Re-transmission of lost packets
• Every packet acknowledged
• Windowing mechanism limits amount of data sent
  before acknowledgement
• Congestion control algorithm

7
TCP, cont

• Problem with fast, high latency networks (100s
  ms)
• long fat networks
• Under-utilization of network
• Can window size and buffer size to match
  bandwidth ? delay
• Many possible TCP alternatives to better estimate
  current link capacity

8
Specialised Transport

• Reliable Blast UDP, Tsunami, SABUL/UDT
• Rate limited UDP for data. TCP control channel.
• FAST, XCP, Highspeed TCP
• Modify traditional TCP stack
• Parallel TCP, MulTCP
• Multiplex many simultaneous TCP connections
  transparently

9
PC Architecture
RAM

• CPU and RAM connected to North Bridge
• Peripherals connected to South Bridge
• ?Ethernet, PCI, ATA not on shared bus
• Fast connect between bridges
• 32bit/33 MHz PCI ? 1 Gb/s theoretically
• Max 800 Mb/s realistically

North Bridge
AGP
Ethernet
PCI
ATA
South Bridge
USB
10
eLBA
11
First eVLBI Fringes

• Distributed set of applications
• Communicate and transfer data via standard UNIX
  sockets
• Currently designed to request data from past
• Handles fraction of a second
• Transfer specific channels
• Does not yet run correlator
• Easily add scheduled transfers

12
Swinburne Cluster
13
DAS 1 Gb/s data

• Standard ATNF DAS will produce four 16?MHz
  channels
• 256 Mb/s
• May be possible to access full 64 MHz sampled
  data
• 512 Mb/s
• 2 DAS at Mopra ATCA. CPSRII Parkes, Mark5B at
  Hobart, Tid
• Alternatively use commercial samplers

14
Recorders

• Metsähovi DMA card can comfortably record 512
  Mb/s
• Gigabit Ethernet up to 800 Mbps user data
• Use 2 recorders per station for 1 Gb/s
• Need to consider packaging
• Rack mounted PC
• OS to ATNF standard (Debian)

15
Networking

• We want the option to
• Record full LBA data at Swinburne, for
  non-realtime correlation (realtime fringe
  checking)
• Realtime single baseline correlation at Swinburne
• Record at Epping for later correlation
• Realtime correlation at Epping

16
Networking, implementation

• Need simple, cheap system
• Want international compatibility
• Do not need 100 reliability
• Randomly loosing small fraction of data will not
  affect science
• Need minimal effort to implement and operate

17
Realtime Transfer

• Preferred option
• Simple cheaper
• 2 GB memory 30 seconds buffer
• Cheap 1000
• Needed at both ends
• Use modified version of wr.c to replace disk
  writes with network writes
• Needs fast or independent PCI or onboard gigabit
  Ethernet
• Will need considerable network tweaks

18
Throttled Transfer

• May want to record at data rates (1?Gb/s) higher
  than network (say 800?Mb/s)
• Need to buffer to disk
• 64 MB/s to disk
• Single disk 20-30 MB/s
• Need RAID solution
• Reading at almost same rate
• Need 8 disk RAID or two 4 disks systems and
  ping-pong recording

19
Delayed Transfer

• Buffer locally to disk then transfer after
  experiment
• Will want to transfer small fraction for fringe
  checking
• 12hr _at_ 1 Gb/s 5.5 TB data
• 8 ? 700 GB disks or 16 ? 350 GB
• Technically simpler than throttled transfer
• May cause scheduling problems

20
Fringe Checking

• Will want to be running continuos realtime fringe
  checking for ALL experiments
• Allocate fraction of cluster at Swinburne or
  Parkes
• Record from S2 output on Metsähovi recording
  using circular buffer
• Transfer just enough data to feed correlator
• Need to implement priority

21
Networking Possibilities

• TCP/IP
• Easy, reliable, standard
• Potentially slow
• UDP
• Fast, easy to send
• Not reliable, more work on rx
• No congestion control
• IP
• Difficult, time consuming
• Can customize to our requirements
• RTP
• Less standard, but better suited?

22
VSI-E

• Longer term try and implement VSI-E standard
• Compatibility with rest of the world
• Currently Haystack working on draft specification
• Using RTP (Real time protocol)
• Designed for real-time transfer of data and a
  internet standard
• Sits on top of Transport layer, can implement
  using TCP, UDP etc

23
Internet2 Land Speed Record

• Fast long haul connections are possible
• Sweden to San Jose, CA
• 4.2 Gb/s over 16343 km
• Shared network, 40 routers
• CERN CalTech
• 6.2 Gb/s over 10949 km