TCP-XM: Unicast-enabled Reliable Multicast

1
TCP-XM Unicast-enabled Reliable Multicast

• Karl Jeacle

karl.jeacle_at_cl.cam.ac.uk
2
Rationale

• Would like to achieve high-speed bulk data
  delivery to multiple sites
• Multicasting would make sense
• Existing multicast research has focused on
  sending to a large number of receivers
• But Grid is an applied example where sending to a
  moderate number of receivers would be extremely
  beneficial

3
Multicast availability

• Deployment is a problem!
• Protocols have been defined and implemented
• Valid concerns about scalability much FUD
• chicken egg means limited coverage
• Clouds of native multicast
• But cant reach all destinations via multicast
• So applications abandon in favour of unicast
• What if we could multicast when possible
• but fall back to unicast when necessary?

4
Multicast TCP?

• TCP
• single reliable stream between two hosts
• Multicast TCP
• multiple reliable streams from one to n hosts
• May seem a little odd, but there is precedent
• TCP-XMO Liang Cheriton
• M-TCP Mysore Varghese
• M/TCP Visoottiviseth et al
• PRMP Barcellos et al
• SCE Talpade Ammar

5
ACK implosion
6
Building Multicast TCP

• Want to test multicast/unicast TCP approach
• But new protocol kernel change
• Widespread test deployment difficult
• Build new TCP-like engine
• Encapsulate packets in UDP
• Run in userspace
• Performance is sacrificed
• but widespread testing now possible

7
TCP/IP/UDP/IP
8
TCP engine

• Where does initial TCP come from?
• Could use BSD or Linux
• Extracting from kernel could be problematic
• More compact alternative
• lwIP Lightweight IP
• Small but fully RFC-compliant TCP/IP stack
• lwIP multicast extensions TCP-XM

9
TCP-XM overview

• Primarily aimed at push applications
• Sender initiated advance knowledge of receivers
• Opens sessions to n destination hosts
  simultaneously
• Unicast is used when multicast not available
• Options headers used to exchange multicast info
• API changes
• Sender incorporates multiple destination and
  group addresses
• Receiver requires no changes
• TCP friendly, by definition

10
TCP
SYN
SYNACK
ACK
Sender
Receiver
DATA
ACK
FIN
ACK
FIN
ACK
11
TCP-XM
Receiver 1
Sender
Receiver 2
Receiver 3
12
TCP-XM connection

• Connection
• User connects to multiple unicast destinations
• Multiple TCP PCBs created
• Independent 3-way handshakes take place
• SSM or random ASM group address allocated
• (if not specified in advance by user/application)
• Group address sent as TCP option
• Ability to multicast depends on TCP option

13
TCP Group Option
kind50
len6
Multicast Group Address
1 byte
1 byte
4 bytes

• New group option sent in all TCP-XM SYN packets
• Non TCP-XM hosts will ignore (no option in
  SYNACK)
• Presence implies multicast capability
• Sender will automatically revert to unicast

14
TCP-XM transmission

• Data transfer
• Data replicated/enqueued on all send queues
• PCB variables dictate transmission mode
• Data packets are multicast (if possible)
• Retransmissions are unicast
• Auto fall back/forward to unicast/multicast
• Close
• Connections closed as per unicast TCP

15
TCP-XM protocol states
16
Fall back / fall forward

• TCP-XM principle
• Multicast if possible, unicast when necessary
• Initial transmission mode is group unicast
• Ensures successful initial data transfer
• Fall forward to multicast on positive feedback
• Typically after 75K unicast data
• Fall back to unicast on repeated mcast failure

17
Multicast feedback Option
kind51
len3
Multicast Packets Received
1 byte
1 byte
1 byte

• New feedback option sent in ACKs from receiver
• Only used between TCP-XM hosts
• Indicates of last n packets received via
  multicast
• Used by sender to fall forward to multicast
  transmission

18
TCP-XM reception

• Receiver
• No API-level changes
• Normal TCP listen
• Auto-IGMP join on TCP-XM connect
• Accepts data on both unicast/multicast ports
• tcp_input() accepts
• packets addressed to existing unicast
  destination
• but now also those addressed to multicast group
• Tracks how last n segs received (u/m)

19
API changes

• Only relevant if natively implemented!
• Sender API changes
• New connection type
• Connect to port on array of destinations
• Single write sends data to all hosts
• TCP-XM in use
• conn netconn_new(NETCONN_TCPXM)
• netconn_connectxm(conn, remotedest, numdests,
  group, port)
• netconn_write(conn, data, len, )

20
PCB changes

• Every TCP connection has an associated Protocol
  Control Block (PCB)
• TCP-XM adds
• struct tcp_pcb
• struct tcp_pcb firstpcb/ first of the mpcbs
  /
• struct tcp_pcb nextm / next tcpxm pcb /
• enum tx_mode txmode / unicasting or
  multicasting /
• u8_t nrtxm / number of
  retransmits for multicast /
• u32_t nrtxmtime / time since last
  retransmit /
• u32_t mbytessent / total bytes sent
  via multicast /
• u32_t mbytesrcvd / total bytes
  received via multicast /
• u32_t ubytessent / total bytes sent
  via unicast /
• u32_t ubytesrcvd / total bytes
  received via unicast /
• struct segrcv msegrcv128 / ismcast boolean
  for last n segs /
• u8_t msegrcvper / of last segs
  received via mcast /
• u8_t msegrcvcnt / counter for segs
  recvd via mcast /
• u8_t msegsntper / of last segs
  delivered via mcast /

21
Linking PCBs

• next points to the next TCP session
• nextm points to the next TCP session thats part
  of a particular TCP-XM connection
• Minimal timer and state machine changes

22
What happens to the cwin?

• Multiple receivers
• Multiple PCBs
• Multiple congestion windows
• Default to min(cwin)
• i.e. send at rate of slowest receiver
• Is this really so bad?
• Compare to time taken for n unicast transfers

23
LAN speed
24
LAN efficiency
25
WAN speed
26
WAN efficiency
27
Multiple TCP-XM flows
28
TCP-XM vs TCP flows
29
Protocol performance
30
Protocol efficiency
31
mcp mcpd

• Multicast file transfer application using TCP-XM
• mcpd on servers
• mcp file host1 host2 hostN on client
• http//www.cl.cam.ac.uk/kj234/mcp/
• Full source code online
• FreeBSD, Linux, Solaris

32
All done!

• Thanks for listening!
• Questions?