Taming the LFN: Application Level Tunneling

1
Taming the LFNApplication Level Tunneling

• Lewis Girod
• 558 Final Presentation
• 7 May 1999
• girod_at_usc.edu

2
Outline

• Introduction
• Related Work
• Implementation
• Experimental Setup
• Performance Analysis
• Future Work

3
Introduction

• LFN Long Fat Network
• RTT is the time constant for TCPs feedback
  mechanisms. Longer RTT
• Slower Response to link conditions
• Oscillation and Overcompensation
• send too much,
• cause losses,
• back off too much

4
A Pretty Picture 40ms
5
300ms Its Not Pretty
6
What do we do?

• Bigger problems in heterogeneous environments
• One network has high delay, another has low delay
• End-to-end TCP will get clobbered

Origin Server
Client
LFN 600ms
This ISP sucks!
40ms RTT
40ms RTT
Loss Here
Causes Overcompensation Here
7
Application Level Tunnels

• Solution Proxy the connections above the network
  layer so that loss is isolated within each
  network
• How to do this most efficiently?
• Want High Throughput and Low Latency

Origin Server
Client
LFN 600ms
40ms RTT
40ms RTT
8
Our Solution

• Caching Web Proxy servers at borders
• Tunnel sessions across transit network
• Tunnel application data, not packets
• Objective Low Latency
• Users want low latency
• Lower latency means less load on Proxy
• Also want to ensure high throughput
• Want to be sure to get BW out of net

9
Multiplexing

• Multiplex sessions over single TCP
• Avoid repeated slow start, overshoot
• Reduce overhead at endpoints
• More efficient use of network resources
• Drawbacks
• Connection stalls
• Competition with other traffic

10
Related Work

• Frystyk Gettys, WebMux
• Very similar multiplexing scheme
• Intended to be used between end client and a
  server or proxy
• Alternative to HTTP Persistent Connections
• Focus on low B/W apps (Cell Modems)
• Flow control too rigid
• Proxy experiences dynamic changes in number of
  concurrent connections

11
Related Work 2

• Numerous papers on TCP over satellite
• Allman, et al, RFC 2488 Large windows, Link
  Layer FEC, TCP SACK
• Allman et al, TCP Performance over Satellite
  Links SACK 4Reno
• Kruse, et al, HTTP Page Transfer Rates over
  Geostationary Satellite Links 4K initial window

12
Related Work 3

• Morris, TCP Behavior with Many Flows
• Conclusion simultaneous connections cause loss,
  do not distribute BW fairly
• Theoretical results do not consider Fast
  Retransmit
• Simulation results do not consider SACK
• Results based exclusively on long-lived
  connections

13
Implementation

• Protocol Design
• Buffer Allocation
• Queueing
• Integration with Squid

14
Protocol Design

• Connection Establishment Release

Send Open
SYN
Send, Recv Open
SYN
Recv Open
FIN
Consume data in buffer.. Recv Closed
FINACK
Send Closed
15
Protocol Design 2

• Connection Establishment Reset

Send Open
SYN
Send, Recv Open
SYN
Recv Open
RST
Nuke data in buffer.. Recv Closed
Send Closed
FIN
16
Buffer Allocation

• Because TCP is reliable, we cannot drop data when
  we run out of buffer space
• Receiver must to allocate buffer space
• Receiver must inform sender
• Space must be allocated fairly
• Small sessions stuck behind large sessions
• Fast servers serving slow clients

17
Buffer Allocation 2

• Token-based scheme
• Each link has fixed buffer to allocate Tokens
  represent free bytes in buffer
• Initial handshake all tokens sent to peer
• Peer allocates as needed to senders, keep
  per-sender count of outstanding tokens
• When buffer freed, ACK designates number of bytes
  freed by which session

18
Buffer Allocation 3

• Must react to session dynamics
• Two metrics
• Fair Share
• TotalBuffer / 2ActiveSessions - Outstanding
• Intuition
• Take half of buffer, divide fairly. Allocate to
  senders the remainder of their share.
• Leaves half of buffer open for new arrivals

19
Buffer Allocation 4

• Second Metric
• Share Remainder
• RemainingBuffer / 2NumDueFairShare
• Intuition
• Take half of remaining buffer space, find fair
  share among those not already over budget
• Leaves half of remaining buffer for new arrivals
• Third Metric Maximum Segment Size

20
Queueing

• Sessions are queued round-robin
• The session queued to send
• Allocated number of bytes according to
• min(FairShare, ShareRemainder, MSS)
• Muxes that share link share token pool
• Muxes given opportunity to obtain tokens in round
  robin queue
• New connections applied to least loaded

21
Integration with Squid
AFD calls

• Squid
• Non-blocking design
• Scheduler manages select() and timers
• Comm module provides non-blocking callback-based
  interface to sockets
• MUXs tentacles intertwined with Squids
• MUX I/O Uses Squid Comm module
• VFD interface transparent MUX sessions

comm
MUX
comm_()
22
Experimental Setup
180P6 BSD-2.2
266PII BSD-2.2.8
180P6 BSD-2.2
Proxy Child
Proxy Parent
Delay Box
de0
de0
xl0
xl1
10Mb Ethernet
ep0
ep0
de0
de0
All have 128M RAM
WPoly Client
WPoly Server
BSD-2.2.8 266PII
BSD-2.2.5 200P6
23
Performance Analysis

• Initial experiments TCP and delay
• Dummynet compiled into FreeBSD kernel
• Test TCP performance, understand reaction to high
  delay networks
• Performance Comparison
• Vanilla Squid
• Squid with large windows
• Squid/MUX, varying number of connections

24
Fast Retransmit, Recovery

• This trace shows a successful FRT and recovery,
  100ms stall.
• RTT is 600ms (red arrow)

25
Clustered loss ? RTO

• Fast retransmits are only allowed once per
  window.
• Here the first two losses were FRT, third caused
  RTO
• 3 second stall!!
• Red arrow is RTT

26
Slow Start Lossage

• Slow start sends two packets for every ack.
• These packets are causing loss, but the feedback
  is delayed.
• When it arrives it is too late the packets have
  all been sent, and half of them were lost.
• RTO slow start six second stall!!

27
Squid/MUX Performance

• How does Squid/MUX behave?
• How does the number of connections influence its
  performance?
• Answers (so far..)
• In absence of other traffic, one connection might
  be best (but very bad if it stalls)
• Too many connections is counterproductive
• It really depends on background traffic

28
Squid/MUX 1 grows to 3
29
Squid/MUX 5 connections

• 200KB/s
• Note1All tests performed under 20r/s request
  load, Poisson distribution.
• Note2 problems with dummynet prevented long
  traces that might more accurately show steady
  state behavior

30
Squid/MUX 7, 10 connections
31
Squid with Large Windows

• Lower average bandwidth is achieved using
  separate connections and large windows.
• The bandwidth achieved is more uniform however.
  This may in part be an artifact of the Poisson
  model used by Polygraph
• Short connections do not overshoot slow start

32
What about Delay?

• The most significant impact of multiplexing is
  latency.
• Avoiding slow start for small connections is a
  huge win, although the 4K initial window proposal
  might also work.
• The following slides show the results of delay
  comparisons between Squid, Squid with large
  windows and Squid/MUX

33
40ms RTT, no B/G Traffic
Log10(RTms)
ile
Log10(DocSize)
34
300ms RTT, no B/G
Log10(RTms)
ile
Log10(DocSize)
35
Is Muxing Good?

• It is clear that Muxing reduces delay
• It is not so clear that throughput is acceptable.
  
• Long connections suffer from the overshoot
  problem
• Short connections dont get big enough to
  overshoot low individual throughput... But what
  about the aggregate throughput?

36
Future Work

• Not quite production code
• protocol is not robustly implemented
• does not conform 100 to Squid package
• scheduler integration pure guesswork
• Integration with WebMux protocol?
• Integration with the Squid codebase
• B/W estimation to optimize scheduling

37
Future Work 2

• Analysis of TCP in delayed conditions expand on
  Morris work
• SACK
• Realistic model of connection durations
• Slow start fixes like packet-pair BW estimation
  and 4K initial window

38
Future Work 3

• Improve testbed
• Ethernet switch
• Fix dummynet (reduces delay under load)
• Self-similar request model, Zipf distributed
  document sizes
• Increase number of available FDs, test tput
• background traffic? diffserv? SACK TCP?
• What are real conditions?

39
Conclusion

• Multiplexing has clear potential as a basis for
  Application Level Tunnels
• Although ALTs are most clearly indicated by
  satellite links and other very long pipes, it may
  have broader applications, for example VPNs.