Influence of heavytailed distributions on load balancing

1
Scheduling Your Network Connections
Mor Harchol-Balter Carnegie Mellon
with Mukesh Agrawal
with Bianca Schroeder
with Nikhil Bansal
2
Q Which minimizes mean response time?
size service requirement
load r lt 1
3
Q Which best represents scheduling in web
servers ?
FCFS
jobs
size service requirement
load r lt 1
jobs
PS
SRPT
jobs
4
How about using SRPT in Web servers as opposed
to the traditional processor-sharing (PS) type
scheduling ?
5
Immediate Objections
1) Cant assume known job size
Many servers receive mostly static web
requests.GET FILEFor static web requests,
know file size
Approx. know service requirement of request.
2) But the big jobs will starve ...
6
Outline of Talk
1) Analysis of SRPT Scheduling Investigating
Unfairness with Nikhil Bansal
2) Implementation of SRPT
Scheduling in Web Servers with Nikhil Bansal and
Bianca Schroeder and Mukesh Agrawal

7
SRPT has a long history ...
1966 Schrage Miller derive M/G/1/SRPT response
time
1968 Schrage proves optimality 1979 Pechinkin
Solovyev Yashkov generalize 1990 Schassberger
derives distribution on queue length
BUT WHAT DOES IT ALL MEAN?
8
SRPT has a long history (cont.)
1990 - 97 7-year long study at Univ. of Aachen
under Schreiber SRPT WINS BIG ON
MEAN! 1998, 1999 Slowdown for SRPT under
adversary Rajmohan, Gehrke,
Muthukrishnan, Rajaraman, Shaheen,
Bender, Chakrabarti, etc. SRPT
STARVES BIG JOBS! Various o.s. books
Silberschatz, Stallings, Tannenbaum
Warn about starvation of big jobs
... Kleinrocks Conservation Law
Preferential treatment given to one class of
customers is afforded at the expense of other
customers.
9
Our Analytical Results (M/G/1)
All-Can-Win Theorem Under workloads with
heavy-tailed (HT) property, ALL jobs, including
the very biggest, prefer SRPT to PS, provided
load not too close to 1. Almost-All-Win-Big
Theorem Under workloads with HT property, 99
of all jobs perform orders of magnitude better
under SRPT.
PS
SRPT
Many more such results in Sigmetrics talk.
10
Our Analytical Results (M/G/1)
Moderate-Load Theorem If load lt .5, for every
job size distribution, ALL jobs prefer SRPT to
PS. Bounding-the-damage Theorem For any load,
for every job size distribution, for every size x,
ö
æ
r

ç

lt
x
T
E
x
T
E
)
(

1
)
(


ç
PS
SRPT
)
-
2(1
r
ø
è
11
Whats the Heavy-Tail property?
Defn heavy-tailed distribution
-
a
lt
lt
gt
a
2
0

,








Pr
x
x
X
Many real-world workloads well-modeled by
truncated HT distribution. Key property HT
Property Largest 1 of jobs comprise half the
load.
12
From theory to practice
What does SRPT mean within a Web server?

• Many devices Where to do the scheduling?
• No longer one job at a time.

13
Previous work on implementation
1998 Almeida, Dabu, Manikutty, Cao
Prioritizing HTTP requests at Web servers.
Nice the low-priority process.
1999 Crovella, Frangioso, Harchol-Balter
SRPT scheduling of requests. User-level
scheduling of reads and writes.
What is the problem with both of these?
14
Network/O.S. insides of traditional Web server
IMPLEMENT
Socket 1
Client1
Network Card
Socket 2
Client2
BOTTLENECK
Socket 3
Client3
Sockets take turns draining --- FAIR PS.
15
Network/O.S. insides of our improved Web server
Socket 1
Client1
S
Network Card
1st
Socket 2
Client2
2nd
M
BOTTLENECK
3rd
Socket 3
Client3
L
priority queues.
Socket corresponding to file with smallest
remaining data gets to feed first.
16
Experimental Setup
1
1
APACHE WEB SERVER
CLIENTS
2
2
3
3
Direct Connection 10Mb/s
200
200
Linux 0.S.
Linux 0.S.

• Implement SRPT-based scheduling
• 1) Modifications to Linux O.S. 6 priority
  Levels
• 2) Modifications to Apache Web server
• 3) Priority algorithm design.

17
Experimental Setup
1
1
APACHE WEB SERVER
2
2
CLIENTS
3
3
200
200
Linux 0.S.
Linux 0.S.

• Comparison of SRPT implementation vs. FAIR
  scheduling standard.
• Comparison done under both APACHE and FLASH web
  servers.
• Comparisons done under trace-based workload and
  under
• Web workload generator.
• Under trace-based workload
• -- Number requests made 1,000,000
• -- Size of file requested 41B -- 2 MB
• -- Distribution of file sizes
  requested has HT property.
• Experiment under range of loads.

18
Preliminary Comments to Results
1
1
APACHE WEB SERVER
2
2
CLIENTS
3
3
200
200
Linux 0.S.
Linux 0.S.

• Measured job throughput, byte throughput, and
  bandwidth
• utilization were same under SRPT and FAIR
  scheduling.
• Same set of requests complete.
• No additional CPU overhead under SRPT
  scheduling.
• CPU utilization always 1 - 5.
• Network was bottleneck in all experiments.

19
Results Mean Response Time
Mean Response Time (ms)
FAIR
SRPT
Load
20
Results Mean Slowdown
FAIR
Mean Slowdown
SRPT
Load
21
Results Mean Response Time vs. Size
Load 0.8
Mean Response Time (ms)
FAIR
SRPT
Requested File Size (bytes)
22
Mean Response Time vs. Size Percentile
Load 0.8
FAIR
Mean Response time (ms)
SRPT
Percentile of Request Size
23
Summary so far ...

• SRPT scheduling yields significant improvements
  in Mean Response Time at the server.
• Negligible starvation.
• No CPU overhead.
• No drop in throughput.

24
More questions

• This study involved a LAN.
• Are the effects of SRPT in a WAN as strong?
• So far weve only experimented with load lt 1.
• What happens under SRPT vs. FAIR when the
• server runs under transient overload?
• -gt new analysis
• -gt implementation study

25
WAN setting
Question
Will improvement of SRPT over FAIR scheduling
appear greater in a LAN setting or a WAN setting?
26
WAN setting
Answer
Normally LAN setting shows more improvement.
But for very high load, WAN setting shows more
improvement. Why?
Load 0.7
Load 0.9
27
Overload
Person under overload
28
Web server under overload
When reach SYN-queue limit, server drops all
connection requests.
SYN-queue
Clients
Server
SYN-queue
ACK-queue
Apache-processes
29
FAIR vs. SRPT
Question What will happen under FAIR vs.
SRPT wrt number of connections at server and wrt
response times? (a) Given persistent
overload (b) Given transient overload
rgt1
rgt1
rgt1
rgt1
rlt1
rlt1
30
Persistent Overload Time until SYN-queue limit
is hit
r1.5
FAIR
FAIR (tuned version)
SRPT
31
Persistent Overload Buildup in number of
connections

FAIR
SRPT
32
Transient Overload Buildup of Connections at
Server
FAIR
SRPT
33
Transient Overload Mean response time (in msec)
FAIR
SRPT
34
Transient overload Response time as function of
job size
FAIR
SRPT
small jobs win big!
big jobs arent hurt!
WHY?
35
Conclusion

• SRPT scheduling yields significant improvements
  in Mean Response Time at the server under
• LAN, WAN, under high load, and under overload.
• Negligible or zero unfairness. Often better
• for all requests.
• All results corroborated via implementation and
  analysis.