Queuing Theory

1
Queuing Theory
2
Web Performance

• We talked about the importance of performance in
  web applications
• It is not easy to separate the different
  components of response time
• It is important to be able to make predictions
  from information that is easy to gather.

3
Approaches to performance analysis

• Try new configuration and observe results
• Scale up from prior results
• Arrival rate .5, mean 43, max 627
• Arrival rate .95, mean 1859, max 4285
• Arrival rate .99, mean 2583, max 5068
• Develop queuing theoretic model
• Run a simulation

4
Predicting response from past experience
5
Queue Model
6
Multiserver queue Model
Single web server forwards requests to cluster
7
Multiple Single-server queues
DNS rotation for cluster
8
A Quick Probability Refresher

• A random variable, X, can take on a number of
  different possible values
• Example the number of students who came to
  office hours is a random variable with values
  1,2,3,
• Each time we observe (or sample) the random
  variable, it may take on a different value
• A random variable takes on each of these values
  with a specified probability
• Example X 0, 1, 2, 3, 4
• PX0 .1, PX1 .2, PX2 .4, PX3
  .1, PX4 .2
• The sum of the probabilities of all values equals
  1
• ?all values PXvalue 1

9
A Quick Probability Refresher

• Example
• Suppose we throw two dice and the random
  variable, X, is the sum of the two dice
• Possible values of X are 2, 3, 4, 5, 6, 7, 8, 9,
  10, 11, 12
• PX2 PX12 1/36
• PX3 PX11 2/36
• PX4 PX10 3/36
• PX5 PX9 4/36
• PX6 PX8 5/36
• PX7 6/36

10
A Quick Probability Refresher

• A probability distribution function matches each
  possible value of a random variable with its
  associated probability

11
A Quick Probability Refresher

• The cumulative distribution function of a random
  variable, X, is defined by
• CDF PX ? x ?all yx Pxy

12
A Quick Probability Refresher

• Expected Value
• Can be thought of a long term average of
  observing the random variable a large number of
  times
• Example dice
• EX 21/36 32/36 43/36 54/36 65/36
  76/36 85/36 94/36 103/36 112/36
  121/36

13
A Quick Probability Refresher

• Average vs. Expected Value
• Short term average
• Suppose a random variable X is sampled N times
• Let ni of X i was observed
• Average of samples
• (n00 n11 n22 n33 )/N
• n0/N0 n1/N1 n2/N2 n3/N3
• As N ? ?, the ratio ni/N becomes pi
• Thus, EX
• limN?? n0/N0 n1/N1 n2/N2 n3/N3
• p00 p11 p22 p33
• ??i0 i pi

14
A Quick Probability Refresher

• Continuous Random Variables
• In many cases, a random variable takes a value
  drawn from a continuous interval
• Ex processing time for a packet may be any real
  value 0, ?)
• The distribution of possible values a continuous
  random variable can take is given by a
  probability density function, F(x)
• P(a ? x ? b) ?ab F(x)dx ?bia P(x i)
• Ex ?-?? xF(x)dx ? i P(x i)

15
Exponential Distribution

• The cumulative distribution function F(x) and
  probability density function f(x) are

In queuing theory, we often assume the service
time follows an exponential distribution. The
service time corresponds to the packet
transmission time and is proportional to the
packet length. When standard deviation is equal
to mean, we estimate it to be exponential
16
Poisson Distribution
17
Poisson Distribution

• The random variable X has a Poisson distribution
  with mean ?, if for non-negative integers i
• PX i (?ie-?)/i!
• Represents the number of arrivals in a time unit
  given an exponential arrival process
• Facts
• Discrete distribution
• EX ?

18
Poisson Point Process

• Definition
• A Poisson point process with parameter ?
• A point process with interpoint times that are
  independent and exponentially distributed with
  parameter ?.

19
Poisson Point Process

• Equivalently
• The number of points in disjoint intervals are
  independent, and the number of points in an
  interval of length t has a Poisson distribution
  with mean ?t

20
Poisson Point Process

• Exercise
• Given a Poisson point process with rate ? 0.4,
  what is the probability of NO arrivals in an
  interval of length 5?

Try to answer two ways, using two equivalent
descriptions of a Poisson process
21
Poisson Point Process
Solution 1 PX gt 5 e-5? 0.1353
Solution 2 PN 0 e-5? 0.1353 (remember
PN i (5?)i (e-5?) / i!, for i 0)
22
Basic Queueing Theory

• Elementary notions
• Things arrive at a queue according to some
  probability distribution
• Things leave a queue according to a second
  probability distribution
• Averaged over time
• Things arriving and things leaving must be equal
• Or the queue length will grow without bound
• Convenient to express probability distributions
  as average rates

23
Littles Law

• Goal
• Estimate relevant values
• Average number of customers in the system
• The number of customers either waiting in queue
  or receiving service
• Average delay per customer
• The time a customer spends waiting plus the
  service time
• In terms of known values
• Customer arrival rate
• The number of customers entering the system per
  unit time
• Customer service rate
• The number of customers the system serves per
  unit time

24
Littles Law

• For any box with something steady flowing through
  it

Allows us to express the natural idea that
crowded systems (large N) are associated with
long customer delays
N ?T
25
Littles Law

• Example
• Suppose you arrive at a busy restaurant in a
  major city
• Some people are waiting in line, while other are
  already seated (i.e., being served)
• You want to estimate how long you will have to
  wait to be seated if you join the end of the line
• Do you apply Littles Law? If so
• What is the box?
• What is N?
• What is ??
• What is T?

26
Littles Law

• Box
• Include the people seated (i.e., being served)
• Do not include the people waiting in line
• Let N the number of people seated (say 200)
• Let T mean amount of time a person stays seated
  (say 90 min)
• Conclusion
• Throughput 200/90 2.22 persons per minute
• Wait time
• If 100 people are waiting, you could estimate
  that you will need to wait 100/2.22 45 min

27
Littles Law

• Suppose ATM streams are multiplexed at an output
  link with speed 622 Mbps
• Question
• If 200 cells are queued on average, what is the
  average time in queue?
• Answer
• T N/?
• T 200 53 8 / 622Mbps
• T 0.136 ms

28
Queue Statistics

• Arrival rate load - Poisson -
• Packet length Distribution - exponential
• E(m) expected value (mean) of random variable
• TsMean service time, TqMean queue time
• Standard Deviation of service time
• Link utilization

29
Fixed Packets

• Average transmission time
• ? average packets serviced/sec capacity

n
Buffer
Arriving Packets
Departing Packets
30
Practical Example
Tr
Ts
31
Littles Law

• Applies to systems where no jobs are lost or
  created
• Arrival rate arrivals/total timeN/Tt
• Mean time in system J/N
• Mean number in system

32
J is the shaded areaExamine time up to D619
T13 T27 T310 T46 T56 T66 J38
Arrival Rate N/Tt6/19.31 Mean Time in system
J/N 38/66.3 Mean number in system
J/Tt38/192 (J/N)(N/Tt)6.3.312 ?Tq
33
Throughput

• Throughpututilization/service time
• Throughput ?/Ts
• For ?.5 and Ts1ms
• Throughput is 500 packets/sec

34
Littles Law Examples N rT

• How long to answer email
• 50 messages/day
• 150 messages in Inbox
• 3 days for answer
• Hospital
• 2.5 days stay
• 5 new cases/day
• 12.5 in the system
• Average Time in Hospital
• 102.5 average patients
• Discharge 67.5 patients/day
• 1.5 days in hospital
• Customers in store
• 10/hour arrival rate
• Stay on average 0.5 hours
• 5 in store

35
Queueing Theory

• Mathematical study of queues
• Enables analysis of several related processes
• Can calculate several steady-state performance
  measures
• average waiting time
• expected number waiting
• probability of a given number of items in the
  queue
• etc.
• Can model several different types of systems
• Different queueing disciplines
• Different arrival, service, and departure
  probability distributions
• etc.

36
Kendall Notation

• A/S/NS/B/K/SD
• A,SInterarrival time, service time distribution
• M Exponential
• Ek Erlang
• Hk Hyperexponential
• D Deterministic
• NSNumber of servers
• BNumber of Buffers
• KPopulation size
• SDService Discipline
• FCFS,FCLS

37
Kendall Notation

• A/S/NS/B/K/SD
• Defaults
• B ? , number of buffers
• K ? , population size
• SDFCFS service discipline
• M/M/1 M/M/1/ ? / ? /FCFS
• exponential arrival rate and service time
• 1 server
• infinite buffers and infinite population
• FCFS queue

38
M/M/1 queue

39
Markov Models - analyze state n(n buffers are
occupied)

• n1
• n
• n-1

departure
Buffer Occupancy

• n

arrival
0 1 ... n-1
n n1
40
Probability of being in state n
one arrived and one departed
None arrived or departed
None arrived, one departed
one arrived, none departed
41
Markov Chains
42
Substituting Utilization
Rate leaving n rate entering n
43
Substituting P1

• Higher states have decreasing probability
• Higher utilization causes higher probability
• of higher states

44
What about P0
Queue determined by
45
(No Transcript)
46
Single Server
?Ts0
(?Ts)/Ts1
(?Ts)/Tsgtgt1
47
Example

• Given things that you can easily measure
• Arrival rate of 1000 requests/sec
• Service capacity of 1100 requests/sec
• Find
• Utilization
• Probability of having 4 buffers filled

48
Example
49
(No Transcript)
50
Examples

• Web Site with 100 Clients and 1 server.
• Service time .6sec and ?? (exponential)
• What is average response time at 20
  queries/min1q/3sec?
• Use M/M/1 model
• ??Ts(1/3)(.6)0.2
• TrTs/(1-?).6/.8.75sec
• If 1.5 seconds is too long, what utilization is
  allowable (90 of responses are less than 1.5sec)
• mTr(r)Trln(100/(100-r))
• mTr(90)Trln(100/10)Ts/(1-?)2.31.5 sec
• 2.3 Ts(1-?)1.5 sec, 1.5 ? 1.5-2.3.6 , ?.08
• The utilization must actually be reduced

51
Multi-server System with separate queues (DNS
Rotation)

• five processors (5 M/M/1), average service time
  0.1 sec.
• Assume that the standard deviation of service
  time is observed to be 0.094 sec (exponential
  service time).
• Web page hits 40 per second.
• Separate Queue Approach
• If processes are evenly distributed among the
  processors, then the load for each processor is
• 40/5 8 requests per second. Thus,
• r ? Ts 8 ? 0.1 0.8
• The residence time is then easily calculated
• tr Ts/(1- r).1/.2.5 seconds

52
Single Queue - Multiple server

• (M/M/5) aggregate arrival rate of 40 processes
  per second. Utilization is still ?(? Ts /5)
  0.8
• To calculate the residence time first calculate
  Erlang C function, table lookup, ? 0.8 for 5
  servers C 0.554
• Tr (C/N)(Ts/(1- ?)) Ts ((0.544)(0.1))/(5(1-
  0.8)) (0.1) 0.1544
• So the use of multiserver single queue has
  reduced average residence time from 0.5 sec
  (previous slide) down to 0.1544 sec,
• which is greater than a factor of 3.
• If we look at just the waiting time, the single
  queue case is 0.0544 seconds compared to 0.4
  seconds, for multiple queues
• which is a factor of 7.

53
Another Example

• Arrival rate
• ? 125 requests/sec
• Service rate
• ?1/0.002 500 requests/sec
• Gateway utilization
• ? ?/ ?0.25
• Probability of n packets in gateway
• (1- ?) ?n 0.75(.25)n
• Mean time spent in web server
• Ts/(1- ?).002/(1-.25)2.66ms

54
Simulation
55
Workload Models

• Identify Performance Problems
• Problems may only occur under high load
• Benchmark Web Components
• Deployment decisions
• Evaluate new features
• Capacity Planning
• Determine network, memory, disk and clustering
  needs

56
Web Workload Characterization

• Based on the results of numerous studies
• Key properties
• HTTP Message Characteristics
• Several request methods and response codes
• Resource Characteristics
• Diverse content-type, size, popularity, and
  modification frequency
• User Behavior
• User browsing habits significantly affect workload

Category Parameter
Protocol Request Method Response Code
Resource Content type Resource size Response size Popularity Modification freqency Temporal Locality embedded resources
Users Session interarrival times clicks per session Request interarrival times
57
Parameter Characterization

• Associate each parameter with quantitative values
• Statistics
• Mean, median, mode
• OK for parameters that dont vary much
• Probability Distributions
• Capture how a parameter varies over a wide range
  of values

58
Probability Distribution

• Every random variable gives rise to a probability
  distribution
• Probability Density Function
• Assigns a probability to every interval of the
  real numbers
• Cumulative Distribution Function
• Describes the probability distribution of a
  real-valued random variable X
• F(x) P(X lt x)
• The probability that a random variable will be
  less than or equal to x
• In the following slides, we will show the CDF of
  commonly used distributions

59
Poisson Distribution

• F(x) (e-??k)/k!
• Used to model the time between independent events
  that happen at a constant average rate

• The number of times a web server is accessed per
  minute is a Poisson distribution
• For instance, the number of edits per hour
  recorded on Wikipedia's Recent Changes page
  follows an approximately Poisson distribution.

60
Exponential Distribution

• F(x) e-?x
• Used to model the time until the next occurrence
  of an event in a Poisson process

• Session interarrival times are exponential
• Time between the start of one user session and
  the start of the next user session

61
Pareto Distribution

• F(x) (x/a)-k
• k is shape, a is minimum value for x
• Power law
• 80-20 rule
• 20 of the sample is responsible for 80 of the
  results

• Response sizes, Resource sizes, Number of
  embedded images, Request interarrival times
• Often used to model self-similar patterns

62
Probability Distributions in Web Workload Models
Distribution Workload Parameter
Exponential Session interarrival times
Pareto Response Sizes Resource Sizes Number of embedded images Request interarrival times
Lognormal Response sizes Resource sizes Temporal Locality
Zipf-like Resource Popularity
63
Probability Distribution Conversion

• Most languages have random number library
  functions
• Uniform distribution
• Must convert from uniform distribution to the
  chosen distribution

• Given the cumulative distribution function, CDF,
  of the chosen distribution
• 1. Generate a random number call this number p
• 2. Compute x such that CDF(x) p
• Determine the inverse of the CDF
• 3. x is the random number you use

Inverse of the CDF For the exponential
distribution