Performance Evaluation

1
Performance Evaluation

• Operating Systems
• Fall 2002

2
Performance evaluation

• There are several approaches for implementing the
  same OS functionality
• Different scheduling algorithms
• Different memory management schemes
• Performance evaluation deals with the question
  how to compare wellness of different approaches
• Metrics, methods for evaluating metrics

3
Performance Metrics

• What is the performance metric for the sorting
  algorithms?

• Is something wrong with the following statement
• The complexity of my OS is O(n)?

• This statement is inherently flawed
• The reason OS is a reactive program

4
Performance metrics

• Response time
• Throughput
• Utilization
• Other metrics
• Mean Time Between Failures (MTBF)
• Supportable load

5
Response time

• The time interval between a users request and
  the system response
• Response time, reaction time, turnaround time,
  etc.
• Small response time is good
• For the user waiting less
• For the system free to do other things

6
Throughput

• Number of work units done per time unit
• Applications being run, files transferred, etc.
• High throughput is good
• For the system was able to serve many clients
• For the user might imply worse service

7
Utilization

• Percentage of time the system is busy servicing
  clients
• Important for expensive shared system
• Less important (if at all)
• for single user systems, for real time systems
• Utilization and response time are interrelated
• Very high utilization may negatively affect
  response time

8
Performance evaluation methods

• Mathematical analysis
• Based on a rigorous mathematical model
• Simulation
• Simulate the system operation (usually only small
  parts thereof)
• Measurement
• Implement the system in full and measure its
  performance directly

9
Analysis Pros and Cons

• Provides the best insight into the effects of
  different parameters and their interaction
• Is it better to configure the system with one
  fast disk or with two slow disks?
• Can be done before the system is built and takes
  a short time
• Rarely accurate
• Depends on host of simplifying assumptions

10
Simulation Pros and Cons

• Flexibility full control of
• Simulation model, parameters,
• Level of detail
• Disk average seek time vs. acceleration and
  stabilization of the head
• Can be done before the system is built
• Simulation of a full system is infeasible
• Simulation of the system parts does not take
  everything into account

11
Measurements Pros and Cons

• The most convincing
• Effects of varying parameter values cannot (if at
  all) be easily isolated
• Often confused with random changes in the
  environment
• High cost
• Implement the system in full, buy hardware

12
The bottom line

• Simulation is the most widely used technique
• Combination of techniques
• Never trust the results produced by the single
  method
• Validate with another one
• E.g., simulation analysis, simulation
  measurements, etc.

13
Workload

• Workload is the sequence of things to do
• Sequence of jobs submitted to the system
• Arrival time, resources needed
• File system Sequence of I/O operations
• Number of bytes to access
• Workload is the input of the reactive system
• The system performance depends on the workload

14
Workload analysis

• Workload modeling
• Use past measurements to create a model
• E.g., fit them into a distribution
• Analysis, simulation, measurement
• Recorded workload
• Use past workload directly to drive evaluation
• Simulation, measurement

15
Statistical characterization

• Every workload item is sampled at random from the
  distribution of some random variable
• Workload is characterized by a distribution
• E.g., take all possible job times and fit them to
  a distribution

16
The Exponential Distribution

• A lot of low values and a few high values
• The distributions of salaries, lifetimes, and
  waiting times are often fit the exponential
  distribution
• The distribution of
• Job runtimes
• Job inter-arrival times
• File sizes

17
Exponential Probability Density Function (pdf)

• If X has an exponential distribution with
  parameter , then its probability density
  function is given by
• where

18
The Exponential Distribution
19
Mean and Variance

• If XExponential( ), then its mean and variance
  are given by

20
Exponential Probabilities
f(x)
f(x)
x
x
a
a
0
0
21
Cumulative Distribution Function

• The cumulative distribution function (cdf), F(x),
  is defined as
• For the Exponential distribution

22
The Exponential CDF
23
Memoryless Property

• The exponential is the only distribution with the
  property that
• For modeling runtimes the probability to run for
  additional b time units is the same regardless of
  how much the process has been running already
• in average

24
Fat-tailed distribution

• The real life workloads frequently do not fit the
  exponential distribution
• Fat-tailed distributions

25
Pareto Distribution

• The more you wait, the more additional time you
  should expect to wait
• The longer a job has been running, the longer
  additional time it is expected to run

26
Pareto dist. CDF
27
Exponential vs Pareto
28
Queuing Systems
queue
Disk A
queue
Disk B
new jobs
finished jobs
CPU
queue

• Computing system can be viewed as a network of
  queues and servers
• Shared queues are also possible

29
The role of randomness

• Arrival (departure) are random processes
• Deviations from the average are possible
• The deviation probabilities depend on the
  inter-arrival time distribution
• Randomness makes you wait in queue
• Each job takes exactly 100ms to complete
• If jobs arrive each 100ms exactly, utilization is
  100
• But what if both these values are on average?

30
Queuing analysis
server
queue
departing jobs
arriving jobs
31
Littles Law
32
M/M/1 queue
server
queue
departing jobs
arriving jobs
Both interarrival time and service time are
exponentially distributed M stands for memoryless
33
How average response time depends on utilization?

• The job arrival and departure are approximated by
  Poisson processes
• The distribution of the number of jobs in the
  system in the steady state is unique
• Use queuing analysis to determine this
  distribution
• Once it is known, can be found
• Use the Little law to determine

34
M/M/1 queue analysis
35
(No Transcript)
36
Response time (utilization)
37
A Bank or a Supermarket?
departing jobs
CPU1
CPU1
shared queue
departing jobs
CPU2
CPU2
arriving jobs
arriving jobs
departing jobs
CPU3
CPU3
departing jobs
CPU4
CPU4
M/M/4
4 x M/M/1
38
It is a Bank!
39
Summary

• What are the three main performance evaluation
  metrics?
• What are the three main performance evaluation
  techniques?
• What is the most important thing for performance
  evaluation?
• Which workload models do you know?
• What does make you to wait in queue?
• How response time depends on utilization?

40
To read more

• Notes
• Stallings, Appendix A
• Raj Jain, The Art of Computer Performance Analysis

41
Next

• Processes