System Performance

1
System Performance

• You understand this if you can
• Discuss use of modelling in evaluating
  performance
• Compare methods for modelling workload
• Carry out simple analyses using Flow Balancing
• Read key papers on analytic modelling
• http//www.cs.gmu.edu/cne/pjd/PUBS/amsci-qn.pdf
• http//www.cs.gmu.edu/cne/pjd/PUBS/csurv-opanal.p
  df

2
Why Measure Performance?

• Prediction
• New system
• Evaluation
• Assess level of performance quality
• Timing of upgrade decisions
• Purchasing decisions
• Example
• Web Server
• Performance important
• Many potentially relevant factors to consider

3
Ways to Measure Performance

• Measurement of real system
• Live monitoring real system real users
• Simulated workload real system pretend users
• Simulation
• Software model of system
• Analytic Modelling
• Equations relating key features

4
Features and Results
Output (Dependent variables) What you want to
measure Response time Throughput Reliability Res
ults Average behaviour, Worst case
behaviour, Graphs how dependent variables change
with independent ones Bottlenecks
Input (Independent variables) Affect the
situation Number of users Types of user Usage
pattern Frequency Size of pages Distribution Netwo
rk System Configuration
System
5
Workload Modelling

• Providing appropriate load on the system
• Trace
• Can use logs
• Statistical
• Measure (a lot of) input over time
• Identify a relevant statistical distribution
• Generate random numbers from distribution

You cant tell that an individual random number
has been generated from a specific distribution,
but when you generate enough of them the
frequency of different values will approximate
the distribution curve.
A typical exponential distribution is the time
between the submission of jobs to a server.
A typical normal distribution would be the height
of individuals of the same sex. The height of a
mixed group of people would be bi-modal two
peaks A computing example might be the size of
documents to be printed.
frequency
frequency
Normal Distribution
Exponential Distribution
value
value
6
Workload Modelling Issues

• Need a lot of input data
• Need many experiments to find averages
• Use real data?
• Logs gathered anyway
• May need filtering to remove irrelevant events
• Use statistical data?
• Can generate any quantity of data
• Likely to give reliable results
• Not sensitive to distribution details

7
Workload Issues

• Can measure real system with artificial load
• Use LAN multiple PCs to generate load
• Scripts to generate requests
• simulated users based on trace
• Can get unexpected interference
• E.g. LAN protocol can affect the delivery
• Actual load on server can differ from expected
• Use threads on PC to simulate many users
• Interactions among threads may reduce load on
  system
• Bugs in script

8
Analytic Modelling

• Equations linking relevant features
• Dependent independent variables
• May be able to solve equations
• May have to use numerical approximations
• Operational approach to analytic modelling
• Only interested in measurable quantities
• Number of arrivals in measured time
• And quantities derived from them
• Arrival rate

9
Queuing Theory

• Used since 1970s
• System is a network of servers queues

T duration of observation B server busy time
during observation
C completed jobs during observation X C/T
output rate, jobs/second
Server
Queue

• A/T arrival rate, jobs/second
• A arrivals during observation period

U B/T, utilization, fraction of time server
busy S B/C, average service time
10
Web Application
If one server is particularly slow, it will be a
bottleneck. The overall performance will depend
on it. May be able to simplify analysis of the
system to a simple queue server.
11
Transient or Steady State?

• Transient
• Things are changing
• E.g. at start up or when load changes
• Steady State
• Things have settled down
• Some fluctuations but ups downs average out
• No build-up or decline in queues
• Number in number out
• Arrivals completions

12
Laws and Assumptions

• Laws
• Essentially true for all systems
• Some statistical fluctuations
• True over long periods of time
• Assumptions
• Reasonable assertions
• True for many situations
• Also affected by fluctuations

13
Definitions

• Service Time
• Time for server to handle a job
• Average is derived from measurable quantities

C completed jobs during observation X C/T
output rate, jobs/second
Server
Queue
T duration of observation B server busy time
during observation

• A/T arrival rate, jobs/second
• A arrivals during observation period

U B/T, utilization, fraction of time server
busy S B/C, average service time
14
Utilization Law

• From the definitions
• U B/T (B/C) (C/T) S X
• Example
• Server delivers 50 pages per second
• Each page requires 1/100 second server time
• Every second, system busy for 50/100 seconds
• Utilisation is 50
• That is, the server is idle for half the time

15
Utilization and Response Time

• Utilization increases, response time grows
• Think of a checkout at a shop
• As number of shoppers increase,
• Checkout will be kept busy
• Utilization increases
• Queue is likely to grow
• Checkout likely to be busy when you get there
• So, overall wait, the response time, will grow
• Maximum utilization 100
• If jobs arrive faster than they can be handled
• Queue length is unbounded

16
Assumption Flow Balance

• A systems input rate its output rate
• Steady state
• On average, no overall build-up of jobs
• Not true over very short times
• Since a small number of jobs may be stored
• Input (A) gt Output (C)

17
Applying Flow Balance

• Consider N threads accessing a database
• Each thread
• Repeat
• Make request
• Wait for response (R seconds) depend on
  database
• Calculate (C seconds) depends on thread
• Forever
• Database
• Processes X requests per second
• But waiting time (R) depends on size of queue, too

18
Applying Flow Balance 2

• A thread makes a request every (RC) sec.
• Database handles X requests per second
• In time T
• Each thread submits T/(RC) requests
• Jobs in NT/(RC)
• The database handles TX jobs Jobs out
• NT/(RC) TX using flow balance assumption
• R N/X C
• Can determine R given N
• Can estimate N given maximum desirable R.

19
Threads Server
Time T
Wait (R)
Calc
Wait (R)
Calc
Wait (R)
Calc
N threads
submit
response
Wait (R)
Calc
Wait (R)
Calc
Wait (R)
Calc
N T/(CR) jobs
Server
X T jobs
20
Overall Response Time
Measurements at fixed load
Average of Measured response time
Response Time (R)
Minimum response time, no contention One job,
straight through
Response time in steady state, with contention
waiting in queues R N/X - Z
Load N number of users
21
Assumptions 2

• Size - independent parameters
• Even if more input is generated, the time taken
  by a server to process a request once it has
  started doesnt change
• Allows us to make predictions
• E.g. if load changes, can assume certain values
  remain constant allows us to estimate others

22
Uses of Analytic Modelling

• Back of the envelope assessment
• Quick estimates of performance
• New systems
• Ensure system is plausible
• Avoid throwing money at impossible tasks
• Explore effect of load changes
• What will be the bottleneck
• What performance can we expect?
• Checking measurements experiments
• Check performance of other system models
• Is a simulation program realistic?
• See Benchmarking Blunders and Things That Go Bump
  in the Night
• Neil J. Gunther http//arxiv.org/pdf/cs/0404043

23
Summary

• Performance Measurement
• Evaluation and prediction capacity planning
• Simulation
• Program that mimics behaviour of components
• Need to prepare appropriate workload model
• Analytic Modelling
• Equations link dependent independent variables
• Bottleneck analysis can simplify complex systems
• Relatively simple models can give useful results

24
Littles Law

• N ? T
• where
• N is the number of jobs in the queuing system
  (queue plus server)
• ? is the arrival rate (the number of jobs that
  arrive to the system per unit of time), and
• T is the Response Time for a job (total time from
  arrival to departure from the system).
• T Queuing Time Service Time

25
Applying Littles Law
Subsystem 1

• X Throughput jobs per second
• N Number of terminals (users)
• Z User average think time
• R Average system response time
• Nthink average number of users thinking
• Nwait average number of users waiting
• Apply Littles Law to subsystem 1
• Nthink XZ
• Apply Littles Law to Server
• Nwait XR
• N Nthink Nwait XZ XR
• N/X Z R
• R N/X - Z

Nthink
This is the same result as for the threads and
database
Similar systems Similar model Different proof of
response law
Subsystem 1 X jobs per sec N jobs
(thinking) Z time to process job
X
Server (Nwait)