TDC 369 TDC 432

1
TDC 369 / TDC 432

• Spring 2003
• Greg Brewster
• DePaul University

2
Agenda

• Course Overview
• What is performance analysis?
• Methods for performance analysis
• Measurements
• Simulation
• Analysis

3
Course Goals

• Study analytical tools for performance analysis
  of web-based computer networks and systems.

4
Course Text

• Required Capacity Planning for Web Services
  Metrics, Models and Methods, Menasce Almeida,
  Prentice-Hall, 2002.
• Excellent coverage of performance analysis issues
  for web services and web servers
• For more math Queueing Networks and Markov
  Chains Modeling and Performance Evaluation with
  Computer Science Applications, Bolch, Greiner, de
  Meer and Trivedi, Wiley, 1998.
• A good reference book covering mathematics and
  analysis aspects of the course in more detail

5
Office Hours and Web Site

• Course Web Site
• http//dlweb.cs.depaul.edu
• Class Times MW 1150 AM 120 PM
• Ill be available in my office for 1.5 hours
  after each class
• Office Hours MW 130 300 PM

6
Course Web Site and Handouts

• Class presentation handouts (PPT slides) will be
  available on the web site
• Monday class slides on web site by 300 PM on
  Friday
• Wednesday class slides on web site by 800 AM on
  Tuesday
• It is your responsibility to print out copies of
  PPT slides for your own note-taking before coming
  to each class session.

7
Performance Analysis

• Our goal is to determine values for performance
  attributes of systems, such as
• Delay (Response Time)
• Elapsed time between two events
• Throughput
• Events per second sustained by a system
• Utilization
• Fraction of time a system resource is busy
• Reliability
• Fraction of time a system resource is available

8
Example Web Server Performance

• Response Time
• What is the delay between the time a user clicks
  on a hyperlink and the time the web page is
  displayed?
• Throughput
• How many web page accesses per second can the
  server support?
• Utilization
• What fraction of time is the server busy?
• Reliability
• How often is the server down?

9
Applications of Performance Analysis Results

• Capacity Planning
• Predicting the resources needed to provide some
  Quality of Service in an existing system
• Systems Design
• Evaluate design tradeoffs for components of a
  system to be constructed
• Systems Research
• Quantify the value of proposed new protocols, etc.

10
Sample Applications

• How many servers does Amazon.com need to provide
  average search times of less than 10 seconds?
• How much faster would DePauls web access be if
  we doubled the bandwidth to our ISP?
• Can performance of the TCP protocol be improved?

11
Capacity Planning

• Capacity Planning Question Given current system
  performance and demand, how much can the demand
  increase before
• the system becomes saturated
• Saturation occurs when some vital system resource
  nears exhaustion example network utilization gt
  95.
• or some critical system service level is exceeded
• Example user response time exceeds 5 seconds

12
Capacity Planning Example

• AnyCo runs an Intranet server which currently
  handles about 60 requests per second and gives
  acceptable performance.
• Management would like to hire more employees,
  which will increase requests/sec.
• Key Service Level Server response time must be
  kept under 8 seconds

13
AnyCo Reponse Time Analysis
14
Response Time Results

• AnyCo finds that the web server, as currently
  configured, can handle up to about 150 requests
  per second before response times become
  unacceptable.
• Most response time graphs have a knee value for
  the graph, where response time starts to increase
  rapidly with increasing load
• For this example, the knee of the curve occurs at
  around 130 requests/second.

15
Server CPU Utilization Analysis
16
Utilization Results

• CPU utilization defines the fraction of the time
  that the CPU is busy processing requests.
• AnyCo finds that the web CPU utilization
  increases linearly with requests up to about 190
  requests per second, when it reaches maximum
  utilization (100).
• This suggests that CPU utilization represents a
  bottleneck resource for this system and that
  increasing the server CPU speed (or installing
  multiple processors or multiple servers) would be
  necessary to increase system capacity beyond 190
  requests per second.

17
AnyCo Throughput Analysis
18
Throughput Results

• Results show that all requests are processed
  (that is, throughput offered load) up to around
  185 requests per second.
• Past 185 requests per second, the system starts
  to drop requests (throughput lt offered load),
  indicating that the system capacity has been
  exceeded.

19
Performance Analysis Approaches

• Systems Measurement
• Simulation
• Analytical Modeling

20
Comparison
From Tools for Designing Data Networks, Al
Edlund, Data Communications Review, Dec. 1999.
21
Systems Measurement

• Idea Directly measure the performance of an
  existing physical system
• Tools
• Hardware analyzers
• Network sniffers
• Metered software

22
Systems Measurement

• Advantage
• Highly accurate performance results
• Disadvantages
• System must exist and be measureable
• Cannot measure proposed systems or modifications
  to existing systems
• Measurement may interfere with system operation
• Generalizeable? Results may apply only to
  particular time period measured

23
Measurements Marketplace

• Measurement Tools
• Traffic generators, Sniffers, probes, etc.
• Network General, Computer Associates, Microsoft,
  etc
• Measurement Services
• Companies will generate traffic to test your web
  site
• Gomez, Keynote Systems, Matrix NetSystems

24
Simulation

• Idea Utilize simulation software that calculates
  progressive system states and measures
  resulting performance values
• Tools
• Simulation packages, e.g. COMNET
• Custom software

25
Simulation

• Advantages
• Can measure performance of systems that do not
  yet exist
• Accurate results if specifications known
• Disadvantages
• Software is complex to write and debug
• Simplifying assumptions are typically required
• Simulation must be run for thousands or millions
  of iterations for accurate results, which may
  require hours of simulation execution time.

26
Simulation Marketplace

• Simulation Packages
• NS (network simulator)
• Free software from Berkeley
• OPNET
• www.opnet.com
• COMNET
• www.compuware.com

27
Analytical Models

• Idea Represent system performance via
  mathematical model that can be solved directly
• Tools
• Probability Theory
• Queueing Theory
• Model solution packages, e.g. GreatSPN, PMVA
• Mathematical software, e.g. MatLab or Mathematica

28
Analytical Models

• Advantages
• Can analyze systems that do not yet exist
• Results are derived much more quickly compared
  with simulation
• Disadvantages
• May be difficult to determine appropriate models
  to use
• Simplifying assumptions are used (more than
  simulation) to ensure that solution is
  mathematically feasible

29
Analysis Marketplace

• Special purpose software packages exist
• MMNK software for queue delay analysis
• MVA software for queueing network analysis
• Petri Net software packages
• Markov chain solvers
• Once a model is understood, some modeling and
  analysis can be done with basic tools, such as
• Excel
• Mathematica
• Companies will sell you analysis services
• NetForeCast.com and others

30
Example AnyCo Web Server

• AnyCo runs a web server and would like to
  support up to 100 users performing transactions
  on this server at any one time
• Key criterion User response time must be less
  than 8 seconds

31
AnyCo Web Server Example
32
AnyCo Web Server

• AnyCo workload characterization
• Each active user reads a web page for an average
  of 5 seconds before clicking a new hyperlink
• HTTP Request message length averages 200 bytes
• HTTP Responses (web pages) average 40,000 bytes
• Each request requires
• 80 ms. of Web server CPU time
• 3 disk accesses avg. 50 ms. per access

33
AnyCo Web Server Systems Measurement Approach

• AnyCo sets up test terminals that generate
  transactions on the web server.
• They generate traffic equivalent to 1, 10, 20,
  30, , 100 users using test terminals and measure
  response times.
• Each set of measurements is repeated many times
  to ensure a statistically significant result.

34
AnyCo Measurements
35
AnyCo Web Server Example Systems Measurement
Approach

• Result Response time exceeds 8 seconds for more
  than 60 users
• Follow-Up
• Look for causes of unacceptable performance
• Detailed server code measurements
• Network utilization measurements
• Problem How to predict impact of any system
  modifications?

36
AnyCo Web Server Systems Measurement Approach

• Effort and Expense are High
• Test measurement software must be purchased and
  configured on test workstations
• Server must be off-line and available for
  measurement
• Network Sniffer or probe must be installed to
  capture performance information

37
AnyCo Web Server Simulation Approach

• AnyCo creates a detailed simulation of their web
  server system.
• Key system parameters are used as inputs to the
  simulation
• Data packet sizes, overheads
• Router delays,
• CPU time required for various operations
• Setting up / writing simulation may take many
  hours or days depending on software used and
  complexity of simulation.

38
AnyCo Web Server Simulation Approach

• Assumptions
• Delay components
• ISP router, T1 access line, AnyCo router,
  Ethernet, Web Server CPU time, Web Server disk
  access times, Ethernet, AnyCo router, T1 access
  line, ISP router.
• Since we do not know details of user access to
  the Internet, we do not include delay for
  transporting data over user access lines
• Router processing times negligible (only include
  data transmission times)
• Ethernet transmission times negligible

39
AnyCo Web Server Simulation Approach

• Simulation is now run
• Each point on performance graph requires one
  simulation run.
• Simulation run must iterate thousands or millions
  of times to get a statistically valid result.
• Each simulation run may typically take 10 minutes
  to one hour to run.

40
AnyCo Web Server Simulation Model
Example Graphical model, simulation and results
analysis using COMNET software.
41
AnyCo Web Server Simulation Approach

• Model Accuracy How accurately does this
  simulation model capture the behavior of the real
  system??
• Must be addressed via model validation study
  where simulation results are compared against
  measured system results.
• Follow-Up After examining simulation results, we
  can modify parameters and do what-if analysis for
  system changes.

42
AnyCo Web Server Analysis Approach

• AnyCo sets up a simple analytical model of their
  web server system.
• Key system parameters are used as inputs to the
  simulation
• Data packet sizes, overheads
• Router delays,
• CPU time required for various operations
• Model can typically be set up quickly, as it only
  captures the few essential performance attributes
  of the system

43
AnyCo Web Server Analysis Approach

• Assumptions
• Delay components
• ISP router, T1 access line, AnyCo router,
  Ethernet, Web Server CPU time, Web Server disk
  access times, Ethernet, AnyCo router, T1 access
  line, ISP router.
• Since we do not know details of user access to
  the Internet, we do not include delay for
  transporting data over user access lines
• Router processing times negligible (only include
  data transmission times)
• Ethernet transmission times negligible
• There are some restrictions on distributions of
  service times and request routing (details later
  in course)

44
AnyCo Web Server Analytical Approach

• Analytical Solution package is run
• Each point on performance graph requires one
  solution of an analytical model.
• Each analytical model solution is exact (just 1
  iteration) and takes about 0.1 second.
• All graph data generated in less than 1 second!!

45
Analytical Models

• The fundamental building block for many
  analytical models is Queueing Theory

46
Queueing Theory

• Queueing Theory deals with the behavior of jobs
  arriving to queues connected to servers.

47
Queue Operations

• Jobs arrive according an arrival process, A(t).
• Arriving jobs are placed in queue
• Server provides service to first job in queue,
  other jobs wait their turn.
• Service time required by each job determined by
  service process, S(t)
• Each job leaves queue after receiving service.

48
AnyCo Web Server Example
49
AnyCo Queueing Network Model
50
AnyCo Web Server Queueing Network Modelmore
details later in course!!

• Each time a user clicks a hyperlink
• Request message sent by ISP router across T1
  (0.001 s) and across Ethernet
• Web server processes request using 3 disk
  accesses (0.05 s. each) interspersed with 4 CPU
  time slots (0.02 s. each 0.08 s.)
• Response message sent across Ethernet and by
  AnyCo router across T1 (0.02 s)
• User examines result for average of 5 sec before
  making next request.

51
AnyCo Web Server Sample of PMVA analysis input
file

• network
• servers
• users is li 5.0000,
• isprtr fcfs li 0.0010,
• corprtr fcfs li 0.2080,
• servercpu ps li 0.0200,
• serverdisk fcfs li 0.0500
• routing
• users gt isprtr gt servercpu,
• servercpu gt serverdisk (0.7500) corprtr
  (0.2500),
• serverdisk gt servercpu,
• corprtr gt users
• end

52
AnyCo Analytical Model Results
53
AnyCo Analytical Model Results
54
AnyCo Web Server Analytical Approach

• Result of Analysis The ISP Access Line is
  overloaded!!
• Solution Increase Bandwidth on Access Line
• Current ISP Access T1 (1.536 Mbps)
• New Model Dual T1 (3.072 Mbps)

55
AnyCo Analytical Model Results
56
AnyCo Web Server Analytical Approach

• Modified System (with Dual T1 Access Line)
• Performance is greatly enhanced.
• System can now handle up to 85 users before
  exceeding 8 second response time

57
AnyCo Analytical Model Results
58
AnyCo Web Server Analytical Approach

• Result of Analysis
• ISP Access Line is no longer a bottleneck
• At high loads, the Web Server Disk is overloaded
• Possible Solution Upgrade Server Disk drive
• Current Disk 50 ms. per disk access
• Upgraded Disk 30 ms. per disk access

59
AnyCo Analytical Model Results
60
AnyCo Web Server Analytical Approach

• Modified System
• Performance is greatly enhanced.
• System can now easily handle the required 100
  users with calculated average user response time
  of around 4 seconds.
• Recommendations to AnyCo
• Increase bandwidth on ISP access line
• Upgrade server disk drive

61
AnyCo Web Server Analytical Approach

• Key Question How accurately does this simple
  analytical model capture the behavior of the real
  web server system??
• Must be addressed via model validation study
  where analytical model results are compared
  against measured system results.
• Note even if model is not perfectly accurate, it
  can still help to predict performance trends.