An Overview of Performance Evaluation

1
An Overview of Performance Evaluation
Simulation
Dr Shamala Subramaniam Dept. Communication
Technology Networks Faculty of Computer Science
IT University Putra Malaysia
2
Overview of Performance Evaluation

• Intro Objective
• The Art of Performance Evaluation
• Professional Organizations, Journals, and
  conferences.
• Performance Projects
• Common Mistakes and How to Avoid Them
• Selection of Techniques and Metrics

3
Intro Objective

• Performance is a key criterion in the design,
  procurement, and use of computer systems.
• Performance ?? Cost
• Thus, computer systems professionals need the
  basic knowledge of performance evaluation
  techniques.

4
Intro Objective

• Objective
• Select appropriate evaluation techniques,
  performance metrics and workloads for a system.
• Conduct performance measurements correctly.
• Use proper statistical techniques to compare
  several alternatives.
• Design measurement and simulation experiments to
  provide the most information with least effort.
• Perform simulations correctly.

5
Modeling

• Model used to describe almost any attempt to
  specify a system under study.
• Everyday connotation physical replica of a
  system.
• Scientific a model is a name given to a
  portrayal of interrelationships of parts of a
  system in precise terms. The portrayal can be
  interpreted in terms of some system attributes
  and is sufficiently detailed to permit study
  under a variety of circumstances and to enable
  the system s future behavior to be predicted.

6
Usage of Models

• Performance evaluation of a transaction
  processing system (Salsburg, 1988)
• A study of the generation and control of forest
  fires in California (Parks, 1964)
• The determination of the optimum labor along a
  continuous assembly line in a factory (Killbridge
  and Webster, 1966)
• An analysis of ship boilers (Tysso, 1979)

7
A Taxonomy of Models

• Predictability
• Deterministic all data and relationships are
  given in certainty. Efficiency of an engine based
  on temperature, load and fuel consumption.
• Stochastic - at least some of the variables
  involved have a value which is made to vary in an
  unpredictable or random fashion. Example
  financial planning.
• Solvability
• Analytical simple
• Simulation complicated or an appropriate
  equation cannot be found.

8
A Taxonomy of Models

• Variability
• Whether time is incorporated into the model
• Static specific time (financial)
• Dynamic any time value (food cycle)
• Granularity
• Granularity of their treatment in time.
• Discrete events clearly some events (packet
  arrival)
• Continuous models impossible to distinguish
  between specific events taking place (trajectory
  of a missile).

9
The Art of Performance Modeling

• There are 3 ways to compare performance of two
  systems
• Table 1.1
• System Workload 1 Workload 2 Average
• A 20 10
  15
• B 10 20
  15

10
The Art of Performance Modeling (cont.)

• Table 1.2 System B as the Base
• System Workload 1 Workload 2 Average
• A 2 0.5
  1.25
• B 1 1
  1

11
The Art of Performance Modeling (cont.)

• Table 1.3 System A as the Base
• System Workload 1 Workload 2 Average
• A 1 1
  1
• B 2 0.5
  1.25

12
The Art of Performance Modeling (cont.)

• Ratio Game

13
Performance Projects
I hear and forget. I see and I remember. I do and
I understand Chinese Proverb
14
Performance Projects

• The best way to learn a subject is to apply the
  concepts to a real-system
• The project should encompass
• Select a computer sub-system a network
  congestion control, security, database, operating
  systems.
• Perform some measurements.
• Analyze the collected data.
• Simulate AND Analytically model the subsystem
• Predict its performance
• Validate the Model.

15
Professional Organizations, Journals and
Conferences

• ACM Sigmetrics Association of Computing
  Machinerys.
• IEEE Computer Society The Institute of
  Electrical and Electronic Engineers (IEEE)
  Computer Society.
• IASTED The International Association of Science
  and Technology for Development (

16
Common Mistakes and How to Avoid Them

• No Goals
• Biased Goals
• Unsystematic Approach
• Analysis without understanding The Problem
• Incorrect Performance Metrics
• Unrepresentative Workloads
• Wrong Evaluation Techniques
• Overlooking Important Parameters
• Ignoring Significant Factors

17
Common Mistakes and How to Avoid Them

• Inappropriate Experimental Design
• Inappropriate Level of Detail
• No Analysis
• Erroneous Analysis
• No Sensitivity Analysis
• Ignoring Errors in Input
• Improper Treatment of Outliers
• Assuming No Change in the Future
• Ignoring Variability

18
Common Mistakes and How to Avoid Them

• Too Complex Analysis
• Improper Presentation of Results
• Ignoring Social Aspects
• Omitting Assumptions and Limitations.

19
A Systematic Approach

• State Goals and Define the System
• List Services and Outcomes
• Select Metrics
• List Parameters
• Select Factors to Study
• Select Evaluation Technique
• Select Workload
• Design Experiments
• Analyze and Interpret Data
• Present Results