Benchmarks

1
Benchmarks
?1998 Morgan Kaufmann Publishers

• Performance best determined by running a real
  application
• Use programs typical of expected workload
• Or, typical of expected class of
  applications e.g., compilers/editors, scientific
  applications, graphics, etc.
• Small benchmarks
• Nice for architects and designers
• Easy to standardize
• Can be abused
• SPEC (System Performance Evaluation Cooperative)
• Companies have agreed on a set of real program
  and inputs
• Can still be abused (Intels other bug)
• Valuable indicator of performance (and compiler
  technology)

2
SPEC System Performance Evaluation Cooperative

• First Round 1989
• 10 programs yielding a single number
  (SPECmarks)
• Second Round 1992
• SPECInt92 (6 integer programs) and SPECfp92 (14
  floating point programs)
• Compiler Flags unlimited. March 93 of DEC 4000
  Model 610
• spice unix.c/def(sysv,has_bcopy,bcopy(a,b,c)
  memcpy(b,a,c)
• wave5 /ali(all,dcomnat)/aga/ur4/ur200
• nasa7 /norecu/aga/ur4/ur2200/lcblas
• Third Round 1995
• new set of programs SPECint95 (8 integer
  programs) and SPECfp95 (10 floating point)
• benchmarks useful for 3 years
• Single flag setting for all programs
  SPECint_base95, SPECfp_base95

3
SPEC First Round

• One program 99 of time in single line of code
• New front-end compiler could improve dramatically

4
Impact of Means on SPECmark89 for IBM 550

• Ratio to VAX Time
• Program Before After Before After
• gcc 30 29 49 51
• espresso 35 34 34 65 67
• spice 47 47 510 510
• doduc 46 49 41 38
• nasa7 78 144 258 140
• li 34 34 183 183
• eqntott 40 40 28 28
• matrix300 78 730 58 6
• fpppp 90 87 34 35
• tomcatv 33 138 20 19
• Mean 54 72 124 108
• Geometric
• Ratio 1.33 Ratio 1.16

5
SPEC 95
6
SPEC 95

• Does doubling the clock rate double the
  performance?
• Can a machine with a slower clock rate have
  better performance?

7
Performance Evaluation

• For better or worse, benchmarks shape a field
• Good products created when have
• Good benchmarks
• Good ways to summarize performance
• Given sales is a function in part of performance
  relative to competition, investment in improving
  product as reported by performance summary
• If benchmarks/summary inadequate, then choose
  between improving product for real programs vs.
  improving product to get more salesSales almost
  always wins!
• Execution time is the measure of computer
  performance!

8
Fallacies and Pitfalls

• Hardware-independent metrics can predict
  performance

9
Amdahls Law
Performance with enhancement Performance
without enhancement
SU SpeedUpOverall
1
SU
FranctionEnhanced SpeedUpEnhanced
(1 - FractionEnhanced)
Example An enhancement that runs 10x faster 40
of the time SU 1/((1 - .4) .4/10) 1/.64
So the speedup is about 1.56 times
10
Amdahl's Law

• Speedup due to enhancement E
• ExTime w/o E
  Performance w/ E
• Speedup(E) -------------
  -------------------
• ExTime w/ E Performance w/o
  E
• Suppose that enhancement E accelerates a fraction
  F of the task by a factor S, and the remainder of
  the task is unaffected

11
Homework Problem 2

• Text problem 1.7 (due next class)
• We will talk about the solution.

12
Solution
13
1.7a
14
1.7b
15
1.7c

• NonFp instructions which are not floating point
• FP instructions which are part of a floating
  point computation
• nonfp total fp (operations for hardware)
• 1.8 10 6 195578 1.6 106
• For Software
• fp 3.8 107 1.6 106 3.6 107
• Inst per operation 3.6 107 / 195578 185

16
1.7 Mflops

• Mflops 195,578/(1.08 106 )
• 0.18 mflops

17
Observations About Computer Design

• Make the common case fast
• Understand Amdahls law
• Performance improvement gained from using some
  faster mode of execution is limited by fraction
  of the time you use the faster mode
• Exploit locality of reference as applied to
  both code and data
• Rule of thumb
  
  90 of the execution time is spent in only
  10 of the code
• Temporal locality
• Something just recently used will be reused again
  shortly
• Spatial locality
• Items near one another will be referenced together

18
Computer Architecture Is

• The attributes of a computing system as seen
  by the programmer, i.E., The conceptual structure
  and functional behavior, as distinct from the
  organization of the data flows and controls the
  logic design, and the physical implementation.
• Amdahl, Blaaw, and brooks, 1964

SOFTWARE
19
Computer Architectures Changing Definition

• 1950s to 1960s computer architecture course
  computer arithmetic
• 1970s to mid 1980s computer architecture course
  instruction set design, especially ISA
  appropriate for compilers
• 1990s computer architecture coursedesign of
  CPU, memory system, I/O system, multiprocessors

20
Instruction Set Architecture (ISA)
software
instruction set
hardware