FS T S: GS S GSO

1
????? ?????F?????S??T??? ??S ????G????S
?????????????S ?????G?S?O?
Benchmarks
?a??µe???? ?a????p? ?402
2
??t??s? ap?d?s?? ?p?????st??
?p?????st?? ?
?p?????st?? ?
???????te???

• ????da µ?t??s?? ??????
• ?????? ap????s?? (response time, execution time)
  ? ?????? µeta?? t?? a???? ?a? t?? ????????s??
  µ?a? µ??ad???? e??as?a?
• ???µ?? d?eµeta????? (throughput) ? s???????
  p?s?t?ta d???e??? p?? ????e se ??a ded?µ???
  ??????? d??st?µa

?p?????st?? ? e??a? n f???? ???????te??? ap?
?p?????st? ?
3
???p?? µ?t??s?? t?? ap?d?s??
?fa?µ??? ?eta???tt?st?? S?st?µ a
?pa?t?se?? a?? µ??a ???s?µe? ?e?t?????e? a?? de?t
/pt?
(millions) of Instructions per second
MIPS (millions) of (Floating Point) operations
per second MFLOP/s
??? . S?????? ??t????
S?st?µa ???µ?? S?st?µ a d?as??des?? ????de? ??s
. / ?? . ?pe?e??ast??
Megabytes per second CPU time Cycles per second
(s??? . ?e?t?????a?)
4
?as???? ?????e?
CPI

• CPI (Cycles Per Instruction) µ?s?? a???µ??
  ?????? a?? e?t???
• CPI (CPU Time x Clock Rate) / Instruction Count
• Clock Cycles / Instruction Count
• ?? CPI(j) p??pe? ?a µet???e? ??at? e?a?t?ta?, ???
  µ??? ap? t?? epe?e??ast? , a??? ?a? ap? t?
  s?st?µa µ??µ??

?p?d?s? epe?e??ast?
CPU time ?????? e?t??es?? e??? p?????µµat?? CPU
time ?????? e?t??es?? t?? p?????µµat?? x
pe???d?? ???????? CPU time ?????? e?t??es??
t?? p???? . / s????t?ta ???????? CPU time
e?t???? p???? . x CPI x pe???d?? ???????? CPI
µ?s?? a???µ?? ?????? a?? e?t???
5
?as???? ?????e?

• MIPS million instructions per second
• G?a ??a d?sµ??? p????aµµa

• Clock rate ??e? ???µ?ste? ??a µ?a µ??a??
• CPI e??a? ??a? µ????? a???µ?? (se a?t??es? µe
  t?? a???µ? e?t???? ? t? ????? e?t??es??)
• ?????? ?a ?p?????ste? ?a? ?a ???e? ?ata???t??.

• ????
• MIPS e?a?t?ta? ap? t? s????? e?t????, ?p?te
  e??a? d?s???? ?a s??????e?? t?? MIPS ?p?????st??
  µe d?af??et??? s????a e?t????
• MIPS d?af??e? µeta?? p????aµµ?t?? st?? ?d??
  ?p?????st?
• MIPS µp??e? ?a d?af??e? a?t?st??f?? ap? t??
  ap?d?s?

6
??t?a s?????s?? ?a? s???pt???? ap?d?s??

• S???????? ?????? e?t??es?? (Total Execution Time)
• ????µ?t???? µ?s?? µ?s?? ???? ?????? e?t??es??
• ?p?? Timei ? ?????? e?t??es?? t?? ?-?st??
  p?????µµat?? ap? ta n s??????? t?? e??as?a?
• ?s?????sµ???? ?????? e?t??es?? (Weighted
  Execution Time)
• ??t?st?????eta? se ???e p????aµµa ??a s??te?est?
  ßa??t?ta? (weighting factor Wi) a?????a µe t??
  s????t?ta p?? eµfa???eta? st?? s??????? e??as?a.
• ?s?????sµ???? a???µ?t???? µ?s??

?p?? Weighti ? s????t?ta eµf???s?? t?? ?-?st??
p?????µµat?? st? s??????? e??as?a ?a?
Timei ? ?????? e?t??es?? t??
7
?a??????p???µ???? ?????? e?t??es?? (Normalized
Execution Time)

• ?a??????p???s? ?????? e?t??es?? se µ?a µ??a??
  a?af???? ?a? ?p?????sµ?? t?? µ?s?? ???? a?t?? t??
  ?????? (?e?µet????? µ?s??)
• Ge?µet????? µ?s??

?p?? Execution time ratioi e??a? ? ??????
e?t??es??, ?a??????p???µ???? st? µ??a?? a?af????
??a t? ?-?st? p????aµµa ap? ta n s??????? t??
e??as?a?
?.?

• ? t?µ? t?? a???µ?t???? µ?s?? e?a?t?ta? ap? µ??a??
  a?af????
• ? t?µ? t?? ?e?µet????? µ?s?? e??a? a?e???t?t? ap?
  ?a??????p???s? se d?af??et???? µ??a???

• ? ?a? ? ????? t?? ?d?a ap?d?s?, ? ??????
  e?t??es?? t?? C e??a? 0.63 f???? t?? ? ? t?? ?
  (1/1.58)
• ? s???????? ?????? e?t??es?? t?? ? e??a? 10 f????
  µe?a??te??? ap? t?? ?, ? ?p???? e??a? µe t? se???
  t?? 3 f???? a???te??? ap? t?? C

8
?at?????e? p????aµµ?t?? ??a e?t?µ?s? ap?d?s??

• ??a?µat??? p?????µµata (Real programs)
• ?.? µeta???tt?st??, epe?e??ast?? ?e?µ????,
  e??a?e?a CAD
• ?????e? (Kernels)
• ?.? Livermoore, Linpack
• ?p??? d???µ?? (Toy Benchmarks)
• ?.? ?? ??s???? t?? ??at?s????, Quicksort
• S???et???? d???µ?? (Synthetic Benchmarks)
• ?.? Whetstone, Dhrystone

9
Benchmark Suites

• ???a? s??????? ap? p?????µµata d???µ?? (real
  programs, kernels) p?? p??spa???? ?a µet??s???
  t?? ap?d?s? t?? ?p?????st??, ???s?µ?p????ta? µ?a
  p??????a efa?µ????
• ??t?? ?? s??????? e??a? µ??? t?s? ?a??? ?s? ??
  a?e???t?te? d???µ??
• Ost?s?, ????? t? p?e????t?µa, ?t? ? ad??aµ?a
  ?p??asd?p?te d???µ?? e?att??eta? e?a?t?a? t??
  pa???s?a? t?? ????? d???µ??
• ???pe? ?µ?? ?a ???s?µ?p??????? ?at?????e? µ???d??
  ??a ?a a????s??? t?? ap?d?s? µ?a? s???????

10
SPEC

• ?? e??a? ? SPEC (Standard Performance Evaluation
  Corp.)

???a? ??a? µ? ?e?d?s??p???? ???a??sµ?? p??
s?st????e ??a ?a d?µ??????se? a???p?ste? ?a?
a?t??e?µe????? d???µ?? ?p?????st?? ??a t??
µ?t??s? ?a? s?????s? t?? ap?d?s?? t?? ?p?????st?
se ep?ped? st???e??? ?a? s?st?µat?? (component-
and system-level computer performance), ??a
d??f??a ?e?t??????? s?st?µata ?a? pe??ß?????ta
(s?µpe???aµßa??µ???? ?????? ap?d?s?? a???µ?t?????
?p?????sµ???, Web servers, s?st?µata ??a ??af???)

• ?p?te?e?ta? ap? 3 d?af??et???? ?µ?de?

OSG Open Systems Group ?s???e?ta? µe d???µ?? se
ep?ped? st???e??? ?a? s?st?µat??, ??a sta?µ???
e??as?a? (workstations) ?a? e??p???te? p?????
???st?? (multi-user servers) ta ?p??a t??????
pe??ß?????ta a????t?? ?e?t???????? s?st?µ?t??.
HPG High-Performance Group ?s???e?ta? µe t??
a??pt??? d???µ?? ??a a???te?t?????? s?st?µ?t??
?????? ap?d?s?? symmetric multiprocessor
systems, workstation clusters, distributed memory
parallel systems, traditional vector and vector
parallel supercomputers.
GPC Graphics Performance Characterization
Group ?s???e?ta? µe t? d?µ??????a p??t?p??
d???µ?? ??a t? ß??µ??a??a t?? ??af????, ??a
efa?µ???? ??af???? ?a? p???µ?s??, ??p.
11
?e?t?????a SPEC

• ???? t?? SPEC
• S????te? s?st?µ?t??
• ????t?? ??????/????sµ???? (hardware/software
  vendors)
• ??e???t???? ???a??sµ??, ?a?ep?st?µ?a
• ??d?te?, S?µß?????
• ?d??te?-?e??te?

• ?a µ??? s?µf????? se e?d?se?? d???µ??, ?? ?p??e?
  pa?????ta? ap? ?p?????se? efa?µ????, ?ts? ?ste
  t?s? ?? s?ed?ast?? ?p?????st??, ?s? ?a? ??
  a???ast?? a?t?? ?a µp????? ?a ap?fas?s??? µe ß?s?
  ?ea??st????-p?a?µat???? e??as?e?.
• ?a µ???, (?at?p?? s?µf???a? µe ?de?a) s?µf?????
  ?a pa?????? ?a? ?a a?af????? ap?te??sµata, ?p??
  ????eta? se ???e ??d?s?.

Benchmark Suites
?s? ße?t???eta? ? te???????a, p??pe? ?a
ße?t?????ta? ?a? ?? a?t?st???e? e?d?se?? d???µ??.
SPEC 89 SPEC 92 SPEC 95 SPEC 2000
12
???t???a ep?????? p????aµµ?t?? d???µ??

• ?etafe?s?µ?t?ta se ??e? t?? a???te?t?????? t??
  SPEC (32- ?a? 64-bit s?µpe???aµßa??µ???? Alpha,
  Intel a???te?t????? , PA-RISC,Rxx00, SPARC, ??p)
• ?etafe?s?µ?t?ta se d??f??a ?e?t??????? s?st?µata,
  e?d??? UNIX ?a? NT
• ?? d???µ?? de? p??pe? ?a eµpe??????? µ?t??s? I/O
• ?? d???µ?? de? p??pe? ?a eµpe??????? st???e?a
  d??t??? ?a? ??af????
• ?? d???µ?? p??pe? ?a t?????? se RAM 256 ?????
  e?a??a?? (swapping)
• ?e? p??pe? ?a ???s?µ?p??e?ta? pe??ss?te?? ap? 5
  t?? ?????? d???µ?? st?? epe?e??as?a ??d??a p??
  de? ??e? d??e? ap? t?? SPEC
• ????t?ta-a???p?st?a, d?af??e?a

• ?a p?????µµata t?? s??????? SPEC2000, ep????t??a?
  ap? µ?a µe???? s?????? ap? ?p???f?e? efa?µ????,
  ?? ?p??e? ?p?ß?????a? st?? d???µas?a t?? µe???
  ?a? t?? t?? ?e????te??? ?????? µ?s? µ?a? ??e??a?
  p?? d?e??????e µ?s? t?? ?st?se??da? t?? SPEC

13
SPEC CPU2000 Benchmark

• ?e????e? d?? ?µ?de? p????aµµ?t?? d???µ??
• CINT 2000
• G?a µ?t??s? ?a? s?????s? ap?d?s?? ?p?????sµ?? se
  a?e?a????
• CFP2000
• G?a µ?t??s? ?a? s?????s? ap?d?s?? ?p?????sµ?? se
  a???µ??? ????t?? ?p?d?ast????

• C Component-level benchmarks

• ????? s?stat???? µet???e t?? ap?d?s?
• ??? epe?e??ast?
• ??? a???te?t?????? µ??µ??
• ??? µeta???tt?st?

• ????? s?stat???? de? µet???e t?? ap?d?s?
• ??? I/O
• ??? s?stat???? t?? d??t???
• ??? ??af????

14
CPU2000 SUITE
15
???a?? a?af????

• ? SPEC ???s?µ?p??e? µ?a µ??a?? a?af???? ??a ?a
  ?a????se? t?? µ???de? µ?t??s?? ap?d?s??. ???e
  d???µ? t???e? ?a? µet??ta? se a?t? t? µ??a??,
  ?ste ?a ?a??e???e? ??a? ?????? a?af???? ??a a?t?
  t? d???µ?. ??t?? ?? ?????? ???s?µ?p?????ta? µet?
  st??? ?p?????sµ??? t?? SPEC.

• ???a?? a?af???? SUN Ultra5_10 µe epe?e??ast? sta
  300MHz

• ?pa?te?ta? d??st?µa pe??p?? 2 ?µe??? ??a ?a ???e?
  ??a ??a t????µ? s?µf??a µe ta p??t?pa t?? SPEC
  (SPEC-conforming run) t?? CINT2000 ?a? CFP2000 se
  a?t? t? µ??a??.
• ?? t????µ? a?t? pe???aµß??e? t??????st?? 3
  epa?a???e?? t?? ?d?a? d???µ?? ??a ?a
  e?asfa??s??µe ?t? ta ap?te??sµata µp????? ?a
  a?apa?a?????.

16
?pe????s? ???? base, non- base rate, non-rate

• ? SPEC, p???e?µ???? ?a p??sf??e? s?????se?? ??a
  d?af??et???? a???te?t?????? ?p?????st??, pa???e?
  t?? p??a?? ??d??a t?? d???µ??. ??t? s?µa??e? ?t?
  p??pe? ?a µeta???tt?st???, p??? ?a t??????.
  ?p????? ?a µeta???tt?st??? µe t?? t??p? p?? ??
  ???ste? µeta???tt????? ta p?????µµata, d??ad?
• ???s?µ?p????ta? t?? ?e????? ?d???e? ??????
  ap?d?s??, p?? p??te??e? ? ?atas?e?ast?? t??
  µeta???tt?st?
• ??te, ?a pe??aµat?st??? µe d?af??et?????
  µeta???tt?st?? ?a? flags µeta???tt?st?? ??a ?a
  ep?t????? t?? ?a??te?? ap?d?s?
• ?pe?d? ? SPEC de? µp??e? ?a p??ß???e? p?? a???ß??
  ???s?µ?p????? t??? µeta???tt?st?? ?? d??f????
  ???ste?, p??sf??e? ??p??a ?a?a?t???st??? µe????
  p?? ?a a?t?p??s?pe???? ta ?a?a?t???st??? a?t??
  t?? 2 ?µ?d??

• Base (conservative) apa?t???ta? ap? ??e? t??
  ep?s?µe? a?af???? ?a? ????? ??se? ?d???e? ??a
  µeta???tt?s? (p.? ta ?d?a flags µe t?? ?d?a se???
  p??pe? ?a ???s?µ?p?????ta? ??a ??a ta p?????µµata
  t?? ?d?a? ???ssa?)
• Non base (aggressive) e??a? p??a??et???? ?a?
  µp????? ?a ???s?µ?p???s??? ????te?? a?st????
  apa?t?se?? (p.? d?af??et???? ep??????
  µeta???tt?st? µp????? ?a ???s?µ?p??????? ??a ???e
  p????aµµa d???µ??).

• ?p?????? d??f???? t??p?? ??a ?a µet???e? ?
  ap?d?s? e??? ?p?????st?

• ???µ?? d?eµeta????? (throughput, capacity, rate
  metrics) ?e???e? p?se? e??as?e? µp??e? ?a
  e?te??se? ??a? ?p?????st?? st? µ???da t?? ??????.
• ?a??t?ta e?t??es?? (non-rate metrics) ?et??e?
  t? ????? p?? e?te?e?ta? µ?a µ??ad??? e??as?a ap?
  t?? ?p?????st?.

17
?et??s?µa µe????
?? e?d?se?? CINT2000, CINT FP2000 µp????? ?a
???s?µ?p??????? ??a ?a µet??s??? ?a? ?a
?p?????s??? ta a??????a ?a?a?t???st??? µe????

• CINT2000
• SPECint2000
• SPECint_base2000
• SPECint_rate2000
• SPECint_rate_base2000

• CFP2000
• SPECfp2000
• SPECfp_base2000
• SPECfp_rate2000
• SPECfp_rate_base2000

• SPECint2000 ? ?e?µet????? µ?s?? t?? 12
  ?a??????p???µ???? ????? (µ?a ??a ???e d???µ?
  a?e?a???) ?ta? µeta???tt????ta? µe aggressive
  optimization ??a ???e d???µ?.
• SPECint_base2000? ?e?µet????? µ?s?? t?? 12
  ?a??????p???µ???? ????? ?ta? µeta???tt????ta? µe
  conservative optimization ??a ???e d???µ?.
• SPECint_rate2000? ?e?µet????? µ?s?? t?? 12
  ?a??????p???µ???? ????? t?? ???µ?? d?eµeta?????
  ?ta? µeta???tt????ta? µe aggressive
  optimization ??a ???e d???µ?.
• SPECint_rate_base2000? ?e?µet????? µ?s?? t?? 12
  ?a??????p???µ???? ????? t?? ???µ?? d?eµeta?????
  ?ta? µeta???tt????ta? µe conservative
  optimization ??a ???e d???µ?.
• ?µ???? ??a t?? d???µ?? a???µ?? ????t??
  ?p?d?ast????

18
?e??e??µe?a pa??t?? SPEC CPU2000

• ???a?e?a ??a µeta???tt?s?, t????µ?, ??e??? t??
  d???µ?? ??a d??f??a ?e?t??????? s?st?µata
• ???a??? ??d??a? ??a ta e??a?e?a, ?ts? ?ste ?a
  µp????? ?a d?µ?????????? ??a s?st?µata p?? de?
  ?a??pt??ta? ap? ta ?d? µeta???tt?sµ??a e??a?e?a
• ???a??? ??d??a? ??a ta p?????µµata d???µ??
• ???a?e?a ??a d?µ??????a a?af???? ap?d?s??
• ?a???e? ??a t????µ? ?a? d?µ??????a a?af???? p??
  ??????? p?? ?? d???µ?? p??pe? ?a ???s?µ?p???????
  ??a ?a pa?????? t?p?p???µ??a ap?te??sµata
• CPU2000 te?µ????s?

• ?pa?t?se?? s?st?µat??
• ?e?t??????? s?st?µa UNIX ? Windows NT
• ?eta???tt?st?? C, C, FORTRAN90
• ???????st?? 1 GB s?????? d?s??? ??a t??
  e??at?stas?, ?t?s?µ? ?a? t????µ? t?? SPEC CPU2000
• ???????st?? 256 RAM

19
??µata ??a ???s? t?? d???µ??

• ???at?stas? t?? pa??t?? CPU2000
• ?????µ? t?? ?at??????? se?a???? e??at?stas??
  (installation scripts) a?????a µe t? ?e?t???????
  s?st?µa
• ?eta???tt?s? t?? e??a?e???, a? ta e?te??s?µa de?
  pa?????ta?
• ?a????sµ?? t?? µe?????? (metric) p?? ?????µe ?a
  µet??s??µe
• ??µ??????a e??? configuration a??e??? ??a a?t? t?
  µ??e??? (se a?t? ??????ta? compiler flags ?a?
  ???e? p????f???e? p?? e?a?t??ta? ap? t? s?st?µa)
• ?????µ? t?? e??a?e??? t?? SPEC ??a µeta???tt?s?,
  t????µ? ?a? e?????? t?? d???µ??
• ?? ta pa?ap??? ß?µata e??a? ep?t???, µp??e? ?a
  d?µ???????e? µ?a a?af??? ßas?sµ??? st??? ???????
  t?e??µat?? ?a? s?????se?? t?? µe?e???

20
??µ? ?ata????? µet? t?? e??at?stas?
? d?µ? t?? d??t??? t?? ?ata????? t?? CPU ??e? ??
e???
SPEC or SPEC - the root directory
benchspec CFP2000 - floating
point benchmarks CINT2000 - integer
benchmarks bin - tools to run
and report on the suite config -
config files result - log files
and reports tools - sources for
the CPU2000 tools
??sa se ???e ???e µ?a a?e???t?t? d???µ?, ? d?µ?
e??a?
nnn.benchmark - root for this benchmark
Spec - SPEC metadata about the benchmark
data all - data used by all
runs (if needed by the benchmark) ref
- the real data set, required for all result
reporting test - data for a simple
test that an executable is functional
train - data for feedback-directed
optimization exe - compiled
versions of the benchmark run -
all builds and runs take place here src
- the sources for the benchmark
21
Runspec
t? ßas??? e??a?e?? ??a ?t?s?µ?, t????µ?, a?af???
ap?te?esµ?t??
runspec options list of benchmarks to run

• --action action (or -a action)
• validate Build (if needed), run, and generate
  reports
• --iterations number (or -n number)
• 3 3 iterations of each benchmark
• --reportable (or -s, --strict, --noloose)
  ! --loose (or -l, --nostrict, --noreportable)
• Loose enforce the CPU2000 run rules, so as to
  produce a result which is suitable for public
  reporting and/or submission to SPEC
• --tune tuning (or --tuning tuning, -T tuning)
• Base Build all the benchmarks in the suite with
  a common set of optimizations
• (Peak Build each benchmark in the suite with a
  set of optimizations individually tailored for
  that benchmark)
• --rate (-r)
• speed run (i.e. non-rate) Select rate run
  instead of speed
• --users number (or -u number)
• 1 Use number copies for a SPECrate run
• --output_format format (or -o format)
• all available formats all (all of the
  following), asc (ASCII text), config (config file
  used for this run),
• html (web page), pdf (Portable Document
  Format), ps (Postscript)
• --size size (or -i size, --input size)
• Ref Selects size of input data to run test,
  train, or ref

22
?a??de??µa ???s??

• F\cpu2000gt runspec --actionbuild --tunebase
  --configintel_nt_visual_studio.cfg gzip

?--tunebase ?a ???s?µ?p????e? ? t?p?p???µ???
µeta???tt?s?

• F\cpu2000gt runspec --sizetest --iterations1
  --noreportable
• More? --configintel_nt_visual_studio.cfg gzip

?--noreportable µa? ep?t??pe? ?a t?????µe t?
s???e???µ??? s?????? ?a? ??? ??? t?
s??????. ?--sizetest ???s?µ?p??e?ta? t?
µ????te?? µ??e??? ded?µ???? st?? e?s?d?

• F\cpu2000gt runspec --configintel_nt_visual_studi
  o --iterations1 --noreportable gzip

????s?µ?p??e?ta? t? p?a?µat??? µ??e??? ded?µ????
st?? e?s?d?
23
???e?? ap?te?esµ?t?? (CINT2000.003.asc)
SPEC CINT2000 Summary
-- --
Wed Dec 1 104202 1999  SPEC License 0
Test date -- Hardware availability
-- Tester --
Software availability -- 
Estimated
Estimated
Base Base Base Peak
Peak Peak Benchmarks Ref Time
Run Time Ratio Ref Time Run Time Ratio
---------------- ------------ -------------
------- ------------- ------------ -----------
164.gzip 1400 1843
75.9 175.vpr
X
176.gcc
X
181.mcf
X
186.crafty
X
197.parser
X
252.eon
X
253.perlbmk
X 254.gap

X 255.vortex
X
256.bzip2
X
300.twolf
X


24
S????e?a
164.gzip 1400 1843
75.9 175.vpr

X 176.gcc
X
181.mcf
X
186.crafty
X
197.parser
X
252.eon
X
253.perlbmk
X
254.gap
X
255.vortex
X
256.bzip2
X
300.twolf
X Est.
SPECint_base2000 --
Est. SPECint2000
-- 

• ?a pe??ss?te?a ap?te??sµata e??a? ?e?? epe?d?
  ?t?e?e µ??? µ?a ap? t?? 12 d???µ?? a?e?a???