Benchmarking PCs and Servers

1
Benchmarking PCs and Servers

• Dr G E South

6 July 2012
Benchmarking
1
2
1. Introduction

• There are three kinds of lies lies, damned
  lies and benchmarks
• This is a quote from a web site which was set up
  to analyse the performance of the Linux operating
  system (www.tux.org) but the statement is often
  used by developers
• It is an adaptation of Benjamin Disraelis there
  are three kinds of lies lies, damned lies and
  statistics

6 July 2012
Benchmarking
2
3
2. www.tux.org direct quotes from website

• This site was focussed on three basic uses of
  benchmarking
• Comparing similar versions of Linux on different
  hardware useful in measuring or predicting the
  performance of hardware for Linux and Linux
  applications, and helpful in selecting hardware
• Comparing different versions of Linux on the same
  hardware helps to spot areas in the Linux kernel
  which need tuning for performance and to measure
  the progress of Linux towards higher performance
• Comparing Linux to other operating systems on the
  same hardware helps to find areas where Linux
  could use improvement, and to identify strengths
  and weaknesses in different operating systems.
  Should be done with a great degree of detachment
  and perspective, as inter-OS benchmarking can
  also be quite a source of controversy to say the
  least!

6 July 2012
Benchmarking
3
4
3. www.tux.org
Modify handout

• The website still exists but has a lot of old
  content and very limited benchmark data
• Suggest the problems of implementing the
  ambitious plans from the previous slides
• very ambitious for an unfunded organisation
• problems with hardware manufacturers if
  benchmarks are not reproducible lots of testing
  needed

6 July 2012
Benchmarking
4
5
4. Website www.tuxradar.com

• This site has features on Linux and some
  comparison of Linux distributions and Windows
• Who would be interested in such a site ?
• How does it compare to www.tux.org ?
• Linux system administrators/ developers
• Conclusions
• A site for Linux enthusiasts

6 July 2012
Benchmarking
5
6
5. Simple PC benchmarks

• The most common use of PC benchmarks is in
  computer magazines. Systems are reviewed and it
  is necessary to provide some form of measurement
• Since it is unlikely the PC-A is going to be
  outstandingly different to PC-B, then in order to
  arrive at the best system there must be a series
  of tests to provide measurements of each PCs
  performance
• Examples PC Pro Magazine has a private
  benchmark suite involves complex tasks -Office/
  2D graphics/ Encoding/ Multi-tasking

6 July 2012
Benchmarking
6
7
6. Early computer benchmarks

• At one time it was easy to arrive at the
  comparative performance of computer systems
• The processor was the most important part of the
  system since computers were mainly used for
  number crunching
• So a useful benchmark was to run a series of
  numerically demanding tasks the computer which
  completed the task in the shortest time was the
  fastest e.g. Whetstone benchmark

6 July 2012
Benchmarking
7
8
7. Modern benchmarking

• Q Why is a processor performance test an
  unsatisfactory method of benchmarking modern
  computer systems ?
• A Computers are now used for a wide range of
  tasks - the overall performance depends on the
  features of many different elements of the system
• Bottlenecks often are more of a problem than the
  performance of the processor

6 July 2012
Benchmarking
8
9
8. Modern benchmarking

• Areas which are likely to impact on the
  performance of a modern PC
• Integer operations e.g. Office
• Floating point operations e.g. CAD
• Multimedia instructions SSE etc e.g. MM encoding
• Speed and size of caches .. Does benchmark code
• Caching algorithms / fit in cache
• Speed, size and technology of main memory
• Turbo memory ! Not used in modern chipsets

6 July 2012
Benchmarking
9
10
9. Modern benchmarking - continued

• Areas which are likely to impact on the
  performance of a PC (continued)
• Pipelines in processor
• Out of order execution, register aliasing
• Disk performance, disk buffering, use of SSDs
• Disk technology RAID etc
• Video subsystem (processor/ VRAM/ access to main
  RAM for rendering
• Network technology
• Bus interface technology PCI/ PCIe/ USB 2 or 3

11
10. Sandra

• The Sandra utility allows many of the features
  listed to be benchmarked
• However these are synthetic benchmarks
• State important differences between synthetic and
  application-based benchmarks
• A Synthetic b/m repeats a set of instructions
  and times how long it takes (may repeat a billion
  times)
• Application benchmarks relate to a real task
  e.g. reformatting a large Word Document
• State a use for Sandra benchmarks
• comparing two processors, two disk drives etc

12
11. Application-based benchmarks for PCs

• List people who can make use of application-based
  benchmarks for PCs
• staff involved in purchasing PCs/ Servers etc
• home users who want the best performance/price
  ratio
• What are the important features of such
  benchmarks ?
• usually there is a score for each phase e.g.
  Office, these can be weighted to give a final
  score
• E.g. Office (80) 4 2D graphics (20) 5
• Final weighted score .8 times 4 plus .2 times 5
  4.2

13
12. PCMark05 a commercial system test
suite

• Futuremark is a company which develops games and
  also produces benchmarking/ testing products for
  PCs
• e.g. 3DMark and PCMark benchmarks
• PCMark05 supports both system level and component
  level benchmarking. It is comprised of different
  test suites and the user also has the ability to
  design custom test suites quote from Futuremark
  website

14
13. PCMark05 a commercial system test
suite

• There are 11 system tests - each one is designed
  to represent a certain type of PC usage. By
  running these tests, PCMark05 stresses the
  components in a similar manner as they are
  stressed in normal home usage
• The system test suite contains tests measuring
  different components of a PC and the results of
  the tests as well as the total score depend on
  all components
• The system tests stress CPU, system bus, Memory
  and Graphics performance

15
14. SPEC Standard Performance Evaluation
Corporation

• See http//www.spec.org/
• The Standard Performance Evaluation Corporation
  (SPEC) is a non-profit corporation formed to
  establish, maintain and endorse a standardized
  set of relevant benchmarks that can be applied to
  the newest generation of high-performance
  computers
• SPEC develops benchmark suites and also reviews
  and publishes submitted results from our member
  organizations and other benchmark licensees
• quote from SPEC website

16
15. A recent SPEC proposal

• 09/04/2009 - The SPECpower committee declares its
  intent to design, develop and deliver a next
  generation tool set that will measure and
  evaluate the performance and power of computer
  servers over a wider spectrum of functionality
• The tool set will have the potential to be
  adopted by the EPA for use in the US EPA Server
  Energy Star program and to form a solid basis for
  the next generation SPECpower benchmark.

6 July 2012
Benchmarking
16
17
16. SPEC CPU2006

• Designed to provide performance measurements
  that can be used to compare compute-intensive
  workloads on different computer systems
• SPEC CPU2006 contains two benchmark suites
  CINT2006 for measuring and comparing
  compute-intensive integer performance
• CFP2006 for measuring and comparing
  compute-intensive floating point performance
• The results on the SPEC website can be used to
  investigate the servers described in the Server
  topic

6 July 2012
Benchmarking
17
18
17. SPEC CPU2006 - System x 3200 M2

• Test Sponsor IBM Corporation
• System Name System x 3200 M2 (Intel Xeon X3360)
• Enabled Cores 4
• Enabled Chips 1
• Cores/Chip 4
• Threads/Core 1
• Results Base 20.8
• Results Peak 24.9 
• OS SuSE Linux Enterprise Server 10 (x86_64) SP1
  
• hw_disk 1 x 146 GB SAS, 15000 RPM  

6 July 2012
Benchmarking
18
19
18. SPEC CPU2006 - ProLiant DL120 G5

• Test Sponsor Hewlett-Packard Company
• System Name ProLiant DL120 G5 (2.66 GHz, Intel
  Xeon X3350)
• Enabled Cores 4
• Enabled Chips 1
• Cores/Chip 4
• Threads/Core 1
• Results Base 20.6 
• Results Peak 22.4  
• OS SuSE Linux Enterprise Server 10 (x86_64) SP1
• hw_disk 1 x 146 GB SAS, 15000 RPM  

6 July 2012
Benchmarking
19
20
19. Whetstone and Dhrystone

• Dhrystone is a synthetic computing benchmark
  program developed in 1984 by Reinhold P. Weicker
  intended to be representative of system (integer)
  programming. The Dhrystone grew to become
  representative of general processor (CPU)
  performance until it was superseded by the CPU89
  benchmark suite from the Standard Performance
  Evaluation Corporation, today known as the
  "SPECint" suite. The name is a pun on a different
  benchmark algorithm called Whetstone.

6 July 2012
Benchmarking
20
21
20. Whetstone

• The Whetstone benchmark primarily measures the
  floating-point arithmetic performance. A similar
  benchmark for integer and string operations is
  the Dhrystone
• A listing is available on the internet

6 July 2012
Benchmarking
21
22
21. Intel - Defining the Client Benchmarking
Suite

• Intel use a combination of off-the-shelf
  performance testing software and internally
  developed scripts
• Preparation (manual operation)
• Connect client to network
• Ensure system meets minimum security requirements
  
• Install a standard OS and application payload
  package
• On a second pass, add new software, remove
  software, or change some system configuration to
  test performance impact against benchmark
• Open and synchronize e-mail application
• Disconnect client from network
• http//download.intel.com/it/pdf/Optimizing_Platfo
  rm_Performance.pdf

6 July 2012
Benchmarking
22
23
22. Intel - Defining the Client Benchmarking Suite

• Technical characterization (uses
  industry-standard benchmarking software to record
  the configuration information of the PC being
  tested)
• CPU/ Hard disk/ Memory/ Graphics processing unit
  (GPU)
• Productivity tests (uses automatic scripts, with
  an average of six runs)
• Word processor crunch test/ Spreadsheet crunch
  test/ Presentation software load test/ Internet
  browser load test
• Timing tests (uses a combination of automatic and
  manual scripts)
• PC startup/ Hibernate/ Standby

6 July 2012
Benchmarking
23
24
23. More info on the Futuremark System Test
Suite from website

• HDD - XP startup
• Physics and 3D
• 2D - Transparent windows
• 3D - Pixel Shader
• Web page rendering
• File Decryption
• 2D - Graphics memory - 64 lines
• HDD - General usage
• Multithreaded test 1 Video encoding Audio
  Compression
• Multithreaded test 2 Text Edit Image
  Decompression
• Multithreaded test 3 File Encryption HDD -
  Virus scan Memory - Latency 16MB File
  compression

6 July 2012
Benchmarking
24
25
24. More info on the Futuremark System Test
Suite

• File compression /File decompression
• File Encryption /File Decryption
• Image decompression
• Audio encoding
• Multithreaded test 1 File compression File
  EncryptionThis test runs the File Compression
  and File Encrypting tests at the same time
  multithreaded.
• Multithreaded test 2 File decompression File
  decryption Audio decompression Image
  decompressionThis test runs the File
  compression, File decryption, Audio decoding and
  Image decoding tests at the same time
  multithreaded.
• Memory Test Suite
• Memory Read tests 16MB, 8MB, 192kB, 4kB block
  sizesMemory Write tests 16MB, 8MB, 192kB, 4kB
  block sizesMemory Copy tests 16MB, 8MB, 192kB,
  4kB block sizesMemory Latency tests 16MB, 8MB,
  192kB, 4kB block sizes
• Graphics Test Suite
• 2D - Transparent Windows /2D - Graphics memory 64
  lines /2D - Graphics memory 128 lines
• 2D - Video playback /3D - Fill rate /3D - Polygon
  throughput /3D - Pixel shader /3D - Vertex shader
  /HDD Test Suite
• XP Startup /Application loading /General usage
  /Virus scanning /File write

6 July 2012
Benchmarking
25
26
9. Modern benchmarking

• "There are three kinds of lies lies, damned
  lies, and benchmarks" -
• This site focuses on three basic uses of
  benchmarking
• Comparing similar versions of Linux on different
  hardware useful in measuring or predicting the
  performance of hardware for Linux and Linux
  applications, and helpful in selecting hardware.
• Comparing different versions of Linux on the same
  hardware helps to spot areas in the Linux kernel
  which need tuning for performance and to measure
  the progress of Linux towards higher performance.
  
• Comparing Linux to other operating systems on the
  same hardware helps to find areas where Linux
  could use improvement, and to identify strengths
  and weaknesses in different operating systems.
  Should be done with a great degree of detachment
  and perspective, as inter-OS benchmarking can
  also be quite a source of controversy to say the
  least!
• What you wil find at the Linux Benchmarking
  Project (LBP)
• Benchmark sources. We have put together a
  collection of GNU/GPL licensed benchmarks for
  download.
• A repository of benchmark results. You can
  select, examine and download benchmark results
  for a variety of Linux systems from our on-line
  SQL database.
• Benchmarking information. Links, documentation
  and a mailing list.
• The LBP is an open project, any Linux user can
  contribute results, documentation or benchmark
  code. Our goal is to provide serious, useful
  benchmarking information and standards for Linux.

6 July 2012
Benchmarking
26
27

• General benchmarking information
• The definitions for benchmark, Whetstone and MIPS
  from the On-Line Computing Dictionary.
• The Yahoo page on computer benchmarks.
• The Usenet group comp.benchmarks is widely read.
• The latest version of the comp.benchmarks FAQ can
  be fetched directly from the author's site. It
  lists 46 different benchmarks.
• BenchWeb is a complete Web site dedicated to
  computer benchmarks, with an impressive list of
  links.
• If you are mainly concerned with CPU performance,
  take a look at the CPU Info Center. And if your
  favorite CPU belongs to the x86 family, you can
  check the sandpile.org site.
• PDS the Performance Database Server is an
  on-line database with benchmark results for a
  variety of benchmarks and configurations.
• Linux benchmarking information
• For basic Linux benchmarking, you will find the
  Linux Benchmarking HOWTO at the LDP site and
  mirrors.
• Samuel S. Chessman has a page with graphics
  comparing the performance of Linux SMP kernels.
• Uwe F. Mayer has set up a Web site for a version
  of the BYTEmark benchmark suite that he modified
  for Linux - nbench.
• Specific benchmarks
• The Doom benchmark site. Will tell you something
  about the game performance of Linux boxes.
• MDBNCH a molecular dynamics benchmark, written
  in FORTRAN. It is a good example of an
  application benchmark.
• LMBENCH is a low-level benchmark suite for UNIX
  systems.
• A Web site with results for the x-bench test
  running on XFree86 servers (Note x-bench is
  considered outdated and inadequate as a measure
  of modern X server performance).
• WebStone 2.01 is a benchmark suite for Web server
  performance analysis.
• High performance computing

6 July 2012
Benchmarking
27
28

• This server has been kindly donated by Erol's
  Internet Services.
• The CPU is a Pentium (R) 166 MHz box with 64 MB
  of 60 ns EDO DRAM. Main storage is provided by 2
  EIDE hard disks connected to the onboard EIDE
  controller and 3 SCSI drives connected to an
  Adaptec PCI SCSI interface board. The PCI
  Ethernet network adapter is based on Digital's
  32-bit bus-mastering DC21140 chip.
• The base Linux distribution is RedHat, version
  4.2. It has been extensively customized, though
  a specific 2.0.29 kernel was compiled, some
  packages were replaced by newer ones and the
  directory structure was somewhat modified.
• Our server runs the latest version of the Apache
  Web server daemon.
• The database that holds benchmark results was
  built on top of the mySQL engine.
• We have used the PHP/FI package installed as an
  Apache module to interface the SQL database to
  the LBP Web pages.
• Most images you will find on this site were
  produced using the GIMP.
• Any questions ? Contact the webmaster_at_wauug.erols.
  com.

6 July 2012
Benchmarking
28
29

• Dhrystone is a synthetic computing benchmark
  program developed in 1984 by Reinhold P. Weicker
  intended to be representative of system (integer)
  programming. The Dhrystone grew to become
  representative of general processor (CPU)
  performance until it was superseded by the CPU89
  benchmark suite from the Standard Performance
  Evaluation Corporation, today known as the
  "SPECint" suite. The name is a pun on a different
  benchmark algorithm called Whetstone.

6 July 2012
Benchmarking
29
30
Custom PC

• Custom PC mainly rate PC components such as
  motherboards
• They use a standard test rig 250 GB hard disk,
  4 GB XMS2-8500 RAM,

6 July 2012
Benchmarking
30
31
Result from PCMARK05

• 32397PCMarks
• TEAM.AU Gigabyte X58-UD4P / ...
• Aug 30, 2009 0504 CDT
• CPU Intel(R) Core(TM) i7 CPU 960 _at_ 3.20GHz
• GPU ATI Radeon HD 4890
• OS Microsoft Windows Vista
• Benchmark Settings Not default

6 July 2012
Benchmarking
31