NGF07 Benchmarks

1
NGF07 Benchmarks David Turner NERSC User
Services Group Oakland Scientific Facility 14
May 2007
2
NGF07 Website

• http//www.nersc.gov/projects/procurements/NGF07/
• Complete Request For Proposal Package
• Windows zip file containing PDF files
• RFP
• Benchmark rules
• Legal-ise (sample subcontract, general
  provisions, etc.)
• Benchmarks
• Run Rules
• MS Word and PDF
• IOR and METABENCH
• gzipped tar files ior.tgz and metabench.tgz

3
NGF07 Website Continued

• http//www.nersc.gov/projects/procurements/NGF07/
• Hardware Inventory
• Current hardware that should be re-used
• HTML
• MS Word
• Plain text
• Questions and Answers
• Send questions to ngf07_at_nersc.gov
• General interest q/a will be posted
• Slides for May 14 Pre-Proposal Conference
• PowerPoint

4
Benchmarks in RFP

• MINIMUM REQUIREMENTS
• Performance
• The facility-wide global file system vendor must
• Commit to the delivery of sustained aggregate I/O
  performance as measured by the benchmarks
  described in the Benchmark Instructions.
• PROPOSAL PREPARATION INSTRUCTIONS
• In the technical proposal, the Offeror shall
  provide
• All requested benchmark documentation detailed in
  the Benchmark Instructions attached to this RFP
• ENCLOSURES
• Result requested in the Benchmark Instructions
• NERSC expects vendors to make best effort in
  providing requested benchmark results

5
Benchmarks in SoW

• Benchmark performance levels maintained through
  life of subcontract
• Additional benchmarks possible, subject to
  negotiation

6
Benchmark Rules

• Run on a small range of processors
• Fully packed
• Obtained from a file system configured as closely
  as possible to the proposed system
• If the proposal includes multiple file access
  protocols (e.g., pNFS or NFS), results should be
  provided for all such protocols
• Performance projections are allowed if they are
  derived from an earlier generation system
• Rigorously derived, easily understood and
  thoroughly documented
• Modifications only permissible for correct
  execution
• Must be fully documented and reported
• No changes related to optimization

7
Additional Rules

• Intent is to measure performance for that access
  disk
• Benchmarks utilize smaller working set sizes
• Optimizations may be possible that enable caching
  or buffering within system memory
• Such optimizations may be beneficial in a
  production environment
• Please do not engage in such optimizations for
  the benchmark runs!

8
Submission Guidelines

• Tables provided for results
• Results on paper with RFP response
• Results on CD with completed tables, benchmark
  source and output files
• Source shall be in a form that can be readily
  compiled
• Do not include object and executable files
• If performance projections are used
• Clearly indicate
• Include output files on which projections are
  based
• Describe of the projection method

9
Additional Guidelines

• Document benchmarking environment
• Client and server configurations, including node
  and processor counts, processor models, memory
  size and speed, and OS (names and versions)
• Storage used for global file system and storage
  configurations
• Network fabric used to connect servers, clients,
  and storage, including network configuration
  settings
• Client and server configuration settings,
  including client and server sysctl settings,
  driver options, network interface options, and
  file system configuration options
• Compiler name and version, compiler options, and
  libraries used to build benchmarks

10
The Benchmarks

• IOR
• Developed by Scalable I/O Project at LLNL
• Requires MPI and MPI-IO
• Download version IOR-2.9.3 from
• http//www.llnl.gov/icc/lc/siop/downloads/download
  .html
• The bandwidth numbers to be reported are the
  results listed as Max Write and Max Read
  measured in MB/s
• For more information on IOR , please read the
  User Guide included with the source distribution
• METABENCH
• MPI program that measures performance of various
  metadata operations.

11
IOR Definitions

• blocksize size (in bytes) of a contiguous chunk
  of data accessed by a single client it is
  comprised of one or more transfers. The
  blocksize is fixed at 1 GB for all multi-client
  tests.
• transferSize size (in bytes) of a single data
  buffer to be transferred in a single I/O call.
  Four transfer sizes are use for all multi-client
  tests 1 KB, 16 KB, 256 KB, and 4 MB.
• File access either a single file per MPI process
  is read/written (unique) or a single shared
  file for all processes is read/written
  (shared).

12
IOR Runs

• 65 runs are required
• POSIX API
• Shared file 4, 16, 32, and 64 processes,
  transfer size 1 KB, 16 KB, 256 KB, and 4 MB
• Unique file 4, 16, 32, and 64 processes,
  transfer size 1 KB, 16 KB, 256 KB, and 4 MB
• Unique file 1 process, blocksize 8 GB,
  transfer size 128 MB
• MPI-IO API
• Shared file 4, 16, 32, and 64 processes,
  transfer size 1 KB, 16 KB, 256 KB, and 4 MB
• Unique file 4, 16, 32, and 64 processes,
  transfer size 1 KB, 16 KB, 256 KB, and 4 MB

13
METABENCH Runs

• Three sets of circumstances
• Creating 30,400 files by one process in one
  directory
• Creating 30,400 files by NP processes in the
  same directory
• Creating 30,400 files by NP processes in
  separate directories, one directory per process
• NP will be either 2, 4, 16, or 32
• Other metadata operations (stat and removal) are
  also timed

14

• Questions?