GRID Benchmarking

1
GRID Benchmarking

• George Tsouloupas, Marios Dikaiakos, Pedro
  Trancoso
• georget,mdd,pedro_at_cs.ucy.ac.cy
• Department of Computer Science
• University of Cyprus

2
Why GRID Benchmarking?

• Users
• Define Quality-of-Service to be expected
• Developers
• Develop and tune applications
• Compare job submission services, resource
  allocation policies, scheduling algorithms
• Architects
• Improve system design
• Compare implementations/systems
• GRID Architecture
• Traditional Benchmarks are not sufficient

3
Outline

• Benchmarks 101
• Difficulties with the GRID
• Benchmarking the GRID
• Current Efforts
• GridBench
• Conclusions

4
Benchmarks 101

• Kernels
• MAPS, STREAM, PALLAS, LINPACK
• Scientific
• SPEC (e.g. SPEC CPU)
• Parallel
• NAS, SPLASH, Perfect
• Commercial Applications
• TPC (e.g. TPC-C and TPC-H)
• Multimedia
• MediaBench

But the GRID Architecture is more complex
than traditional systems!
5
Difficulties with GRID Benchmarking

• Architecture
• Heterogeneous system
• Non-exclusive use
• Dynamic environment resources added and deleted
  online
• Middleware extra overheads
• Metrics
• Quantitative and Qualitative
• Understanding results

6
Difficulties with GRID Benchmarking
7
Benchmarking the GRID

• Benchmark Hierarchy!
• Micro-benchmarks (or probes)
• Measure single performance aspect at different
  levels
• Micro-kernel benchmarks
• Performance stress-test with realistic workloads
• Application benchmarks
• Derived from real-world applications

8
Current Efforts

• Probes
• Benchmark Probes for Grid Assessment Chun et
  al. 2003
• Grid Benchmarking Research Group (Global Grid
  Forum) NAS Grid Benchmarks (NGB)
• Specification Version 1.0 Wijngaart and Frumkin
• Benchmarks for Grid Computing Snavely et al.
  2003
• GridBench (CrossGRID WP2)
• Prototype Documentation for GridBench version
  1.0, 2003.
• Design Document for GridBench, version 1.0, 2002.
  
• Software Requirements Specification, version 1.1
  for GridBench , 2002.
• GridBench A Tool for Benchmarking the Grid
  Tsouloupas and Dikaiakos 2003
• Performance Evaluation
• Performance evaluation on Grid Z. Németh

9
Probes

• Data intensive
• Exercise basic Grid operations
• 3-Node Probe (DB ? Comp ? Res)
• Circle Probe (A ? B ? ? A)
• Gather Probe (DB0, DBn ? Comp ? Res)
• Evaluated on the GrADS testbed (UCSD, UTK, UIUC,
  and UH) Site level
• Diagnostic tool
• Unpredictability, expired keys

10
NAS Grid Benchmarks

• Based on NAS Parallel Benchmarks
• Four families of problems
• Embarrassingly Distributed (ED)
• Helical Chain (HC)
• Visualization Pipeline (VP)
• Mixed Bag (MB)
• Different problem sizes

11
GridBench

• Framework for Grid benchmarks
• Conduct experiments
• Measure performance
• Interpret results
• Performance at different levels
• Resources (e.g. cluster node, storage element)
• Site (e.g. cluster, SMP)
• Grid Constellation (e.g. Virtual Organization)
• Middleware

12
Grid Model Architecture
CrossGrid Testbed
13
GridBench Benchmark Categories

• Micro-benchmark
• Micro-kernel benchmark
• Application benchmarks

14
GridBench Components

• RSL/JDL Compiler
• XML benchmark description (GBDL) to job
  description language
• Orchestrator
• Manager of execution and result collection
• Benchmark Components
• Benchmark executable code
• Monitoring Component
• Collects information using e.g. R-GMA or OCM-G

15
GridBench Components (2)

• Archive Database
• Benchmark definition, monitor, and results
• Information Provider
• Publishes results
• Benchmark Definition UI
• GUI for defining and executing benchmarks
• Benchmark Browser
• GUI for browsing and analyzing results

16
GridBench Software Architecture
17
GridBench Definition Lang (GBDL)

• platform-independent XML benchmark specification
• Benchmark Components
• Parameters
• Location
• Co-requisites
• Prerequisites
• Constraints
• Metrics
• Metrics
• Monitors
• Archive

18
Metrics Representation

• Benchmark-level metrics Vs component-level
  metrics (e.g. benchmark completion time Vs
  component completion time)
• Single-value metrics
• ltmetric name"epwhetstone_IPS type"value"gt
• ltvector unit"MIPS"gt53lt/vectorgt
• lt/metricgt
• Multi-value metrics
• ltmetric name"xfer-rate" type"list"gt
• ltvector unit"bps"gt
• 2039430,2083930,1909830,2184750,...lt/vectorgt
  
• ltvector unit"second" toffset"1055327287"gt
• 0,10,20,30,...lt/vectorgt
• lt/metricgt

19
Monitoring

• GBDL can contain monitoring definitions.
• ltcomponent name"data-transfer"
  ID"xfer01"gt...lt/componentgt
• ltmonitor type"RGMA" source"ccwp71.in2p3.fr3306
  "
• query"select from NetworkTCPThroughput
• where NMIdSource'adc0003.cern.ch'
• and NMIdDestination'ccwp7.in2p3.fr'
• ltparameter name"begin"gtcomp-begin"xfer01"lt/p
  arametergt
• ltparameter name"end"gtcomp-end"xfer01"lt/param
  etergt
• lt/monitorgt
• Implies that one or more monitoring client/s must
  be available

20
GDBL Compilation

• XML-based GBDL to Job Description
• Support for simple jobs can be through the use of
  simple templates. (executable, parameters and
  locations are transformed to simple RSL/JDL
  scripts)
• Most benchmarks need special command-line
  parameter formatting, or parameter files.

GBDL
Paramhandler
compiler
RSL
JDL

21
Compilation Example GBDL to RSL
GBDL ltbenchmark date name"nbody"gt ltcomponent
id"A" name"nbody type"mpi"gt
ltlocation type"single"gt ltresource
cpucount"2 name"ce1.grid.ucy"/gt
ltresource cpucount"2
name"ce2.grid.ucy"/gt lt/locationgt
ltparameter name"executable"
type"attribute"gt /bin/nbody.exec
lt/parametergt ltparameter name"nparticles"
type"value"gt
1000 lt/parametergt lt/componentgt
lt/benchmarkgt

• RSL
• ((resourceManagerContact
• ce1.grid.ucy")
• (label"subjob 0")
• (environment
• (GLOBUS_DUROC_SUBJOB_INDEX 0))
• (count2)
• (arguments"-n 1000")
• (executable"/bin/nbody.exec" ))
• ((resourceManagerContact
• "ce2.grid.ucy ")
• (label"subjob 1")
• (environment
• (GLOBUS_DUROC_SUBJOB_INDEX 1))
• (count2)
• (arguments"-n 1000")
• (executable"/bin/nbody.exec" ))

22
Archival and Publication of Results

• Benchmark results are archived in a native XML
  database (Apache Xindice)
• The Benchmark results are stored together with
• GBDL definition.
• Results are meaningless without the specific
  parameters
• Monitoring data
• Comprehension/Analysis of results is enhanced
  when combined with monitoring data.
• Benchmark results related to a Site are published
  to the local MDS
• easy access by users and schedulers

23
GridBench Micro-benchmarks

• _at_ Worker node level
• EPWhetstone embarrassingly parallel adaptation
  of the serial whetstone benchmark (MIPS)
• BlasBench evaluate serial performance of the
  BLAS routines (FLOPS and MB/s)
• _at_ Site level
• Bonnie Storage I/O performance (read and write
  MB/s)
• MPPtest MPI performance measurements (MPI
  Performance)
• _at_ VO level
• MPPtest MPI performance measurements (spanning
  sites)
• gb_ftp File Transfer Benchmark (transfer rate)

24
GridBench Micro-kernels

• _at_ Site level
• High Performance Linpack
• kernels from NAS Parallel benchmarks (NPB)
• _at_ VO level
• computational intensive Grid Benchmarks (e.g.
  NGB)

25
Experiment
User Interface
CE
GBDL
CE
Archiver SE
26
GridBench UI

• Definition

27
GridBench UI

• GDBL

28
GridBench UI

• JDL

29
GridBench Browser
30
Experimental Results

• Micro-benchmark _at_ Worker node level
• EPWhetstone
• Micro-benchmark _at_ VO level
• gb_ftp

31
Experimental Results

• Micro-kernel benchmark _at_ Site level
• High Performance Linpack

32
Conclusions

• Benchmarking the GRID is different from
  traditional benchmarking
• GridBench Complete framework
• Define Benchmark
• Platform-independent GDBL
• Execute benchmarks
• Hierarchical benchmarks
• Archive and Publish results
• Graphical User Interface

33
Future Work

• Work-In-Progress
• Complete GBDL specification
• Implementation of more benchmarks
• Monitoring
• Interpret results