A Case Study of HPC Metrics Collection and Analysis

1
A Case Study of HPC Metrics Collection and
Analysis
Philip Johnson and Michael Paulding, University
of Hawaii, Honolulu, Hawaii.

• Goals of the case study
• Provide a complete implementation of one Purpose
  Based Benchmark problem definition, called
  Optimal Truss Design
• Implement the Optimal Truss Design system in C
  using MPI on a 240 node Linux cluster at the
  University of Hawaii
• Develop and evaluate automated support for HPC
  process and product measurement using Hackystat
• Assess the utility of the metrics for
  understanding HPC development

Results Process and Product Telemetry Charts
Results Daily Diary with CLI and Most Active
File

• Metrics Collected
• Size (Number of files, Total SLOC, Parallel
  SLOC containing an MPI directive, Serial SLOC
  not containing an MPI directive, Test code)
• Active Time (amount of time spent editing
  Optimal Truss Design files)
• Performance (wall clock time on 1, 2, 4, 8, 16,
  and 32 processors)
• Milestone Tests (indicates functional
  completeness)
• Command Line Invocations

• Insights and Lessons Learned
• Productivity (22 LOC/hour) and test code density
  (27) seem in line with traditional software
  engineering metrics.
• Speedup data indicates almost linear speedup to 4
  processors, then falls off sharply, indicating
  that current solution is not scalable.
• Parallel and serial LOC were equal at start of
  project, then most effort was devoted to serial
  code, with some final enhancements to parallel
  code at end of project.
• Performance data was not comparable over course
  of project (only final numbers available no
  telemetry)
• Hackystat provides effective infrastructure for
  collection of process and product metrics.
• This case study provides useful baseline data to
  compare with future studies.
• Future research
• Compare to OpenMP or JavaParty implementation.
• Gather metrics while improving scalability of
  system
• Compare metrics against other application types.
  
• Analyze CLI data for patterns, bottlenecks

Thanks to our sponsors

• For More Information
• Understanding HPCS development through automated
  process and product measurement with Hackystat,
  Philip M. Johnson and Michael G. Paulding,
  Proceedings of the Second Workshop on
  Productivity and Performance in High-End
  Computing.