Scaling to New Heights

1
Scaling to New Heights

• Retrospective
• IEEE/ACM SC2002 Conference
• Baltimore, MD
• Trimmed Distilled for SOS7 by M. Levine
• 4-March-2003, Durango

2
Contacts and References

• David ONeal oneal_at_ncsa.uiuc.edu
• John Urbanic urbanic_at_psc.edu
• Sergiu Sanielevici sergiu_at_psc.edu
• Workshop materials
• www.psc.edu/training/scaling/workshop.html

3
Introduction

• More than 80 researchers from universities,
  research centers, and corporations around the
  country attended the first "Scaling to New
  Heights" workshop, May 20 and 21, 2002, at the
  PSC, Pittsburgh.
• Sponsored by the NSF leading-edge centers (NCSA,
  PSC, SDSC) together with the Center for
  Computational Sciences (ORNL) and NERSC, the
  workshop included a poster session, invited and
  contributed talks, and a panel.
• Participants examined issues involved in adapting
  and developing research software to effectively
  exploit systems comprised of thousands of
  processors. Fred/Neils Q1.
• The following slides represent a collection of
  ideas from the workshop

4
Basic Concepts

• All application components must scale
• Control granularity Virtualize
• Incorporate latency tolerance
• Reduce dependency on synchronization
• Maintain per-process load Facilitate balance
• Only new aspect, at larger scale, is the degree
  to which these things matter

5
Poor Scalability?(Keep your eye on the ball)
6
Good Scalability? (Keep your eye on the ball)
7
Performance is the Goal! (Keep your eye on the
ball)
8
Issues and Remedies

• Granularity Q2a
• Latencies Q2b
• Synchronization
• Load Balancing Q2c
• Heterogeneous Considerations

9
Granularity

• Define problem in terms of a large number of
  small objects independent of the process count
  Q2a
• Object design considerations
• Caching and other local effects
• Communication-to-computation ratio
• Control granularity through virtualization
• Maintain per-process load level
• Manage comms within virtual blocks, e.g. Converse
• Facilitate dynamic load balancing

10
Latencies

• Network
• Latency reduction lags improvement in flop rates
  Much easier to grow bandwidth
• Overlap communications and computations Pipeline
  larger messages
• Dont wait Speculate! Q2b
• Software Overheads
• Can be more significant than network delays
• NUMA architectures
• Scalable designs must accommodate latencies

11
Synchronization

• Cost increases with the process count
• Synchronization doesnt scale well
• Latencies come into play here too
• Distributed resource exacerbates problems
• Heterogeneity another significant obstacle
• Regular communication patterns are often
  characterized by many synchronizations
• Best suited to homogeneous co-located clusters
• Transition to asynchronous models?

12
Load Balancing

• Static load balancing
• Reduces to granularity problem
• Differences between processors and network
  segments are determined a priori
• Dynamic process management requiring distributed
  monitoring capabilities Q2c
• Must be scalable
• System maps objects to processes

13
Heterogeneous Considerations

• Similar but different processors or network
  components configured within a single cluster
• Different clock rates, NICs, etc.
• Distinct processors, networking segments, and
  operating systems operating at a distance
• Grid resources
• Elevates significance of dynamic load balancing
  Data-driven objects immediately adaptable

14
Tools Q2d?

• Automated algorithm selection and performance
  tuning by empirical means, e.g. ATLAS
• Generate space of algorithms and search for
  fastest implementations by running them
• Scalability prediction, e.g. PMaC Lab
• Develop performance models (machine profiles
  application signatures) and trending patterns
• Identify/fix bottlenecks choose new methods?

15
Topics for Discussion

• How should large, scalable computational science
  problems be posed?
• Should existing algorithms and codes be modified
  or should new ones be developed?
• Should agencies explicitly fund collaborations to
  develop industrial-strength, efficient, scalable
  codes?
• What should cyber-infrastructure builders and
  operators do to help scientists develop and run
  good applications?

16
Summary Comments (MJL)

• Substantial progress, with scientific payoff, is
  being made.
• It is hard work without magic bullets.
• gtgtgt Dynamic load balancing ltltlt
• Big payoff, homogeneous and heterogeneous
• Requires considerable people work to implement
• Runtime overhead very small.

17
Case StudyNAMD Scalable Molecular Dynamics

• Three-dimensional object-oriented code
• Message-driven execution capability
• Fixed problem sizes determined by biomolecular
  structures
• Embedded PME electrostatics processor
• Asynchronous communications

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Case StudySummary

• As more processes are used to solve the given
  fixed-size problems, benchmark times decrease to
  a few milliseconds
• PME communication times and operating system
  loads are significant in this range
• Scaling to many thousands of processes is almost
  certainly achievable now given a large enough
  problem
• 700 atoms/process x 3,000 processes 2.1M atoms