Simulation Laboratories:

1

• Simulation Laboratories
• An InnovativeCommunity-Oriented Research
  andSupport Structure

2
Outline

• Part 1 Current and near future HPC activities in
  Europe
• DEISA, EGEE
• Activities of European bodies
• Structuring SC in Germany
• PRACE
• Part 2 Simulation Laboratories
• Current user support structure
• Concept of Simulation Laboratories
• Example
• Summary

3
DEISA

• Consortium of leading national HPC centers in EU
• Deploy and operate a persistent, production
  quality, distributed, heterogeneous HPC
  environment

IDRIS CNRS, France FZJ, Jülich, Germany RZG,
Garching, Germany CINECA, Bologna, Italy EPCC,
Edinburgh, UK CSC, Helsinki, Finland SARA,
Amsterdam, NL HLRS, Stuttgart, Germany BSC,
Barcelona, Spain LRZ, Munich, Germany ECMWF,
Reading, UK
4
Services

• Dedicated 10 Gb/s network as a basis
• High-performance Data Grid via GPFS
• Extended to non-AIX Linux like SGI Altix, Mare
  Nostrum
• Job migration across sites
• Used to load balance the global workflow when a
  huge partition is allocated to a DEISA project in
  one site
• UNICORE as Grid Middleware for access
• Global data management to include tertiary
  storage and hierarchical data management system
• Science Gateways and Portals to facilitate the
  access of new, non-traditional users communities

5
Application Support

• DEISA Extreme Computing Initiative (DECI)
• Participation of NIC users 3 in 2005/6, 6 in
  2007, 8 in 2008For comparison in total 29 in
  2005/6, 41 in 2007, 63 in 2008
• Exceptional CPU-time grants
• Provision of sustained application enabling
  support
• Workflows and coupled applications
• Focus on parallel I/O and portable data formats
• Benchmark activities
• Codes of NIC users have become part of DEISA
  Benchmark Suite

6
EGEE

• Flagship Grid infrastructure project co-funded by
  the European Commission
• More than 90 partners in more than 30 countries
• Main objectives
• Operate a large-scale, production- quality Grid
  infrastructure for e-Science
• Attract new resources and users from industry as
  wellas sciences

7
Towards European HPC (I)

• EU Framework Programme 7
• Promoting a European High-Performance Computing
  Service
• Funding of international cooperations and
  networks, however, supercomputer hardware only to
  a small extent
• e-IRG e-Infrastructure Reflection Group
• Grid is unlikely to solve all scientific
  problems, and there are domains, which require
  very large centralized computing resources
• The establishment of supercomputers of the
  highest performance class in Europe cannot be
  funded by a single country

8
Towards European HPC (II)

• ESFRI European Strategy Forum on Research
  Infrastructures
• Government representatives of the member states
• Published a Research Infrastructures Roadmap
  (Sep 2006)which includes the installation of a
• European High-Performance Computing Service
• Starting 2009/10
• Total costs 200-400 M initially, 100-200 M
  every 2-3 years
• Preparatory phase 2007/8 10 M
• HET (HPC in Europe Taskforce)
• Members of HET built a consortium to realize such
  an infrastructure ? PRACE

9
Forschungszentrum Jülich (FZJ)
10
Supercomputing at FZJ (I)

• Supercomputing at Forschungszentrum Jülich is
  being supported by theJülich Supercomputing
  Centre (JSC, former ZAM) and
• the virtual institute John von Neumann
  Institute for Computing
• JSC is responsible for the operation of the
  supercomputers, for user support, for RD work in
  the field of computer and computational science,
  for education and training
• NIC is responsible for the peer-reviewed
  provision of computer time to national and
  European projects

11
Supercomputing at FZJ (II)

• NIC has been founded in 1987 by
• Forschungszentrum Jülich (FZJ),
• Deutsches Elektronen-Synchrotron (DESY),
• Gesellschaft für Mathematik und Datenverarbeitung
  (GMD)
• First and one of three German national
  High-Performance Computing Centres
• Restructured in 1998, now supported by FZJ, DESY
  and since April 2006 by GSI (Gesellschaft
  für Schwerionenforschung)

12
NIC Organisation Chart
Assembly of Contract Partners Chairmen of
Board of Directors of DESY, FZJ, GSI
Scientific Council
Peer Review Board
Board of Directors Director FZJ/ZAM, Chairman
Board Member of DESY Board Member of GSI
NIC Scientific Secretariat
FZJ Operation of the supercomputers at
JSC Research Group Computational Biology
and Biophysics
DESY Operation of special-purpose computers
in Zeuthen Research Group Elementary Particle
Physics
GSI Provision of computer
resources Research Group Nuclear and Hadron
Physics
13
Future Organisation Structure
German Research School (GRS)
Jülich Platform for Simulation Sciences (JPSS)
IAS Institute for Advanced Simulation
Institute forComputational Nanoscience
John von Neumann Institute for Computing
Institute forComputational Biology
Institute for Systems Biology
Jülich-Aachen Research Alliance JARA-SIM
Jülich Supercomputing Centre (JSC)
Partnership for Advanced Computing in Europe
(PRACE)
14
FZJ Dual Supercomputer Complex

• 2004
• 2005/6
• 2007/8
• 2009/10

IBM p690 JUMP
IBM Blue Gene/LJUBL
IBM Blue Gene/Pgt 220 Teraflop/s
JUMP successorgt 150 Teraflop/s
File ServerFunctionality
JUMP successorgt 150 Teraflop/s
File ServerFunctionality
15
JUGENE Jülich Blue Gene/P

• IBM Blue Gene/P
• 16 racks65,536 processorsPowerPC 450, 850 MHz
• 222.8 Tflop/s peak
• 32 TByte
• Small footprint
• Low power consumption per processor
• Outstanding price-performance ratio
• Highly scalable leadership-class system, No 1 in
  Europe

16
JUGENE Jülich Blue Gene/P
17
NIC User Research Fields
JUM
P
JUB
L
120 Projects
20 Projects
18
NIC Users and Access

• Eligibility Proposals accepted from Germany
  and Europe From academia, research
  institutions and industry
• Procedure Peer review by NIC Scientific
  Council International referees Scientific
  quality counts One year grants
• Access via Grid

19
European Users

• DEISA Distributed Super-computing
  Infrastructure
• I3HP Hadron Physics
• NIC Initiative towards new Member States
• Other Collaborations

20
Supercomputingin Germany
DKRZ Hamburg
HLRN Berlin
HLRN Hannover
Wuppertal
NIC Jülich
Dresden
Aachen
DWD Offenbach
HLRS Stuttgart
RZG Garching
LRZ Garching
21
National HPC Pyramid
National HPC centres
Garching, Jülich, Stuttgart
3
Aachen, Berlin (ZIB), DKRZ, Dresden, DWD,
Karlsruhe, Hannover, MPG/RZG, Paderborn
Topical HPC centres, centres with regional tasks
10
100
HPC-Server
University/Institute
22
Gauss Centre for Supercomputing

• Alliance of the three German SC centres HLRS,
  LRZ, NIC
• Largest supercomputer complex in Europe
• German representative in PRACE
• Applicant for an European supercomputer centre in
  FP7
• More information http//www.gauss-centre.eu

23
PRACE

• PRACE Partnership for Advanced Computing in
  Europe
• Consortium of Austria, Finland, France,
  Germany, Greece, Irland, Italy, the Netherlands,
  Norway, Poland, Portugal, Spain, Switzerland,
  Turkey and United Kingdom. German
  Representative GCS
• Memorandum of Understanding signed April 17, 2007
  
• Submission of a joint application (also named
  PRACE) to the EU for the realization of the
  Preparatory Phase on May 2, 2007
• Launch of PRACE project Jan 1, 2008

24
PRACE Towards European HPC
GENCI
Ecosystem
tier 1
tier 0
Principal Partners
General Partners
Associated Partner
25
Part 2
Simulation Laboratories
26
Traditional User Support
Advisor
Scientific partnerships
Specialist
Methods and optimisation
Technical support Standby team
Helpdesk
27
Improved User Support
Scientific partnerships Strong links with
universities Simulation Laboratories (SL)
SL orAdvisor
SpecialistParaTeam
Methods and optimisation
Technical support Standby team
Helpdesk
28
Simulation Laboratory Idea
In analogy to major experimental facilities
like LHC_at_CERN or ILL Grenoble or Hubble
telescope provide specialised instruments to
specific user groups. Application codes and
their associated analysis tools are seen as the
instruments of computational science ?
simulation laboratory
29
Simulation Laboratory Concept

• An SL is a targeted research and support
  organisation for a scientific community
• An SL cooperates with its community and supports
  it in performing simulations on supercomputers.
• SLs are adjusted according to the needs of the
  scientific communities.
• Each SL is itself part of its community and
  strengthens it.
• An SL consists of a core group located at a
  super-computer or a topical centre and a number
  of associated scientists at universities and
  research institutes.
• SLs should act in a structuring way.

30
Community-oriented Simulation Laboratory
supercomputing or topical centre
core group
Access to all necessary facilities at the
supercomputing or topical centre, e.g. VR
projection system
2 scientists, 1 PhD, 1 technician
associated communitymembers
universities, research institutes
SL
community groups
31
Simulation Laboratories
SL QCD
Supercomputing Centre
Topical Center DESY/NIC
Multi- Gigabit Backend Network
Topical Center GSI/NIC
Topical Center GridKa
High-end System
Global ParallelFile System
gt 150
T
eraFlop/s
Multi-PBTape Archive
Teams
NIC
SL QuantumChemistry
SL Earth Env, Sc.
Support
Capability
Computing
Topical Center ICG/FZJ
Topical Center JSC/FZJ
SL Nanoscience
SL Plasma Physics
SL Biology
Topical Center AWI
Topical Center IFF/FZJ
Topical Center IEF-4/FZJ

Topical Center NIC
32
Simulation Laboratory Tasks

• Disciplinary science
• Participation in research projects of the
  community
• Participation in consortia, doing 3rd
  party-funded research
• Software development and maintenance for the
  community
• Data management for the community, data curation
• Supercomputing
• Development of computational methods for HPC
  systems
• Adoption of software to new and highly scalable
  HPC systems (Cell, Multi-Core etc.)
• Training and education
• Training in computational methods for HPC systems
• Participation in education
• Community building
• Web portal, organisation of workshops, scientific
  exchange

33
SL Organisational Structure I

• Core Group
• staffing1 experienced scientific head,
  well-accepted by the community,1 PostDoc with
  experiences in development of methods 1 PhD
  student, 1 technician
• Associated scientists
• The research groups of the community should name
  associated members of the simulation laboratory
  who work at their home institutions, but
  participate actively in the work of the
  simulation laboratory.

34
SL Organisational Structure II

• Steering Committee
• Established by the HPC centre and the community
• Consists of representatives of the different
  research branches and the head of the core group
• Defines the work plan, evaluates incoming work
  package proposals and decides which work packages
  should be executed by the SL, when and how long.
• Calls a detailed documentation of the work
  distribution between core group and associated
  members
• Accompanying measures
• SL offers a formal guest program
• SL receives a certain amount of supercomputer
  resources for software development and testing
  tasks

35
Cross-Sectional Groups

• Tasks
• Basic research in computational methods
• Assist simulation scientists in applying methods
• Examples
• Numerical algorithms and application software,
  visualisation tools
• Programming tools
• Grid Services, data management and exploration

36
SLs and Cross-Sectional Groups
37
Simulation Laboratory Example
Simulation Laboratory Plasma Physics
Community Groups
FZ Jülich
RU Bochum
IEF-4
JSC
U Düsseldorf
SL PP
RWTH Aachen
Ranking
Steering Committee
WP 1
WP 3
WP 4
WP 2
WP 3
WP 4
WP 1
WP 2
Work packages
38
Simulation Laboratory Example
Simulation Laboratory Plasma Physics
CG RWTH Aachen
JSC
SL PP
WP execution

• Consultation working group
• Task analysis / Goal definitions
• Work allocation SL/CG
• Code analysis
• Algorithm (re)development
• Performance tuning
• Testing/verification
• Scalability certificate
• Reporting / dissemination

Continuous activities

• Algorithm RD
• Software management
• Library development
• Code refactoring
• Workshop hosting
• Guest programme

39
Summary

• The European HPC eco system is being formed by
  PRACE new structures and a new level of
  performance.
• As the national German partner in PRACE, the
  Gauss centre has been founded (Garching, Jülich,
  Stuttgart).
• Simulation Laboratories are a required component
  of the European HPC eco system significantly
  increasing the quality of user support.
• Simulation Laboratories serve as structure
  building institutions on a regional, national and
  European scale.