George Mozdzynski and Yannick Tremolet ECMWF

1
George Mozdzynski and Yannick TremoletECMWF
IFS developments for future systems
2
Outline

• High performance computing in Meteorology
• Integrated Forecasting System (IFS) today
• IFS on future HPC systems
• HPC developments
• IFS developments

3
High performance computing in Meteorology

• Performance, Programmability and Portability
• Operational vs. Research workloads
• Reproducibility, Reusability
• User friendly tools
• Development (PrepIFS, XCdp, MetView)
• Source management (Perforce)
• Debugging (Totalview, DDT)
• Performance and Hardware analysis
• Collaboration (ECMWF/Meteo-France, Hirlam, UM)
• RAPS (Real Applications for Parallel Systems)

4
HPC top500
Higher s of cores gt Less memory per core
Less General Purpose
IBM Roadrunner, Opteron / Cell
IBM BlueGene P
More General Purpose
Cray
XT4 / XT5, Opteron
Hitachi, Opteron
HP 3000,
Xeon IBM 575 Power6 (e.g. ECMWF)

NEC SX8, SX9 SGI
Itanium / Xeon Sun, Opteron/ClearSpeed
Bull, Itanium
Performance
5
Key Architectural Features of a Supercomputer
CPU Performance
Parallel File-system Performance
Interconnect Latency / Bandwidth
MEMORY Latency / Bandwidth
a balancing act to achieve sustained Teraflop
performance
6
Integrated Forecasting System (IFS) today

• IFS Spectral model, Semi Implicit, Semi
  Lagrangian
• 4D-Var, Deterministic Forecast model, Ensemble
  Prediction System, Seasonal Forecasting,
• IFS (Det. F, 4D-Var outer loop)
• TL799L91 25 km now
• TL1279LXXX 16 km 2009
• Mixed MPI/OpenMP
• Do we really want two programming models?
• Mainly Fortran 90/95
• Some C, C, ksh, perl
• Standards adherence (for portability)

7
T399 model (EPS resolution)
Operations
8
T799 model
Operations (eff0.80)
9
IFS physics computational imbalance (T799L91,
384 tasks)
11 imbalance in physics, 5 imbalance (total)
10
IFS on future HPC systems

• This year ECMWF will have two IBM Power6 clusters
  installed each with about 8000 cores
• Within the next 5 to 10 years we expect our
  applications will need to run on systems with gt
  100K cores and continue to meet tight operational
  deadlines
• After this 1M cores?
• Concerns
• Power costs an increasing constraint
• Reliability
• Scaling

11
HPC developments

• Computer architecture
• New materials
• A new programming paradigm?
• Partitioned Global Address Space (PGAS)
• Computer languages
• Government support (USA, Japan, EU) for
  development of future systems
• If nothing new appears, then we will probably get
• modest clock cycle improvements
• Some additional instructions (e.g. SSE)
• And a lot more cores
• Should this influence our science?

12
IFS developments

• Evolution, not revolution
• Development of a totally new model is unlikely
• Scientific developments are continually being
  made, more obs, ...
• Improve IFS scalability
• Technical developments SL halos, load imbalance
  (static and dynamic), more OpenMP, etc.
• Rethinking Algorithms
• Legendre transform, Long-window 4D-Var, EPS
  4D-Var
• Further modularisation
• Fortran 2003
• Fortran 2008 - Coarrays