Changes at ECMWF

1
Changes at ECMWF

• Walter Zwieflhofer
• Head of Computer Division

2
Operational system (October 2001)

• Data assimilation
• 4D-Var with a 12-hour data window
• Horizontal resolution 120 km for inner and 40 km
  for outer loops
• Global atmospheric forecasts
• Forecasts to 10 days from 12 UTC at 40 km
  resolution
• Ensemble forecasts to 10 days with 51 members at
  80 km resolution
• Four forecasts per day to 3 days to provide
  boundary conditions for some of the ECMWF Member
  States (40 km, 3D-Var)
• Ocean wave forecasts
• Global forecasts to 10 days at 40 km resolution
• European waters forecasts to 5 days at 15 km
  resolution
• Seasonal forecasts
• Global forecasts to 6 months (ensemble with 30
  members)

3
Forecasting-system changes since 1999
19r2 Increased resolution in stratosphere 21r1
Assimilation of raw TOVS/ATOVS radiances 21r4
New error statistics for 4D-Var Jb, SSMI wind
data, corrected use of humidity
observations, increased PBL resolution,
improved cloud, convection and
orography. 22r1 Improved SSM/I QC and second
SSM/I satellite
22r3 New error variances (for Jb and Jo)
additional ATOVS data, improved land-surface,
sea-ice and LW radiation 23r1 12-hourly
4D-Var 23r3 T511/159 resolution, second ATOVS
satellite, increased angular resolution
in wave model
4
Evolution of forecast skill for northern and
southern hemispheres
5
Error distribution for Day 5 (Europe)
6
Annual-mean RMS errors against analyses from WMO
scores
7
Planned improvements

• 2002 Increase of the inner loop resolution of
  4D-Var from 120 km to 80 km
• 2003-2004 Assimilation of cloud and rain in
  4D-Var
• Higher vertical resolution in free
  troposphere and lower stratosphere
• Assimilation of AIRS data
• Operational running of a Severe Weather
  forecasting system
• 2005-2006 Assimilation of IASI data

8
Computer configuration October 2001
9
PrepIFS

• ECMWF tool to prepare forecast and data
  assimilation experiments
• Uses the operational configuration of the
  forecasting system and allows researcher to
  introduce modifications to
• input data (including the selection of
  observations)
• the model code
• the resolution to be used and many other
  parameters
• Written Java, with a client part that can be
  installed on remote machines
• Allows registered external users to configure and
  submit experiments via the Internet

10
Major changes to thecomputing environment

• Desktop systems (procurement completed in 2000)
• Decision was taken to replace all proprietary
  UNIX workstations with PCs running LINUX over a
  period of three years
• Data Handling System DHS (procurement in
  progress)
• A scalable system that will be able to support
  the expected growth over 6 years is required
• Invitation to Tender for a new system was issued
  earlier this year
• Supercomputer HPC (procurement in progress)
• Fujitsu systems are rented and the contract ends
  on 31 Dec 2002
• Invitation to Tender for a replacement to cover
  the period 2003-2006 was issued in March this
  year

11
DHS procurement

• Archive holds currently some 400 TB of data
  (primary data) and has been growing by some 60
  per annum
• Two services need to be supported at ECMWF
• ECFS, a large-scale UNIX-like filesystem for
  general use
• MARS, a customised meteorological archival and
  retrieval system
• Vendors were asked to offer a scalable storage
  system and were given the option to offer a
  replacement for the ECFS and/or MARS software
  layer
• DHS and HPC procurements were separated because
• Lifetime of the data handling systems tends to be
  considerably longer than that of the HPC systems
• Allows a simpler HPC procurement not affected by
  the complexities associated with the replacement
  of the archive system

12
HPC procurement

• ECMWF asked for a phased introduction of
  resources covering the period 2003-2006, with a
  parallel run during the second half of 2002
• The specifications allowed for some flexibility,
  enabling vendors to adapt their response to their
  product development plans. This included the
  option to propose a 3-month shift of the service
  period
• The funds available were published
• Vendors were required to commit to performance
  increases, expressed as factors, over the
  performance of the Fujitsu systems installed at
  ECMWF
• Requirements for disk space, I/O and network
  bandwidth were expressed as function of these
  factors

13
The Benchmark Suite

• IFS tests (for the performance commitments)
• 4D-VAR (T511/255 L60 2, 4 or 6 copies depending
  on PHASE)
• DETERMINISTIC FORECAST (T799 L90 2 or 4 copies)
• EPS (T399 L60 50 or 100 copies)
• Burton Smith taxonomy
• Size g gtgt 2 O L
• Suitable for both T-type and C-type systems
• IFS model runs well on all architectures
• 4D-Var is very sensitive to communication
  bandwidth and has sections on which scalar
  machines do well
• Resolutions representative for the workload
  planned for 2003-2006 were used

14
Observations

• Vendors were allowed to optimize the codes
• However, changes that would affect the
  readability or portability were not permitted
• Some vendors used a hybrid MPI / OpenMP approach
• Some vendors found that such a hybrid approach
  did not improve the results obtained by using MPI
  only
• Percentages of peak
• 4D-Var scaled not as well the deterministic
  forecast and the EPS
• Scalar machines 3 to 12, depending on
  test/product
• Vector machines 14 to 38, depending on
  test/product
• Rule of thumb for ECMWFs codes was confirmed
• 10 for scalar architectures and 30 for vector
  architectures

15
Status of both procurements

• The Tender Evaluation Boards have completed their
  work and have made recommendations to the ECMWF
  Director
• The Director has awarded both contracts, subject
  to approval by ECMWF Council
• Documents detailing the respective procurement
  process and the proposed solutions have been sent
  to the technical and financial committees of the
  ECMWF Council
• These committees will meet in November to
  consider ECMWFs proposals and to make
  recommendations to Council
• Council will meet in December and (hopefully)
  authorize the Director to sign the appropriate
  contracts

16
Changes over the next 18 months
Linux PCs