BODAS: An Ensemblebased Optimal Interpolation System

1
BODASAn Ensemble-based Optimal Interpolation
System
2

• Observations
• Quality control
• Super-observations
• Define H
• Innovations

• Zonal sections Halo
• Construct
• Localised inversions

• Collect increments
• Combine with restart
• Generate diagnostics

3

• Observations O(105)
• Satellite altimeter (OCEANIDS, AVISO)
• Profiles (GTS, USGODAE)
• Quality control
• OFAM Statistics
• Define Obs. Error covariance
• Age weighting
• Super-observations
• Altimeter (along track)
• Profiles
• Define H
• Observation position
• Localisation (Lx,Ly)
• Innovations
• Background (daily average)

4
(No Transcript)
5
Super-observations SLA example
6
Super-observations SLA example
7
Super-observations SLA example
8
(No Transcript)
9
Bluelink ReANalysis (BRAN)

• OFAMBODAS
• Covariances from a 72-member ensemble of
  intraseasonal anomalies

10
Domain Decomposition
11
Domain Decomposition
gives a seamless analysis
12
Bluelink ReANalysis (BRAN) configuration

• Applied to the Ocean Forecasting Australia Model,
  OFAM
• Bluelink Ocean Data Assimilation System, BODAS
• Covariances from a 72-member ensemble of
  intraseasonal anomalies
• Covariances are localised approx. 3o x 3o
  e-folding
• Assimilates observations of
• along-track sea-level anomalies (T/P, ERS, GFO,
  Jason, Envisat)
• coastal tide-gauges from around Australia
• temperature and salinity profiles (from WOCE,
  TAO, Argo etc.)
• Assimilates observations in an 11 day
  data-window, every 3 days using daily mean as the
  background field / forecast
• Inversion is approximated by a psuedo-inverse,
  ensuring the condition number of the truncated
  innovation covariance matrix is lt 1000
• Implemented on 40 processors
• Uses approximately 80 Gb memory
• Takes 20 minutes of wall-clock time to complete
  on an NEC SX6 machine

13
BODAS I/O

• (Dynamic) Observations
• (Static) 72 state vectors, anomalies
• (Static) OFAM statistics
• (Dynamic) Background, OFAM daily average
• (Dynamic) Increment
• (Dynamic) Analysis statistics

14
BODAS Performance
Data type SLA Temp Salt
Total Number of obs 40923 16133
16254 73310 Domain decomposition into
44 jobs
BODAS elapse time is 1953 secs. (32 minutes)
PREP user time is 14.5 of elapse time ( 5
minutes) CALC user time is 37.5 of elapse time
(12 minutes) POST user time is 2.2 of
elapse time ( 1 minutes) Unaccounted time
amounts to 45.8 ( 15 minutes)
15
BODAS Performance
Data type SLA Temp Salt
Total Number of obs 40923 16133
16254 73310 Domain decomposition into
44 jobs
CALC and POST have significant I/O wait times
for data. CALC I/O wait times are due to
loading ensemble members POST I/O wait times
are due to loading restart files
16
Additional Tuning Required
CALC and POST apps perform poorly relative to
elapse time CALC uses up to 272GB of core
memory (6.1GB per cpu) Opportunities to reduce
elapse time thru - system I/O tuning and
optimization - memory mapping of disk files
- integration of PREP, CALC, and POST into
single MPI app
17
BODAS Performance
Data type SLA Temp Salt
Total Number of obs 0 5243
0 5243 Domain decomposition into
20 jobs
PREP is 27 of BODAS elapse time (7
minutes) CALC is 13 of BODAS elapse time (3
minutes) POST is 60 of BODAS elapse time (15
minutes)
18
BODAS Performance
Data type SLA Temp Salt
Total Number of obs 0 5243
0 5243 Domain decomposition into
20 jobs
All BODAS apps have significant I/O wait times
for data. PREP and POST have little vector
processing CALC executes 20 simple jobs in
parallel use decomposition
19
Additional Tuning Required
All BODAS apps perform poorly relative to
elapse time PREP and POST opportunity to reduce
elapse time thru - system I/O tuning and
optimization - parallel execution of serial
tasks via MPI/OpenMP CALC can be improved by
utilizing better I/O management
20
(No Transcript)
21
Dynamical consistency between sea-level
increments and temperature/salinity increments
Sea-level increment
Dynamic height increment (rel. 1000) m)
22
Dynamical consistency
32.5oS ?
23
Domain Decomposition
24
Domain Decomposition
25
Domain Decomposition