Observing System Evaluation

1
Observing System Evaluation
2
Talk Outline

• Methodology
• Observing System Experiments (OSEs) and
• Observing System Simulation Experiments (OSSEs)
• Caveats
• Number of altimeter missions
• Complimentary data types
• Assessing the value of future observation types
  and observing system design
• Emerging techniques
• Conclusions
• Acknowledgements

3
Introduction OSEs and OSSEs
Assimilated and with-held observations
Assimilated observations
Evaluation/ Validation
Forecast or BGF
Analysis or Forecast
4
Introduction OSEs and OSSEs
Assimilated and with-held observations
Assimilated observations
Evaluation/ Validation
Forecast or BGF
Analysis or Forecast
T/S

• Observing System Experiments (OSEs)
• Assimilate real observations
• Systematically with-hold observation types

5
Introduction OSEs and OSSEs
Simulated observations
Assimilated observations
Evaluation/ Validation
Forecast or BGF
Analysis or Forecast

• Observing System Simulation Experiments (OSSEs)
• Assimilate pretend observations from a model
• Systematically include different observation
  types including future observation types

• Observing System Experiments (OSEs)
• Assimilate real observations
• Systematically with-hold observation types

6
Compatibility of Models and Observations
Observations and analysis/prediction systems are
not a perfect match. GODAE systems are still
being developed and improved. No system can yet
fully exploit all observations.
7

• Number of Altimeters

8
Number of altimeter missions DUACS system
RMS of SLA signal using data from 4 missions
Difference between RMS of SLA signal using data
from 4 missions and 2 missions
1/3o DUACS system October 2002 -August 2003
during the T/P interleaved mission
Approximately half the SLA signal is lost when
data from only 2 missions is used. Errors of up
to 10 cm.
Pascual A., Y. Faugère, G. Larnicol and P.-Y. Le
Traon (2006) Improved description of the ocean
mesoscale variability by combining four satellite
altimeters, Geophys. Res. Lett., 33,
doi10.1029/2005 GL024633.
9
Number of altimeter missions DUACS system
1/3o DUACS system October 2002 -August 2003
during the T/P interleaved mission
Data from 4 altimeters in NRT equivalent to 2
altimeters in delayed-mode
Pascual A., C. Boone, G Larnicol, P.Y. Le Traon
(2008) On the quality of real time altimeter
gridded fields comparison with in situ data,
Journal of Atmospheric and Oceanic Technology, in
press.
10
Number of altimeter missions Mercator system

• 7-day forecast is under operational conditions
• Nowcast is under operational conditions
• Hindcast uses delayed-time data

RMS errors compared to Jason-1 SLA (m)

• 5-7km res Mercator System
• North Atlantic
• MvOI
• 6-month OSEs
• 2004/05 during T/P Jason-1 tandem missions

Data from 4 altimeters in NRT equivalent to 2
altimeters in delayed-mode
Benkiran, M., E. Greiner, S. Giraud St Albin, E.
Dombrowsky, D. Jourdan and M. Faillot (2008)
Impact study of the number Space Altimetry
observing systems on the altimeter data
assimilation in the Mercator-Ocean system. In
preparation.
11
Number of altimeter missions UK Met system
1/9o FOAM System North Atlantic OI 3-month
OSEs 2006
Martin, M.J., A. Hines, M. J. Bell (2007) Data
assimilation in the FOAM operational short-range
ocean forecasting system a description of the
scheme and its impact. Q. J. R. Met. Soc., 133,
981-995.
12

• Complimentary data types

13
Complimentary data types mesoscale prediction

• 1/10o Bluelink system
• 6-month long OSEs starting December 2005

Oke, P. R., and A. Schiller (2007) Impact of
Argo, SST and altimeter data on an eddy-resolving
ocean reanalysis. Geophys. Res. Lett., 34,
L19601, doi10.1029/2007GL031549.
14
Complimentary data types mesoscale prediction
Estimated SLA Errors

• 1/10o Bluelink system
• 6-month long OSEs starting December 2005

SST and Argo partially compensate for no ALTIM
but ALTIM is clearly necessary to represent the
mesoscale
Oke, P. R., and A. Schiller (2007) Impact of
Argo, SST and altimeter data on an eddy-resolving
ocean reanalysis. Geophys. Res. Lett., 34,
L19601, doi10.1029/2007GL031549.
0 8 16
24 RMS residuals (cm)
15
Complimentary data types mesoscale prediction
Estimated SST Errors
1/10o Bluelink system 6-month long OSEs starting
December 2005
Argo partially compensates for no SST but not
over wide shelfs and shallow seas
Oke, P. R., and A. Schiller (2007) Impact of
Argo, SST and altimeter data on an eddy-resolving
ocean reanalysis. Geophys. Res. Lett., 34,
L19601, doi10.1029/2007GL031549.
16
Complimentary data types seasonal prediction

• 1o ORA-S3
• ECMWFs Seasonal Prediction System
• 6-year OSEs producing 1-7 month forecasts
  (2001-2006)

Seasonal forecast skill degrades by 10-20 when
Argo T/S is not assimilated
Argo T/S is particularly important for seasonal
prediction ? thermocline, stratification, heat
content
Balmaseda, M. A., and D. Anderson (2008b) Impact
on initialization strategies and observations on
seasonal forecast skill. Geophys. Res. Lett.,
submitted.
17

• Assessing the value of future observation types

18
Assessing the value of future observation types
SMOS

• 1/3o Mercator System
• North Atlantic
• SEEK filter
• 1-year OSSEs
• 2003
• SMOS errors are spatially varying _at_ 0.2-2.5 psu
  depending on brightness temp, SST wind.

SMOS w/ 2x expected errors SMOS w/ 1/2x expected
errors SMOS w/ expected errors Control Run

• SMOS/Aquarius data should be very beneficial
• but depends on errors

Tranchant, B., C.E. Testut, L. Renault, N. Ferry,
F. Birol, P. Brasseur (2008) Expected impact of
the future SMOS and Acquarius ocean surface
salinity missions in the Mercator Ocean
operational systems new perspectives to monitor
ocean circulation. Remote Sens. Env., 112,
1476-1487.
19

• Observing system design

20
Observing system design
Pressure and vertical velocity perturbation
fields associated with the leading singular
vector ? the optimal perturbation for the target
region.

• 1/10o JMA MRI.COM System
• North Pacific
• Tangent linear and Adjoint to identify the
  fastest growing perturbations that lead to the
  Kuroshio meander using singular vector analysis.

Modelled SSH after the perturbation is applied
Perturbations _at_ 1000-1500m depth around 132E, 31N
lead to a meander in the Kuoshio ? could
compliment GOOS with extra obs here
Fujii, Y., H. Tsujino, N. Usui, H. Nakano, and M.
Kamachi (2008a) Application of singular vector
analysis to the Kuroshio large meander. J.
Geophys. Res., 113, C07026, doi10.1029/2007JC0044
76.
21

• Emerging techniques

22
Analysis sensitivity and adjoint techniques

• Conventional OSEs are not ideal
• very expensive cannot be performed routinely
• removing one type of observation may change the
  impact of a different observation type
• Adjoint techniques and analysis sensitivities
• relatively inexpensive
• show impact of all individual observation in a
  single calculation
• could be applied routinely to operational
  systems

23
Analysis sensitivity
How much information does a specific assimilation
system get out of different observation
types? ?analysis self-sensitivities are readily
estimated ?impact of individual obs on
analyses ?help quantify observation impact and
system performance/problems
?
1/10o Bluelink system A single estimate from
1/1/2007 Australian Region
?
?
Chapnik, B., G. Desroziers, F. Rabier and O.
Talagrand (2006) Diagnosis and tuning of
observational error in a quasi-operational data
assimilation setting, Quarterly Journal of the
Royal Meteorological Society, 132, 543-565.
24
Adjoint Technique
Observation sensitivity gradients (?J/?y),
quantify the impact of individual observations on
forecasts
Coming soon to an NRL lab near you !
Langland and Baker (2004) Estimation of
observation impact using the NRL atmospheric
variational data assimilation adjoint system.
Tellus, 56A, 189-201.
Many moderately beneficial Radiosonde impacts in
CONUS and Europe best outcome
criteria Langland and Baker (2004)
25
Conclusions

• Number of altimeter missions
• 4 missions in NRT 2 missions in delayed-time
• Complimentary data types
• Different data types have different information
• ALTIM critical for mesoscale
• SST critical for mixed layer properties and bias
  reduction
• Argo is the only constraint on salinity critical
  for stratification, heat contents and seasonal
  prediction
• Assessing the value of future observation types
• SMOS/Aquarius program looks promising the
  magnitude of the error will be important
• Observing system design
• GODAE products can be readily employed to gain
  insights for the design of observing systems we
  should get use to using them.
• Emerging techniques
• routine observing system assessment like NWP
  gaining momentum

Operational ocean forecasting critically depends
on altimeter, Argo and SST observations.
?
Non-operational
?
?
Operational
26
Acknowledgement

• Most of the work presented here is based on
  presentations at the inaugural OOPC-GODAE meeting
  on OSSEs and OSEs at UNESCO/IOC in Paris, France
  in November 2007 (www.godae.org/OSSE-OSE.html).
• Continued international coordination and
  cooperation on observing system design and
  evaluation is very important for operational
  oceanography.

27
Conclusions

• Number of altimeter missions
• 4 missions in NRT 2 missions in delayed-time
• Complimentary data types
• Different data types have different information
• ALTIM critical for mesoscale
• SST critical for mixed layer properties and bias
  reduction
• Argo is the only constraint on salinity critical
  for stratification, heat contents and seasonal
  prediction
• Assessing the value of future observation types
• SMOS/Aquarius program looks promising the
  magnitude of the error will be important
• Observing system design
• GODAE products can be readily employed to gain
  insights for the design of observing systems we
  should get use to using them.
• Emerging techniques
• routine observing system assessment like NWP
  gaining momentum

Operational ocean forecasting critically depends
on altimeter, Argo and SST observations.
?
?
?
28
Thankyou
29
Complimentary data types mesoscale prediction

• 1/10o Bluelink system
• 6-month long OSEs starting December 2005

No data types are redundant All contribute
different information
Oke, P. R., and A. Schiller (2007) Impact of
Argo, SST and altimeter data on an eddy-resolving
ocean reanalysis. Geophys. Res. Lett., 34,
L19601, doi10.1029/2007GL031549.
30
Degradation 0 ? no change from REF
Degradation 50 ? Error increased by 50 of
observed signal compared to REF Degradation
-50 ? Error decreased by 50 of observed signal
compared to REF
Benkiran, M., E. Greiner, S. Giraud St Albin, E.
Dombrowsky, D. Jourdan and M. Faillot (2008)
Impact study of the number Space Altimetry
observing systems on the altimeter data
assimilation in the Mercator-Ocean system. In
preparation.
31
OSSEs using observation-based mapping systems
Larnicol, G., S. Guinehut, M.-H. Rio, M.
Drevillon, Y. Faugere and G. Nicolas (2006) The
Global Observed Ocean Products of the French
Mercator Project, Proceedings of 15 Years of
progress in radar altimetry Symposium, ESA
Special Publication, pp. 614.
32
OSEs using the Bluelink reanalysis system
RMSE of different data types for each
OSE Conclusions Complimentary nature of
observation types No data types are redundant
Oke, P. R., and A. Schiller (2007) Impact of
Argo, SST and altimeter data on an eddy-resolving
ocean reanalysis. Geophys. Res. Lett., 34,
L19601, doi10.1029/2007GL031549.
33
OSEs using observation-based mapping systems

• Synthetic T/S by projecting of SLA and SST (after
  Guinehut et al. 2004)
• Impact of combining Argo data with synthetic
  profiles
• Figure shows RMSE of T/S (compared to Argo T/S)
• gt 40 of T signal from synthetic method alone
• Additional 10-20 of signal by adding Argo
• Conclusion Complimentary nature of observation
  types

Larnicol, G., S. Guinehut, M.-H. Rio, M.
Drevillon, Y. Faugere and G. Nicolas (2006) The
Global Observed Ocean Products of the French
Mercator Project, Proceedings of 15 Years of
progress in radar altimetry Symposium, ESA
Special Publication, pp. 614. Guinehut, S., P. Y.
Le Traon, G. Larnicol and S. Phillips (2004)
Combining Argo and remote-sensing data to
estimate the ocean three-dimensional temperature
fields - a first approach based on simulated
observations. Journal of Marine Systems, 46,
85-98.
34
OSSEs using the Bluelink system

• Ensemble-based approach to look for mooring
  locations that when mapped will minimise
  analysis error
• Example uses 3 different models, with different
  resolution, run for different times
• Conclusions
• Most important mooring of the proposed array is
  around 90E

Sakov, P., and P. R. Oke (2008) Objective array
design Application to the tropical Indian Ocean,
J. Atmos. Ocean. Tech., 25, 794-807.
35
OSEs using the UK Met Office system
3-month long OSEs starting January 2006 Assess
the impact of number of altimeters on forecast
skill of the UK system (left) anomaly correlation
of forecast SLA and along-track SLA (right)
anomaly correlation between forecast (U,V) and
drifter Conclusions 1 altimeter much better
than none Even a benefit of 4 altimeters to 3
altimeters Different altimeters have different
impact
Martin, M.J., A. Hines, M. J. Bell (2007) Data
assimilation in the FOAM operational short-range
ocean forecasting system a description of the
scheme and its impact. Q. J. R. Met. Soc., 133,
981-995.
36
Observation Impact Concept -
Observations move the forecast from the
background trajectory (Xb) to the trajectory
starting from the new analysis (Xa)
Observation impact is the combined effect of
all of the observations on the difference in
forecast error (ef - eg)
Xg
?
OBSERVATIONS ASSIMILATED
eg
Xf
?
Forecast Error
Xb
ef
?
?
VERIFYING ANALYSIS
Xa
?
t-24 t0
t24
Forecast Lead Time
37
Adjoint-based Data Impact System
Using observation sensitivity gradients (?J/?y),
the impact of any or all observations assimilated
on a measure of forecast error (J) can be
estimated from
Forecast error difference from observations
Adjoint sensitivity gradients in model grid-point
space
Observations assimilated
Adjoint of assimilation
38
Observation impact interpretation -
For any observation assimilated, if ...
lt 0.0 the observation is BENEFICIAL gt
0.0 the observation is NON-BENEFICIAL
forecast errors decrease
the effect
of the observation is to make the error of the
forecast started from xa less than the error of
the forecast started from xb
forecast errors increase
the effect of the
observation is to make the error of the forecast
started from xa greater than the error of the
forecast started from xb
39
Identification of observation impacts -

• Beneficial impacts
• associated with observations more accurate than
  the background in regions where adjoint
  sensitivity gradients are large
• extreme beneficial impacts from isolated
  observations indicate the need for greater
  observation density
• Non-beneficial impacts
• not expected, all observations should decrease
  forecast error
• if occurs, look for problems in data QC,
  instrument accuracy, model error, specification
  of assimilation error statistics

• Best Outcome
• - many observations that produce equal or similar
  impacts, not few, isolated observations that
  produce large impacts

40
NOGAPS Radiosonde Profile Observation Impact
1 Jan 28 Feb 2006
Most beneficial (04320)
Least beneficial (84401)
Many moderately beneficial Radiosonde impacts in
CONUS and Europe best outcome
criteria Langland and Baker (2004)
Most beneficial (lt - 0.1 J kg-1)
41
Argo Oceanographic Analog to Radiosonde Network
Floats profile from 2000 m to the surface every
10 days, measuring temperature and salinity (TS)
as a function of depth. Adjoint-based system will
determine impact of each float cycle on reducing
model forecast errors. Impact results can be
used to help guide float deployment strategies.
42
Multiple Uses of Adjoint-based System -
Hypothetical Observations and Targeted Observing
are variants of core Observation Impact system
43
Introduction OSEs and OSSEs
OSEs and OSSEs not Aussies
44
Number of altimeter missions Mercator system

• 5-7km res Mercator System
• North Atlantic
• MvOI
• 6-month OSEs
• 2004/05 during T/P Jason-1 tandem missions

Degradation (RMSEX ALTIM - RMSE3 ALTIM) /
RMSOBS x 100 Zero reference with 3
altimeter missions Positive errors are greater
than with 3 missions Negative errors are less
than with 3 missions . RMSE is based on
difference of 7-day forecast from Jason-1
observations
Data from 4 altimeters in NRT equivalent to 2
altimeters in delayed-mode
Benkiran, M., E. Greiner, S. Giraud St Albin, E.
Dombrowsky, D. Jourdan and M. Faillot (2008)
Impact study of the number Space Altimetry
observing systems on the altimeter data
assimilation in the Mercator-Ocean system. In
preparation.
45
Number of altimeter missions Aviso/CLS system
RMS of EKE signal using data from 4 missions
Difference between RMS of EKE signal using data
from 4 missions and 2 missions
1/3o DUACS system October 2002 -August 2003
during the T/P interleaved mission
Greater than half the EKE signal is lost when
data from only 2 missions is used. Errors of over
400 cm2/s2.
Pascual A., Y. Faugère, G. Larnicol and P.-Y. Le
Traon (2006) Improved description of the ocean
mesoscale variability by combining four satellite
altimeters, Geophys. Res. Lett., 33,
doi10.1029/2005 GL024633.