CURRENT STATUS AND FUTURE PLANS FOR THE NCEP GLOBAL ENSEMBLE FORECAST SYSTEM

1
CURRENT STATUS AND FUTURE PLANS FOR THENCEP
GLOBAL ENSEMBLE FORECAST SYSTEM

• Zoltan Toth
• Yuejian Zhu, Richard Wobus,
• Mozheng Wei, Dingchen Hou, Lacey Holland
• Environmental Modeling Center
• NOAA/NWS/NCEP
• Acknowledgements
• David Michaud, Yuqiu Zhu, Brent Gordon NCO
• Hua-Lu Pan, Steve Lord , Fred Toepfer EMC
• http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

2
OUTLINE

• CONFIGURATION
• MARCH 2004 IMPLEMENTATION
• PLANS
• COMPARISON WITH OTHER CENTERS
• ECMWF, Met. Service Canada
• Ensemble performance strongly influenced by
  quality of data assimilation, model
• INITIAL PERTURBATIONS
• Testing Ensemble Transform Kalman Filter
• Test regional breeding with input from GSI
  analysis scheme
• MODEL PERTURBATIONS
• Multiple model versions (RAS SAS)
• Stochastic perturbations (represent uncertainty
  due to sub-grid scale motions)
• UTILITY OF ENSEMBLE FORECASTS
• Capturing case dependent fluctuations in
  predictability
• Probabilistic forecasting

3
NCEP GLOBAL ENSEMBLE FORECAST SYSTEM
RECENT UPGRADE (Apr. 2003)
NEW CONFIGURATION MARCH 2004
10/50/60 reduction in initial perturbation size
over NH/TR/SH
CURRENT SYSTEM
4
MARCH 2004 CHANGES - SUMMARY
1. Extend T126 portion of forecast from 84 to
180 hours
2. Run 4 cycles per day, adding 06Z and 18Z
forecasts
3. Extend out to 180 hrs the high resolution
enspost and ensstat files, and add new
precipitation-related enspost and ensstat files
4. Shorten the time step for the T126 part of
the forecast
5
RMS ERROR, ENSEMBLE SPREAD, OUTLIER STATISTICS, NH
RESULTS BASED ON 29-DAY NCO PARALLEL TESTING
RMS error slightly reduced Ensemble spread
significantly increased
Number of cases when ensemble misses verification
significantly reduced
6
PATTERN ANOMALY CORREL, RANKED PROBABILITY SKILL
SCORE, NH
PAC slightly improved
Probabilistic skill scores improve by 0.5 day
around day 7 lead time
7
PAC FOR SH, BRIER SKILL SCORE FOR TROPICS
Performance improves over SH extratropics as well
Both reliability and resolution components of
probabilistic performance improves over the
tropics
8
ENSEMBLE CONFIGURATION
USE OF CURRENT CONFIGURATION Time lagged approach
Is this helpful? Fixed number of 10 members,
mix in older members with newest members Not
much impact from lagged approach Increasing
membership (10 newest 10 older10 even older 10
oldest) Adding even more members may
help FUTURE CONFIGURATIONS Complex problem
Difficult to assess objectively Answer depends
on Scientific merit of proposed
configurations Applications (Users, statistical
post-processing, etc) Based on evidence (in-house
and outside) experience, judgement
call Trade-offs between Spatial resolution
(horizontal vertical) vs. membership Downgradin
g model with lead time vs. using same model (for
easier bias correction) Guideline for computer
resources used in global forecast system Hires
control (GFS) Ensemble suite Examples 10-membe
r ensemble with half horizontal
resolution 20-member ensemble with half
horizontal vertical resolution
9
TIME LAGGED ENSEMBLES
10-MEMBER ENSEMBLES Mix fcsts from latest cycle
with 6, 12, 18-hrs older members Week 1 Little
negative effect on skill (Outlier stats improved
due to increased spread) Week 2 Slight positive
(or no) effect on skill Not much impact from
lagged approach
Control Low res. cntrl 10m Latest 10 10m1 55
latest 10m2 433 latest 10m3 3322 latest
10
TIME LAGGED ENSEMBLES
10, 20, 30, 40-MEMBER ENSEMBLES Add more and
more older members Days 1-7 Some negative effect
on skill Days 8-14 Positive effect on skill
Adding even more members may help
GFS Hires control 10m Latest 10 20m 00Z18Z 30m 00
Z18Z12Z 40m 00Z18Z12Z06Z
11
COMPARISON OF ECMWF, MSC, AND NCEP ENSEMBLES
12
PATTERN ANOMALY CORRELATION (PAC)

• METHODCompute standard PAC for
• Ensemble mean Control fcsts
• EVALUATION
• Higher control score due to better
• Analysis NWP model
• Higher ensemble mean score due to
• Analysis, NWP model, AND
• Ensemble techniques
• RESULTS
• CONTROL
• ECMWF best throughout
• Good analysis/model
• ENSEMBLE VS. CONTROL
• CANADIAN poorer than hires control
• Poorer (old OI) ensemble analysis
• NCEP performs well compared to control
• Despite lack of model perturbations
• ENSEMBLE
• ECMWF best throughout

Y. Zhu et al.
13
PERTURBATION VS. ERROR CORRELATION ANALYSIS (PECA)

• METHOD Compute correlation between ens
  perturbtns and error in control fcst for
• Individual members
• Optimal combination of members
• Each ensemble
• Various areas, all lead time
• EVALUATION Large correlation indicates ens
  captures error in control forecast
• Caveat errors defined by analysis
• RESULTS
• Canadian best on large scales
• Benefit of model diversity?
• ECMWF gains most from combinations
• Benefit of orthogonalization?
• NCEP best on small scale, short term
• Benefit of breeding (best estimate initial
  error)?
• PECA increases with lead time
• Lyapunov convergence
• Nonlilnear saturation
• Higher values on small scales

M. Wei
14
EXPLAINED ERROR VARIANCE AS A FUNCTION OF
ENSEMBLE SIZE

• METHOD Compute correlation between ens
  perturbtns and error in control fcst for
• Individual members
• Optimal combination of members
• Each ensemble
• Various areas, all lead time
• EVALUATION Large correlation indicates ens
  captures error in control forecast
• Caveat errors defined by analysis
• RESULTS
• SPATIAL SCALES
• Global/hemispheric scales No saturation seen up
  to 50
• Continental scales Gains level off, especially
  at longer lead
• LEAD TIME
• Very little gain beyond 30 members at longer
  ranges

M. Wei
15
SUMMARY OF 3-WAY INTERCOMPARISON RESULTS

• Results depend on time period
• CONTROL FORECAST
• ECMWF best overall control forecast
• Best analysis/forecast system
• ENSEMBLE FORECAST SYSTEM
• Difficult to separate effect of analysis/model
  quality
• ECMWF best overall performance
• NCEP
• Days 1-3 - Very good (best for PECA)
• Value of breeding?
• Beyond day 3 Poorer performance
• Lack of model perturbations
• CANADIAN
• Days 6-10 Better than NCEP
• Value of model diversity?

16
GENERATION OF INITIAL ENSEMBLE PERTURBATIONS

• CURRENT METHOD Breeding technique (Toth Kalnay
  1993,1997)
• Characteristics
• Nonlinearly fastest growing perturbations -
  Desirable, want to keep
• Initial amplitudes rescaled to the same level
  every day - Not realistic, Make amplitudes
  dependent on actual observational density
• Multiple breeding cycles run independently
  Possible dependency
• Make initial perturbations orthogonal
• POSSIBLE UPGRADE ETKF method (Bishop et al.
  01, Wang Bishop 03)
• Similar to breeding Based on same principle,
  cycled perturbations
• Different from breeding Perturb. amplitudes
  depend on obs locations/errors
• Initial perturbations orthogonal in
  observ. space
• CAVEAT
• ETKF initial perturbation amplitudes reflect
  analysis uncertainty in ETKF (and not SSI)
  technique
• For a fully consistent data assimilation /
  ensemble sytem
• Develop full ETKF data assim./ensemble system
  (THORPEX project)
• Revise regional rescaling in breeding, use
  uncertainty information from grid-point based
  3D/4D-VAR (GSI)

17
CONCEPTUAL DESCRIPTION OF USING THE ETKF
METHODTO GENERATE INITIAL ENSEMBLE PERTURBATIONS

• ETKF - Statistical procedure developed to
  assimilate observational data into NWP models
• INPUT Ensemble of (N) first guess (FG, 6-hr)
  fcst fields, valid at analysis time
• Observational data
• OUTPUT Ensemble of N perturbed analysis fields
  valid at analysis time
• PROCEDURE Linear combination of FG fields
• CONSTRAINTS Fit observations, given assumed
  observational errors (and FG errors)
• Use subspace of FG ensemble as error
  covariance matrix
• (N-1) orthogonal perturbations in space of
  observations
• USE OF ETKF FOR GENERATING INITIAL PERTURBATIONS
  -
• Statistical procedure to estimate error
  variance/covariance in analysis field derived by
  SSI
• ALGORITHM As above, except
• INPUT Observation locations - observation
  values not used
• OUTPUT Ensemble of N perturbation fields - not
  full fields, to be added to SSI analysis
• CONSTRAINTS Force ensemble variance to be
  consistent with uncertainty estimate from
  ETKF-DA algorithm

18
Differences in simulated analysis uncertainty due
to insertion of WSR (Winter Storm Reconnaissance)
data (20 dropsonde observations each of 7 day)
1-2 REDUCED VARIANCE
SIMULATING THE EFFECT OF VARIATIONS IN DATA
COVERAGE
M. Wei
19
Vertical Distribution of Energy Perts (Spread)
SIMILAR SHAPE
ETKF VS. BREEDING CHARACTERISTICS COMPARISON
Latitudinal Distribution of Energy Perts
DIFFERENT DISTRIBUTION
ETKF Too high amplitude in extratropics? Too
low amplitude in tropics?
M. Wei
20
Amplification factor of perts in energy norm at
500mb
ETKF VS. BREEDING CHARACTERISTICS COMPARISON
Independent degrees of freedom (dof) SIMILAR DOF
VALUES
SAME OR SLIGHTLY LOWER GROWTH RATES FOR ETKF
6-hr amplification factor of pert. energy,
500hPa
GL TR NH SH
Breeding 1.212 1.139 1.246 1.283
ETKF 1.194 1.258 1.212 1.181
M. Wei
21
ETKF VS. BREEDING CHARACTERISTICS COMPARISON
Explained forecast error variance (PECA) BREEDING
BETTER AT SHORT LEADS
M. Wei
Forecast vs. actual 6-hr error variance SIMILAR
PERFORMANCE ETKF better at low, Breeding at high
values?
22
SUMMARY OF ETKF VS. BREEDING COMPARISON

• Positive Aspects for ETKF
• ETKF perturbations respond to variations in
  distributions of observations
• Degrees of freedom
• Slight increase for subspace of global initial
  perturbations in observational space
• No significant effect in grid-point space or on
  smaller domains
• Neutral Results
• Error variance prediction similar
• Growth rates similar (perhaps breeding slightly
  higher)
• Negative Aspects for ETKF
• Latitudinal distribution of initial perturbation
  amplitude looks unrealistic
• Effect of low growth rates in Tropics (model
  bias not considered good or bad?)
• Effect of radiance data not simulated
• Noisier variance distributions in time and space
• Less error covariance (PECA) explained at short
  lead
• Future work
• Test results with 40-member ensemble

23

• ACCOUNTING FOR MODEL RELATED ERRORS
• USE OF IMPERFECT MODELS LEADS TO
• Closure/truncation errors related to
• Spatial resolution
• Time step
• Type of physical processes explicitly resolved
• Parameterization scheme chosen
• Structure of scheme
• Choice of parameters
• Geographical domain resolved
• Boundary condition related uncertainty (Coupling)
• NOTES
• Two main (initial cond. vs. model) sources of
  forecast errors hard to separate gt
• Very little information is available on model
  related errors
• Tendency to attribute all forecast errors to
  model problems
• Houtekamer, Buizza, Smith, Orrell, Vannitsem,
  Hansen, etc

24

• SAMPLING FORECAST ERRORS
• REPRESENTING ERRORS DUE TO USE OF
• IMPERFECT MODELS
• CURRENT METHODS
• Change structure of model (eg, use different
  convective schemes, etc, MSC)
• Model version fixed, whereas model error varies
  in time
• Random/stochastic errors not addressed
• Difficult to maintain
• TESTING COMBINED USE OF RAS SAS convective
  schemes on different members
• Add stochastic noise (eg, perturb diabatic
  forcing, ECMWF)
• Small scales perturbed
• If otherwise same model used, larger scale
  biases may not be addressed
• DEVELOPING A STOCHASTIC PERTURB. METHOD (with H.
  Juang and M. Iredell)
• NEED NEW
• MORE COMPREHENSIVE AND
• THEORETICALLY APPEALING
• 
  APPROACH

25
RESULTS FROM COMBINED USE OF RAS SAS
NO POSITIVE EFFECT ON PRECIP OR HEIGHT SCORES
500 hPa height RMS error, NH extratr.SAS, RAS,
Combination
Precipitation Forecast Scores Day 3SAS, RAS,
Combination
26
RESULTS FROM COMBINED USE OF RAS SAS
CONVECTIVE SCHEME DOES NOT SEEM TO HAVE PROFOUND
INFLUENCE ON FORECASTS EXCEPT PRECIP
500 hPa height NH extratrop. RMS error for RAS,
SAS, and NAS (no convection) NO DIFFERENCE
WHETHER CONVECTIVE SCHEME IS USED OR NOT
Rank histogram comparing distributions of
sub-ensembles relative to each other AFTER BIAS
CORRECTION, SAS RAS SUB-ENSEMBLES COVER SAME
SUBSPACE
27
STOCHASTIC PERTURBATIONS - PLANS

• AREA OF ACTIVE RESEARCH
• ECMWF operational (Buizza et al, 1999), A random
  numbe (sampled from a uniform distribution)
  multiplied to the parameterized tendency
• ECMWF research (Shutts and Palmer, 2004),
  Cellular Automaton Stochastic Backscatterused to
  determine the perterbation
• Simple Model Experiment (Peres-Munuzuri, 2003),
  multiplicative and additive stochastic forcing
• METHOD UNDER DEVELOPMENT (EMC, sponsored by OGP)
• Addition of flow-dependent perturbations to
  tendencies in course of integration
• DETAILS Add to each perturbed member
• Difference between single high low-res
  forecasts (after scaling and filtering)
• Perturbation based on the differences among the
  ensemble members at previous step
  in integration
• Use global or localized perturbation approach
• Random or guided selection of members (e.g., use
  difference between most similar
  members)
• TO BE TESTED

28
OUTLINE / SUMMARY

• CONFIGURATION
• MARCH 2004 IMPLEMENTATION
• Increased resolution
• More frequent updates
• PLANS
• Half horizontal and vertical resolution of hires
  GFS control?
• 20 members per cycle?
• COMPARISON WITH OTHER CENTERS
• ECMWF, Met. Service Canada ECMWF best overall
  performance
• Ensemble performance strongly influenced by
  quality of data assimilation, model
• When perturbations evaluated directly, NCEP
  is competitive
• INITIAL PERTURBATIONS
• Testing Ensemble Transform Kalman Filter Mixed
  results
• Test regional breeding with input from GSI
  analysis scheme To be tested
• MODEL PERTURBATIONS
• Multiple model versions (RAS SAS) Not
  encouraging
• Stochastic perturbations (represent uncertainty
  due to sub-grid scale motions) -
• 
  To be
  tested
• UTILITY OF ENSEMBLE FORECASTS

29
BACKGROUND
30
The test period and the verification measures

• The test period is May-June-July 2002 (MJJ02).
• Scores for Z500 forecasts over NH (2080N) are
  shown.
• All forecasts data are defined on a regular
  2.5-degree latitude-longitude grid.
• Each ensemble is verified against its own
  analysis.
• For a fair comparison, only 10 perturbed members
  are used for each ensemble system (from 00UTC for
  MSC and NCEP and from 12UTC for ECMWF).
• Probability forecasts accuracy has been
  measured using the Brier skill score (BSS), the
  area under the relative operating characteristic
  curve (ROC) and the ranked probability skill
  score (RPSS). Probabilistic forecasts are average
  scores computed considering 10 climatologically
  equally likely events (see talk by Z. Toth for a
  definition).

31
Local spread and rescaling factor of energy perts
(ave over 0115-0215/03)