STATISTICAL POSTPROCESSING OF ENSEMBLE FORECASTS

1
STATISTICAL POSTPROCESSING OF ENSEMBLE FORECASTS

• Zoltan Toth,
• Yuejian Zhu, Dingchen Hou, and Richard Wobus (4)
• Ackn. S. Lord, H.-L. Pan, S. Saha, J. Schaake
  (1), P. Dallavalle (2),
• D. Unger(3) , W. Ebisuzaki 3)
• (1) Office of Hydrologic Developments, NWS
• (2) Meteorological Development Laboratory, NWS
• (3) Climate Prediction Center, NCEP/NWS

2
OUTLINE

• DEFINITION
• SOURCES OF STATISTICAL INCONSISTENCY
• HOW TO IMPROVE STATISTICAL CONSISTENCY
• CONDITIONS FOR USE OF STAT. METHODS TO IMPR.
  CONSISTENCY
• MEASURES OF STATISTICAL CONSISTENCY
• CHARACTERISTICS OF ENSEMBLE POSTPROCESSING
• ENSEMBLE POSTPROCESSING METHODS

3
DEFINITION

• STATISTICAL CONSISTENCY OF FORECASTS WITH
  OBSERVATIONS
• Select a particular forecast
• Consider that the same forecast is issued many
  times
• Construct distribution of verifying observations
  (analysis), given selected fcst
• If selected fcst is EQUIVALENT to distribution of
  obs. conditioned on fcst gt
• Forecast is statistically consistent
• If selected forecast is NOT EQUIVALENT to
  distribution of obs gt
• Statistically post-process forecast to improve
  consistency

EXAMPLES CONTROL FCST ENSEMBLE
ENSEMBLE
4
SOURCES OF STATISTICAL INCONSISTENCY

• TOO FEW FORECAST MEMBERS
• Single forecast inconsistent by definition,
  unless perfect
• MOS fcst hedged toward climatology as fcst skill
  is lost
• Small ensemble sampling error due to limited
  ensemble size
• (Houtekamer 1994?)
• MODEL ERROR (BIAS)
• Deficiencies due to various problems in NWP
  models
• Effect is exacerbated with increasing lead time
• SYSTEMATIC ERRORS (BIAS) IN ANALYSIS
• Induced by observations
• Effect dies out with increasing lead time
• Model related
• Bias manifests itself even in initial conditions
• ENSEMBLE FORMATION (INPROPER SPREAD)
• Not appropriate initial spread
• Lack of representation of model related
  uncertainty in ensemble
• I. E., use of simplified model that is not able
  to account for model related uncertainty

5
HOW TO IMPROVE STATISTICAL CONSISTENCY?

• MITIGATE SOURCES OF INCONSISTENCY
• TOO FEW MEMBERS
• Run large ensemble
• MODEL ERRORS
• Make models more realistic
• INSUFFICIENT ENSEMBLE SPREAD
• Enhance models so they can represent model
  related forecast uncertainty
• OTHERWISE gt
• STATISTICALLY ADJUST FCST TO REDUCE INCONSISTENCY
• Unpreferred way of doing it
• What we learn can feed back into development to
  mitigate problem at sources
• Can have LARGE impact on (inexperienced) users

6
WHAT WE NEED FOR POSTPROCESSING TO WORK?

• LARGE SET OF FCST OBS PAIRS
• Consistency defined over large sample need same
  for post-processing
• Larger the sample, more detailed corrections can
  be made
• BOTH FCST AND REAL SYSTEMS MUST BE STATIONARY IN
  TIME
• Otherwise can make things worse
• Subjective forecasts difficult to calibrate

HOW WE MEASURE STATISTICAL INCONSISTENCY?

• MEASURES OF STATIST. RELIABILITY
• Time mean error
• Analysis rank histogram (Talagrand diagram)
• Reliability component of Brier etc scores
• Reliability diagram

7
CHARACTERISTICS OF ENSEMBLE POSTPROCESSING

• TRUTH
• Observations (arbitrary location)
• Analysis field (usually gridded field)
• Feedback to ensemble system development
• Further post-processing (Downscaling) is needed
  as separate step
• COVARIANCES
• Point-wise correction (spatiotemporal covariances
  ignored)
• Spatial/temporal covariances considered
• Added level of complexity (need more data?)
• Downscaling can be integral part of it
• REALIZATIONS
• Individual fcst trajectories corrected
• Spatio-temporal cross-variable correlations may
  be useful
• Possibly realistic ensemble solutions for users
  (hydro, energy power, etc)
• Only probability distributions corrected, no
  individual realizations
• If users need trajectories, how to create them?

8
ENSEMBLE POSTPROCESSING METHODS

• GENERAL PROCEDURE
• COMPARE CERTAIN PROPERTIES OF OBS. FCSTS
• Various choices related to different measures of
  inconsistency
• Need obs fcst data pairs
• MAKE ADJUSTMENTS
• Individual ensemble forecasts
• Ensemble-based pdfs
• SYSTEMATIC ERROR DEPENDS ON
• Model / Initial perturbations
• Lead time
• Geographic location
• Season / Flow regime
• SEEK BALANCE BETWEEN
• NEED FOR LARGE SAMPLE FOR STABLE STATISTICS
• Pool together as much data as possible
• DESIRE FOR DETAILED CORRECTIONS
• Subdivide sample to make corrections more
  specific

9
EXISTING/PROPOSED APPROACHES

• STATISTICAL MOMENTS
• Estimate time mean error adjust each fcst
  accordingly (1st moment correction)
• Estimate time mean bias in spread (2nd moment
  correction)
• Adjust each member separately for both 1st 2nd
  moments
• CUMULATIVE DISTRIBUTIONS
• Compare cumulative distributions (not only 1st
  moment)
• Need more data
• DRESSING
• Fit predetermined pdf on each member
• ADJUST FCST PROBABILITY VALUES
• Based on Reliability diagram, fcst prob obs.
  frequency
• Does not correct model bias
• Sub-optimal , does not improve statistical
  resolution
• OTHERS
• Not necessarily designed for use with ensembles

10
BACKGROUND
11
BIAS CORRECTION / DOWNSCALING
TWO GOALS Adjust sample ensemble time
trajectories, covariances, only then Construct
bias-corrected pdf for individual
variables APPROACH (A) Bias corrected anomalies
on model grid, then downscaling 1) ESTIMATE
BIAS Compare time mean fields
FIRST MOMENT B DIFFERENCE BETWEEN Ensemble
mean forecast and Verifying analysis
SECOND MOMENT R RATIO BETWEEN Ensemble mean
error and Ensemble spread

• STATISTICAL SAMPLING (Increase sample size)
• Use data from surrounding grid-points (with
  Gaussian weighting)
• Use climate means if available and forecast
  system is stable
• Use most recent past data with decaying
  averaging otherwise
• Ability to quickly learn bias of new NWP
  systems before upgrade
• Adjust temporal/spatial sampling domain to
  optimize performance
• REMOVE BIAS Compare time mean fields
• 1st moment Ensemble mean - B 2nd moment
  Ensemble spread R

12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
BIAS CORRECTION / DOWNSCALING, APPROACH (A)

• FORM FORECAST ANOMALIES

FIRST MOMENT Ai DIFFERENCE BETWEEN Each
ensemble forecast and Reanalysis climate mean
SECOND MOMENT SAi RATIO BETWEEN Anomaly
and Reanalysis Standard Deviation

• BIAS-CORRECTED STANDARDIZED ANOMALY FORECAST ON
  MODEL GRID
• Temporal/spatial resolution can degrade with
  lead time / loss of predictability
• 4) COMBINE ENSEMBLES FROM DIFFERENT CENTERS
• Follow steps 1-3 for each ensemble separately
• Determine weights for each ensemble based on
  error statistics (D. Unger)
• Combine anomalous ensemble forecasts (with
  weights)
• DOWNSCALE
• Add coarse resolution forecast anomaly to NDFD
  (or other local) climate distribution

FIRST MOMENT Forecast anomaly Plus Local climate
mean
SECOND MOMENT Multiply Standardized
Anomaly and Local climate standard deviation
BIAS CORRECTED LOCAL FORECAST Only climatology
is stored at high resolution, anomaly forecast is
on coarse grid
17
BIAS CORRECTION / DOWNSCALING APPROACH (B)
BIAS CORRECTION DIRECTLY ON NDFD GRID (or local
sites, not on model grid) METHODS SUGGESTED BY
MDL NEURAL NETWORK APPLICATIONS Input Raw
ensemble forecasts and lat, lon, elevation,
climatology, etc Output Bias corrected ensemble
forecasts Penalty BSS (or RPSS), assuring whole
distribution is corrected ENSEMBLE TRANSFORM
KALMAN FILTER (ETKF) APPLICATIONS INCREASE
ENSEMBLE SIZE Transform lagged (older) ensembles
to possess same mean spread as
current ensemble How old ensembles are still
useful? MANUAL FORECAST MODIFICATION Transform
operational ensemble to describe limited manually
prescribed information Propagate signal in time,
space, and across variables using transformed
ensemble
18
CHALLENGE PROVIDE OBJECTIVE GUIDANCE FOR NDFD

• NATIONAL DIGITAL FORECAST DATABASE
• Seamless suite of forecasts across different time
  ranges (hours seasons)
• 15 variables
• NEED PROBABILISTIC INFORMATION ADD 2nd MOMENT
• Requirements
• 2.5x2.5 spatial grid
• High temporal resolution (hours)
• TEMPORAL/SPATIAL SCALES OF PREDICTABLE SIGNAL
  INCREASES WITH LEAD TIME
• ADD FORECAST SIGNAL ON REDUCED SPACE/TIME GRID TO
  
• CLIMATE (INCLUDING DAILY CYCLE) ON FULL GRID
• OBJECTIVE GUIDANCE FOR NDFD
• Based on best available NWP products
• 1st moment - Ensemble mean?
• 2nd moment - Ensemble spread
• IMPERFECT MODEL/ENSEMBLE BIAS CORRECTION /
  POSTPROCESSING
• MANUAL MODIFICATION OF OBJECTIVE FIRST GUESS
• Focus on 1st moment initially
• PROPAGATE INFO ACROSS TIME/SPACE/VARIABLES
  ENSEMBLE-BASED TOOLS?
• 2nd moment can be handled similarly later
  NEED TRAINING

19
OBJECTIVE GUIDANCE FOR NDFD
BIAS-CORRECT AND DOWNSCALE NWP ENSEMBLE GUIDANCE
COMPARE MODEL GRID FORECAST HIRES OBS
DATA A) Based on NWP analysis and hires
observational climatology ON MODEL
GRID Bias-correct forecast wrt
reanalysis Express forecast as anomaly from
reanalysis ON NDFD GRID (or any local
data) Add forecast anomaly to NDFD (or
observed) climatology Problem Must construct
NDFD climatology first Advantage Works with
predictable signal - simple, more flexible
approach? B) Based on hires observational
data ON NDFD GRID (or any local data
points) Bias-correct forecast wrt observations
or NDFD analysis Problem How to interpolate
between obs data points? Easier done for clim.
mean? Advantage Can potentially correct flow
dependent systematic error
20
(No Transcript)