COMPLETING THE FORECAST: ASSESSING AND COMMUNICATING FORECAST UNCERTAINTY

1
COMPLETING THE FORECASTASSESSING AND
COMMUNICATING FORECAST UNCERTAINTY

• Zoltan Toth
• Environmental Modeling Center
• NOAA/NWS/NCEP
• Acknowledgements
• NRC Report Louis Uccellini, Steve Lord
  Ensemble Team
• http//wwwt.emc.ncep.noaa.gov/gmb/ens/index.html

2
OUTLINE

• REVIEW OF 3RD ENSEMLBE USER WORKSHOP
• MAJOR RECOMMENDATIONS REGARDING NRC REPORT
• WHAT, WHY, AND HOW TO CHANGE?
• SPECIFIC RECOMMENDATIONS

3
3rd ENSEMBLE USER WORKSHOP

• Logistics
• Oct. 30 Nov. 1 2006, Laurel, MD
• Close to 100 participants
• NWS Regions (12), Headquarters (15), NCEP (37)
• OAR (6), other government (4), private (4),
  academic (8) sectors international (8)
• For further info, see http//wwwt.emc.ncep.noaa.g
  ov/gmb/ens/UserWkshop_Oct2006.html
• Topics
• Assessing propagating uncertainty throughout
  entire forecast process
• From observations to users
• Working group discussions
• Ensemble configuration
• Statistical post-processing
• Data depository / Interrogation / Product
  generation/dissemination / Verification
• User support / Outreach / Training
• Outcome
• Enthusiastic discussions
• Convergence on number of topics
• Open questions needing further research
  identified
• Strong support for sustained effort/engagement,
  annual meetings, etc

4
MAJOR RECOMMENDATIONS - 1

• FOLLOW NRC REPORT RECOMMENDATIONS
• All major recommendations embraced
• Develop roadmap for assessing communicating
  forecast uncertainty
• Based on science, technology, workforce
  considerations
• Consistent with NOAAs mission, NWS plans,
  embraced by Enterprise
• Define end goal, 5-10 years horizon
• Adopt ensemble-based forecast process
• Maximize forecast skill
• Unify scientific, software, and technological
  infrastructure across NOAA
• Weather, water, climate applications
• Form high level planning and oversight team
  March 2007
• Each NWS office to delegate one representative
• Team reports to Corporate Board
• Name programmatic and technical leads
• Develop roadmap Sept. 2007

5
MAJOR RECOMMENDATIONS - 2

• Develop roadmap for assessing communicating
  forecast uncertainty
• Revise operational requirements to make them
  probabilistic
• Make probabilistic format the primary requirement
• For each forecast application
• Replace single value/categorical format with
  probabilistic format as primary requirement
• Revise/supplement corresponding performance
  measures (GPRA)
• Essential for orderly transition from traditional
  to new forecast process
• NWS is requirement-driven organization
• Without clear new requirements, process is doomed
• Phased implementation schedule in consultation
  with responsible offices
• Allows orderly transition to new requirements
• New requirements prepared by Planning oversight
  team (or its designate)
• Assisted by responsible NWS office
• Presented to Corporate Board for approval By
  March 2008

6
MAJOR RECOMMENDATIONS - 3

• Develop roadmap for assessing communicating
  forecast uncertainty (.5 yr)
• Revise operational requirements to make them
  probabilistic (1 year)
• Design, develop, and gradually implement new
  forecast process
• Focus on missing scientific, technological, and
  human components
• Identify self-contained components with
• Clear requirements and interfaces between
  components
• Define basic capabilities achievable in 2-3 years
• Limited but consistent with and leading to end
  goal
• Full capabilities in 5-10 years
• Interface with NOAA THORPEX program
• Research/development to improve skill utility
  of probabilistic forecasts
• Leverage related major NOAA, national, and
  international efforts
• Integrate with NWS, NOAA, national activities
• New NOAA CONOPS process
• NOAA-wide regional service plans
• NUOPC planning/development
• Provide long-term funding support through PPBES

7
PROPOSED CHANGE

• Major paradigm shift
• Incorporate assessment and communication of
  uncertainty in forecast process
• Is it a major change in course of Weather Ship?
• Ie, abandon course of ever improving single
  forecast scenario (expected value)?
• No Expand, not abandon
• Keep improving fidelity of forecasts, PLUS
• Add new dimension
• Capture other possible scenarios ensemble
  forecasting
• Use a flotilla, instead of one ship, in exploring
  nature
• Existing activities are subset of expanded
  forecast process
• Single value forecast is expected value of full
  probability distribution
• Can keep serving forecasts in old format to users
  who prefer that

8
Single forecast (driven by GFS winds) example for
drifting virtual ice floe
7 September 2006
Initial position
Bob Grumbine, EMC
9
Ensemble forecast for drifting ice floe for same
case
Initial position
Bob Grumbine, EMC
10
Most likely forecast for drifting ice floe for
same case
Initial position
Bob Grumbine, EMC
11
WHY CHANGE IS NEEDED?

• Why users (should) care about forecast
  uncertainty?
• They admittedly want minimal or no uncertainty in
  forecasts
• Distinction between no uncertainty in the
  forecast, vs. not talking about it
• Forecast uncertainty cannot be arbitrarily
  reduced
• Despite major ongoing continuing efforts, they
  persist forever
• Chaotic nature of atmosphere - land surface
  ocean coupled system initial/model errors
• Level of uncertainty is determined by nature and
  level of sophistication in forecast system
• Forecast uncertainty can be ignored though
• Negative consequence on informed users
• Not able to prepare for all possible outcomes
• Assumes a certain scenario and remains vulnerable
  to others
• Possibly serious loss in social/economic value of
  forecast information
• Why forecasters (should) care about forecast
  uncertainty?
• Imperfect forecasts are consistent w.
  observations (reliable) only if in prob format
• If in other format, must be brought into
  probabilistic format through
• Verification / bias correction

12
ADVANTAGES OF PROBABILISTIC FORMAT

• More rationalized and enriched forecaster - user
  interactions
• Old paradigm
• Convoluted forecaster-user decision process
• User expects forecaster to make decision for them
  in presence of uncertainty
• Will it rain? 80 But tell me, will it
  rain?
• New paradigm
• Forecaster and user decision processes enhanced
  and better linked
• Allows forecasters to capture all knowledge about
  future conditions
• Provision of information related to multiple
  decision levels in probabilistic format critical
• Provider helps interpret probabilistic info and
  modify user decision process if needed
• Option to continue providing single value or
  other limited info until user ready
• Allows users to decide about most beneficial
  course of action given all possibilities
• Proper use of probability or other uncertainty
  information needed - Training
• User requests critical weather forecast info
  depending on their sensitivity

13
TRADITIONAL FORECAST PROCESS

• Focus on single forecast scenario
• Reducing uncertainty in single forecast is main
  emphasis
• Loss of accuracy in forecast estimate of expected
  value of distribution
• Mean of ensemble cloud provides better estimate
• Ignores or simplifies forecast uncertainty
• Uncertainty assessed as statistically averaged
  error in single fcst (second thought)
• Ensemble cloud provides better estimate of case
  dependent variations in uncertainty
• Use of single value / categorical forecast format
• Difficulty in formulating/communicating plausible
  alternate scenarios
• Ensemble member forecasts can directly feed into
  Decision Support Systems
• One-way flow of information from observations to
  users
• Not adaptable to case dependent user requirements
• Ensemble can propagate back user requirements to
  adaptive
• Observing, assimilation, modeling/ensemble,
  post-processing and application components
• Applications in planning and execution of new
  CONOPS in high impact events

14
PROPAGATING FORECAST UNCERTAINTY
z
Distribution
Single value
Ensemble Forecasting Central role bringing the
pieces together
15
HOW CAN IT BE DONE? NEW PARADIGM

• Adopt ensemble approach across all environmental
  prediction activities
• Expand forecasting with new dimension of
  uncertainty
• Multiple scenarios (in place of single scenario)
• Provides best forecast estimate for both expected
  value (as before) and uncertainty (new)
• Unified scientific, technological, human approach
• Sharing resources across NWS NOAA
• Ensemble is centerpiece both symbolically and
  figuratively in forecast process
• Ensembles act as a glue two-way information
  channel
• Observing system, data assimilation, numerical
  modeling
• ENSEMBLES
• Statistical post-processing, product generation,
  decision making
• Design, develop, implement missing components
  of new forecast process
• Gradual, measured steps
• Basic capability - Short-term, 2-3 yrs, leading
  to
• Full implementation - Long-term, 5-10 yrs

16
ENSEMBLES AND THE RESEARCH COMMUNITY LINKED
THROUGH THORPEX MAJOR INTERNATIONAL RESEARCH
PROGRAM GOAL Accelerate improvements of high
impact weather forecasts
INTEGRATED DATA ASSIMILATION FORECASTING
GLOBAL OPERATIONAL
TEST CENTER
GLOBAL INTERACTIVE FORECAST SYSTEM (GIFS)
Days 15-60
NWS OPERATIONS
CLIMATE FORECASTING / CTB
GLOBAL OPERATIONAL
SOCIOECON.
SYSTEM
TEST CENTER
MODEL ERRORS HIGH IMPACT MODELING
17
SPECIFIC RECOMMENDATIONS - 1

• Continue development of expanded forecast process
• Focus on adaptive methods applicable for high
  impact events
• Collection use of observations (targeted
  observations)
• Data assimilation (case dependent background
  error estimation)
• Numerical modeling (adaptive resolution high
  impact modeling)
• Ensemble forecasting (case dependent variations
  in membership composition)
• Decision support systems (flexible user actions
  depending on forecast probabilities)
• Bias correction downscaling methods for
  ensembles
• Estimate/correct lead-time dependent bias in
  ensemble forecasts (on model grid)
• Generate fine resolution (NDFD grid)
  uncertainty/ensemble data
• Establish connection between (bias corrected)
  coarse model vs. fine user grids
• Use reanalysis hind-casts with operational
  systems as needed
• Define summary ensemble information to be used to
  (ST)
• Collapse vast amount of ensemble data for
  inclusion in expanded NDFD/NDGD
• E.g., 10, 50, and 90 percentile of forecast
  distribution (in place of single value)
• Manually inspect/modify summary ensemble
  information

ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
18
SPECIFIC RECOMMENDATIONS - 2

• Contribute to establishment of NOAA-wide
  environmental data depository
• Expand NDFD/NDGD database to include forecast
  uncertainty (ST)
• Develop capability to hold all ensemble
  trajectories (LT)
• All members, variables, lead times
• Develop ensemble interrogation, modification,
  product generation tools to
• Derive summary information from ensemble (ST)
• Manually modify summary ensemble info (ST)
• Derive additional statistics from summary info
  (product generation, ST)
• Automatically modify ensemble trajectories based
  on modified summary info (LT)
• Derive any info from full ensemble data (product
  generation, LT)
• Develop telecommunication facilities to access
  data
• Summary info limited derived products (ST)
• All ensemble forecasts derived products (LT)

ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
19
SPECIFIC RECOMMENDATIONS - 3

• Develop unified NWS/NOAA probabilistic
  verification package to (ST)
• Assess statistical reliability and resolution for
• Computing official performance measures
• Evaluating value added along forecast chain
• Assessing value in newly developed vs.
  operational techniques
• Develop implement comprehensive training to
• Prepare all participants for their new roles in
  expanded forecast process, incl.
• Statistical background
• Ensemble methods
• Best forecast practices in assessing uncertainty
• Applications of probabilistic other uncertainty
  information
• Develop outreach program on use communication
  of uncertainty
• In partnership with entire Weather, Water
  Climate Enterprise
• Determine best ways of communicating uncertainty
• Compile sample of Decision Support Systems using
  uncertainty information
• Establish close partnership with public sector
  users (e.g., emergency, water management)
• User feedback on new activities

ST Short Term (2-3 yrs) LT Long Term (5-10
yrs)
20
NFUSE PRIORITIES?

• Issues to be discussed/worked out in this order
• NRC Recommendations
• Do we agree on major recommendations?
• If not, lets discuss any issues
• If so, can we accept those as the
  basis/guidelines for our work
• 3rd Ensemble User Workshop Recommendations
• Three major recommendations in response to NRC
  Report
• High level roadmap guiding our collaboration
  (Sept 2007)
• New operational requirements (March 2008)
• Design, develop, and gradually implement new
  forecast process
• Basic capabilities (2009-2010)
• Full capabilities (2012-2017)
• Nine specific recommendations on design,
  development, implementation
• Work areas identified, links to be worked out
• Continue development of expanded forecast process
• Bias correction downscaling methods for
  ensembles
• Define summary ensemble information to be used
  (ST)
• Contribute to establishment of NOAA-wide
  environmental data depository
• Develop ensemble interrogation, modification,
  product generation tools