MOS Performance

1
MOS Performance

• MOS significantly improves on the skill of model
  output.
• National Weather Service verification statistics
  have shown a narrowing gap between human and MOS
  forecasts.

2
Cool Season Mi. Temp 12 UTC Cycle
Average Over 80 US stations
3
Prob. Of Precip. Cool Season(0000/1200 UTC
Cycles Combined)
4
MOS Won the Department Forecast Contest in
2003 For the First Time!
5
(No Transcript)
6
(No Transcript)
7
Average or Composite MOS

• There has been some evidence that an average or
  consensus MOS is even more skillful than
  individual MOS output.
• Vislocky and Fritsch (1997), using 1990-1992
  data, found that an average of two or more MOSs
  (CMOS) outperformed individual MOSs and many
  human forecasters in a forecasting competition.

8
Some Questions

• How does the current MOS performancedriven by
  far superior models compare with NWS forecasters
  around the country.
• How skillful is a composite MOS, particularly if
  one weights the members by past performance?
• How does relative human/MOS performance vary by
  forecast projection, region, large one-day
  variation, or when conditions vary greatly from
  climatology?
• Considering the results, what should be the role
  of human forecasters?

9
This Study

• August 1 2003 August 1 2004 (12 months).
• 29 stations, all at major NWS Weather Forecast
  Office (WFO) sites.
• Evaluated MOS predictions of maximum and minimum
  temperature, and probability of precipitation
  (POP).

10
National Weather Service locations used in the
study.
11
Forecasts Evaluated

• NWS Forecast by real, live humans
• EMOS Eta MOS
• NMOS NGM MOS
• GMOS GFS MOS
• CMOS Average of the above three MOSs
• WMOS Weighted MOS, each member is weighted by
  its performance during a previous training period
  (ranging from 10-30 days, depending on each
  station).
• CMOS-GE A simple average of the two best MOS
  forecasts GMOS and EMOS

12
The Approach Give the NWS the Advantage!

• 08-10Z-issued forecast from NWS matched against
  previous 00Z forecast from models/MOS.
• NWS has 00Z model data available, and has added
  advantage of watching conditions develop since
  00Z.
• Models of course cant look at NWS, but NWS looks
  at models.
• NWS Forecasts going out 48 (model out 60) hours,
  so in the analysis there are
• Two maximum temperatures (MAX-T),
• Two minimum temperatures (MIN-T), and
• Four 12-hr POP forecasts.

13
Temperature Comparisons
14
Temperature
MAE (?F) for the seven forecast types for all
stations, all time periods, 1 August 2003 1
August 2004.
15
Large one-day temp changes
MAE for each forecast type during periods of
large temperature change (10?F over 24-hr), 1
August 2003 1 August 2004. Includes data for
all stations.
16
MAE for each forecast type during periods of
large departure (20?F) from daily climatological
values, 1 August 2003 1 August 2004.
17
Number of days each forecast is the most
accurate, all stations. In (a), tie situations
are counted only when the most accurate
temperatures are exactly equivalent. In (b), tie
situations are cases when the most accurate
temperatures are within 2?F of each other.
Looser Tie Definition
18
Number of days each forecast is the least
accurate, all stations. In (a), tie situations
are counted only when the least accurate
temperatures are exactly equivalent. In (b), tie
situations are cases when the least accurate
temperatures are within 2?F of each other.
Looser Tie Definition
19
Highly correlated time series
Time series of MAE of MAX-T for period one for
all stations, 1 August 2003 1 August 2004. The
mean temperature over all stations is shown with
a dotted line. 3-day smoothing is performed on
the data.
20
Cold spell
Time series of bias in MAX-T for period one for
all stations, 1 August 2003 1 August 2004.
Mean temperature over all stations is shown with
a dotted line. 3-day smoothing is performed on
the data.
21
MAE for all stations, 1 August 2003 1 August
2004, sorted by geographic region.
MOS Seems to have the most problems at high
elevation stations.
22
Bias for all stations, 1 August 2003 1 August
2004, sorted by geographic region.
23
Precipitation Comparisons
24
Brier Scores for Precipitation for all stations
for the entire study period.
25
Brier Score for all stations, 1 August 2003 1
August 2004. 3-day smoothing is performed on the
data.
26
Precipitation
Brier Score for all stations, 1 August 2003 1
August 2004, sorted by geographic region.
27
Reliability diagrams for period 1 (a), period 2
(b), period 3 (c) and period 4 (d).
28
NWS Main MOS site http//www.nws.noaa.gov/mdl/sy
nop/products.shtml
29
Ensemble MOS
Ensemble MOS forecasts are based on ensemble
runs of the GFS model included in the 0000 UTC
ensemble suite each day. These runs include
the.operational GFS, a control version of the GFS
(run at lower resolution), and 10 pairs (positive
and negative) of bred perturbation runs (20
members). The operational GFS MOS prediction
equations are applied to the output from each of
the ensemble runs to produce separate bulletins
in the same format as the operational message.
30
(No Transcript)
31
Gridded MOS

• The NWS needs MOS on a grid for many reasons,
  including for use in their IFPS
  analysis/forecasting system.
• The problem is that MOS is only available at
  station locations.
• A recent project is to create Gridded MOS.
• Takes MOS at individual stations and spreads it
  out based on proximity and height differences.
  Also does a topogaphic correction dependent on
  reasonable lapse rate.

32
Gridded MOS SITE
http//www.nws.noaa.gov/mdl/synop/gmos.html
33
(No Transcript)
34
(No Transcript)
35
Current Operational Gridded MOS
36
(No Transcript)
37
Grid-Based Model Bias Removal
We need to get rid of them
Model biases are a reality
38
Grid-Based Bias Removal

• In the past, the NWS has attempted to remove
  these biases only at observation locations (MOS,
  Perfect Prog)--exceptiongridded mos recently
• Removal of systemic model bias on forecast grids
  is needed. Why?
• All models have significant systematic biases
• NWS and others want to distribute graphical
  forecasts on a grid (IFPS)
• People and applications need forecasts
  everywherenot only at ASOS sites
• Important post-processing step for ensembles

39
A Potential Solution Obs-BasedGrid Based Bias
Removal

• Based on observations, not analyses.
• Base the bias removal on observation-site land
  use category, elevation, and proximity. Land use
  and elevation are the key parameters the control
  physical biases.

40
Spatial differences in bias
41
(No Transcript)
42
The Method

• Calculate model biases at observation locations
  by interpolating model forecasts to observation
  sites.
• Identify a land use, elevation, and lat-lon for
  each observation site.
• Calculate biases at these stations hourly. Thus,
  one has a data base of biases.
• For every forecast hour At every forecast grid
  point search for nearby stations of similar land
  use and elevation and for which the previous
  forecast value is close to that being forecast at
  the grid point in question..
• E.g., if the forecast temperature was 60, only
  use biases for nearby stations of similar
  land-use/elevation associated with forecasts of
  55-65.
• Collect a sufficient number of these (using
  closest ones first) to average out local effects
  (roughly a half dozen). Average the biases for
  these sites and apply the bias correction to the
  forecast.

43
Raw 12-h Forecast
Bias-Corrected Forecast
44
Sal Lake City
45
Bozeman
46
(No Transcript)
47
The End
http//www.atmos.washington.edu/jbaars/mos_vs_nws
.html