University of Innsbruck, 24 Nov 2005

1
What is a good weather forecast?
2
In the 1930s the first private weather forecast
firm was established in California by Irving Krick
The official US Weather Bureau pointed out that
these privately made forecasts were very bad
3
The Weather Bureau forecast for the public Sunny
and mostly dry
The Irving Krick forecast for some of his
clients Probably rain
4
The Weather Bureau forecast for the public
Probably rain
The Irving Krick forecast to some of his clients
Probably dry
5
But Kricks private weather firm earned millions
of
6
1. The accuracy of a weather forecast
7
The verification yielded RMSE5.0
24 hour 2 m temperature forecast for Kiruna in
Lapland winter 2001-2002
8
The forecast system is improved
but the RMSE is only reduced from 5.0 to 4.6
9
Observations and forecasts swapped
obs
model
10
The RMSE is reduced to reduced from 5.0 to 2.9
11
Range 20 K
Too warm forecasts
12
Range 30 K
Non-biased forecasts
13
The complete formula for RMSE
14
a
Ef-a
f
Root Mean Square Error
a-c
f-c
ß
90º
c
ACCcosß
15
The RMS error saturation level

• When the forecast is lacking skill the f-c
  vector is perpendicular to the verifying vector
  a-c
• Cosine of the angle 90 is zero which is also
  the value of the ACC (Anomaly Correlation
  Coefficient)
• It is easy to see that the maximum RMSE
  equals the variability times ?2

16
Error saturation level
The error growth for a very good NWP model
The error level for a climate forecast
The pre-NWP forecaster
The error growth for a good customer orientated
forecast system
17
Hedging
The forecast variability is less than the observed
a
90º
f
Aa
Af
This will lower the RMSE
18
The mathematics of Hedging
a
Unchanged variability the RMSE increases to
sqrt(2) limit
Decreasing variability the RMSE increases to
1.0 limit
f
90º
Aa
Af
19
2. Skill of weather forecasts
20
Trivial correlations
Observed temperature
D10
Sep, Oct and Nov
Forecast temperature
21
Not so trivial correlations
Observed temperature anomaly
D10
Sep, Oct and Nov
Forecast temperature anomaly
22
The SMHI monthly forecasts scored 65-70 anomaly
correlation until.
100
70
what went wrong?
Introduction of more ECMWF data
35
0
June 2003
Aug 2001
April 2005
23
Observed anom.
Observed anom.
Forecast anom.
Forecast anom.
Both sets of forecasts correlate badly, but
only the forecast to the right might be
considered bad
24
Two short periods can both have correlation
0 If they are combined the correlation
increases considerably
Observed anom.
Observed anom.
Observed anom.
Period 2
Period 1
Period 12
Fcst anom.
Fcst anom.
Fcst anom.
High ACC
Low ACC
Low ACC
25
3. The usefulness of weather forecasts
back to Irving Krick and the 1930s California
26
The first type of customer, for whom the rain was
over-forecast, came from Hollywood movie
producers who did not want to take out all their
expensive equipment if there was any risk of rain
The second type of customer, for whom the rain
was under-forecast, came from the Californian
water authorities, who did not want to lower the
water levels in the water reservoirs unless rain
was pretty sure to fall
27
The Expected Monetary Value (EMV) cost of
protection losses, if no protection
EMV number of occasions when rain is forecast
? cost number of occasions with unforecast
rain ? loss
28
Always protect EMV total number of days ? cost
N ? C
Never protect EMV number of rainy days ? loss
R ? L
Breaking point when R ? LN ? C then C/LR/N
Action should be taken when the risk exceeds the
users personal cost-loss ratio
But the risk does not have to be climatological,
it can be predicted
29
Assume the adverse weather has a 30
climatological probability How should different
actors, with different cost-loss ratios react if
they stand to loose 1000?
Obs rain dry
30
Costs when no weather forecasts are available
Never protect
Always protect
Perfect forecasts
Water authorities
Hollywood
31
The official Weather Bureau (A) had in principle
two rival forecasts one (B) serving Hollywood,
the other (C) the water authorities
B
Obs rain dry
Obs rain dry
Obs rain dry
A
C
2
1
0.5
3
2
0
rain Fc dry
rain Fc dry
rain Fc dry
1
6
5
0
2.5
7
Very good public forecasts
Over forecasts (Hollywood)
Under forecasts (Water authorities)
32
Costs when weather forecasts are available
Never protects
C
A
Always protects
B
33
4. Confidence in the weather forecast
34
It is not enough that the weather forecast is
good.
If it is not believed, it is in effect a useless
and bad forecast
35
Great problem varying day-to-day skill
36
Last winters 500 hPa forecast quality at D5 over
Europe
CMC
Error
USA
ECMWF 12 UTC
ECMWF 00 UTC
January-February 2005
Correlation
ECMWF 00 UTC
USA
ECMWF 12 UTC
CMC
UKMO
37
How we would like forecasts to evolve
38
Instead it is often like this!
39
Would we really like it to be like this?
40
This isnt that bad after all!
41
The Blame Game or The Passing of The Puck
The atmosphere is chaotic
Atmosphere
Errorneous observations misled the NWP
Scientists
Computer models
The NWP misled me
Forecaster
The forecaster misled me
Customer/Public
42
5. Ultimate solution -probabilities
43
The official Weather Bureau (A) had two ways to
fight the private firms forecasts to Hollywood
(B) and to the water authorities (C)
1. Slow and expensive improve the deterministic
weather forecasting by scientific research
2. Quick and cheap probabilities
44
The Weather Bureaus forecast matrix
Obs rain dry
A
rain Fc dry
45
The Weather Bureau could become less categorical
observation
rain dry
rain
Does not know (50)
forecast
dry
46
The Weather Bureau and the low cost/loss customers
observation
rain dry
rain
Does not know (50)
forecast
rain
dry
dry
Cost0 EMV0 Cost100 EMV500
47
Costs when weather forecasts are available
48
The Weather Bureau and the high cost/loss
customers
observation
rain dry
rain
rain
Does not know (50)
forecast
dry
Cost0 EMV100 Cost100 EMV300
dry
49
Costs when weather forecasts are available
50
The Weather Bureau Could have got tougher
observation
100 80 60 40 20 00
rain dry
for c/L ratios gt80
for c/L ratios gt60
forecast
for c/L ratios gt40
51
Costs when weather forecasts are available
52
Costs when weather forecasts are available
Continuous probabilities
53
The Weather Bureau could have used probabilities!
B
C
A
54
Probability forecast are never wrong except
when we have stated 0 or 100
So how do we know if the forecasts are good or
bad?
We can never say for an individual forecast, just
for the forecast system (weather service) as a
whole
55
The reliability diagram
100
Event occurred 15 times 50
Outcome
Event occurred 6 times 20
Event occurred 3 times 10
100
0
20 was forecast 30 times
Probability statement
56
Ideal reliability and good resolution
Size of balls proportional to number of fcsts
Good resolution forecasts draw towards 0 and
100
57
Good reliability, but poor resolution
58
Poor reliability, but good resolution
59

• Summary
• Use more than one verification method
• Computer output normally needs post-processing
• The public is more interested in posterior
  verification
• Probabilities is the ultimate solution
• Decision theory will teach us more about the
  problem