Probabilistic Forecast

1
Probabilistic Forecast
Kiyoharu Takano Climate Prediction Division JMA
2
3-month(Dec.-Feb.) forecast issued by JMA at
25,Nov.
3
Contents (1) Predictability of Seasonal
Forecast (2) Example of Probability Forecast (3)
Quality of Probability Forecast
Reliability Resolution (4) Economical
Benefit of Forecast Deterministic forecast
Probabilistic forecast
4
(1) Predictability of Seasonal Forecast cf. Dr.
Sugis presentation
lt Predictability of 1st kind gt Originates
from Initial condition
Deterministic forecast fails beyond a
few weeks due to the growth of errors contained
in the initial states (Lorenz, 1963 1965). lt
Predictability of 2nd kind gt Lorenz (1975)
Originates from boundary condition Effective
for longer time scale Month to season
There remains internal variability which is not
controlled by boundary conditions
5
Growth of forecast error (Predictability of 1st
kind)

• 1. Error growth due to imperfectness of
  numerical prediction
• ?Improvement of numerical model
• 2. Growth of initial condition error
• ? Improvement of objective analysis
• However...
• (1) There remains finite (non zero) error in
  initial condition,
• (although it will be reduced as improvement of
  objective analysis).
• We cannot know true initial condition
• (2)Small initial error(difference) grows
  fast(exponentially) as time progresses and the
  magnitude of error becomes the same order as
  natural variability after a certain time. ?Chaos

Deterministic forecast becomes meaningless after
sufficiently long time
6
Chaos Lorenz,E. N. ,1963,
J.A.S. 20, 130- Equations for simplified role
type convection.
The Lorenz system
X Fourier component of stream function Y,ZFourie
r component of temperature Nonlinear terms XZ,XY
rStability parameter
7

• Findings of Lorenz
• The solution (x(t),y(t),z(t))
• behaves strangely
• The solution (x(t),y(t),z(t))
• is bound.
• Non-periodic
• Slight initial difference causes
• large difference. Chaos

Observation
Numerical prediction
Time steps (x0.01))
8
What can we do on the predictability
limit? Predictability limit varies depending on
flow pattern.
9

• What can we do on the predictability limit?
• Predictability limit varies depending on
  atmospheric flow pattern and initial condition
• Prediction of uncertainty of forecast are
  important

warm
cold
10
The Ensemble Prediction
a forecast prediction of uncertainty of forecast
Temperature anomalies at 850hPa over the Northern
Japan (7day running mean)
6/ 5
Initial5 June 2003
Initial24 July 2003
11
Longer than one month forecast
Seasonal forecast
One-month forecast
One-week forecast
Longer than one month forecast based on initial
condition is impossible at least generally!
12
Predictability of second kind In seasonal time
scale, forecast based on initial condition is
impossible at least generally The forecast is
based on the influences of boundary conditions
such as SSTs or soil wetness.
Prediction of second kind However ..
Regression OLR map with Nino3 SSTs (DJF)
13
Atmospheric variation is not fully controlled by
variation of boundary condition such as SSTs but
there are internal variation.. Examples of
internal variations are baroclinic instability ,
typhoon, Madden-Julian oscillation e.t.c. and
these can be predicted as initial value problem
in short time scale but they are unpredictable in
seasonal time scale. Then a variation X is
written as XXextXin Xext variation controlled
by boundary conditions(Signal) Xin
internal variation(Noise) (cf. Mr. Sugis
presentation)
14
30day mean 850hPa Temperature prediction at the
Nansei Islands
Noise
Signal
Individual numerical prediction
Average of individual predictions (ensemble mean)
Observation
15
Reduction of noise Since the internal variation
can be reduced by time mean but the signal is
not reduced, Time-mean is taken in seasonal
forecast.
Unpredictable reduced by time mean
predictable
This time mean is effective especially in
tropics. In addition, main SST signals such as
ENSO are in tropics. Seasonal forecast in tropics
is easier than in mid-latitude.
16
The noise reduction effect of time mean
5day mean
Individual numerical prediction
Average of individual predictions (ensemble mean)
Observation
30day mean
90day mean
17
Though time mean is effective to reduce
noise, the noise is not removed
completely. Therefore there remains uncertainty
from internal variation and again probabilistic
forecast is necessary.
Individual numerical prediction
Average of individual predictions (ensemble mean)
Observation
90day mean
18
(2) Example of probabilistic forecast (a)
One-month forecast
NJ
WJ
Temperature at 850hPa
EJ
Nansei
Surface Temperature Forecast
1 month 1st week 2nd week
3rd-4th week Forecast Period 11.22-12.21
11.22-11.28 11.29-12.5 12.6-12.19
category - 0 - 0
- 0 - 0 Northern Japan 20 40 40
20 40 40 20 50 30 20 40 40 Eastern Japan
20 30 50 20 30 50 20 40 40 20 40 40
Western Japan 10 40 50 10 40 50 20 30 50
20 40 40 Nansei Islands 10 40 50 10 30 60
20 30 50 20 40 40 ( category below normal,
0 near normal, above normal, Unit )
19
(2) 3 month forecast
Temperature Forecast Period 3
months 1st month 2nd month 3rd month
Nov.-Jan. Nov.
Dec. Jan. Category - 0
- 0 - 0 - 0
Northern Japan 30 50 20 30 50 20 30 50
20 20 50 30 Eastern Japan 20 50 30 20
50 30 20 50 30 20 40 40 Western Japan
20 40 40 20 50 30 20 40 40 20 40 40
Nansei Islands 20 30 50 20 40 40 20 30
50 20 30 50 ( category ? below normal,
0 near normal, above normal, Unit )
20
Precipitation Forecast (3 month forecast) Period
3 months 1st month
2nd month 3rd month
Nov.-Jan. Nov. Dec.
Jan. category -
0 - 0 - 0 - 0
Northern Japan Japan Sea side
20 50 30 20 50 30 20 50 30 30 40 30
Pacific side 30 50 20 30 50
20 30 50 20 30 40 30
Eastern Japan Japan Sea side 20 50
30 20 50 30 20 50 30 30 50 20 Pacific
side 20 50 30 20 50 30
20 50 30 20 40 40 Western Japan Japan Sea
side 20 50 30 20 50 30 20 50
30 30 40 30 Pacific side 20
50 30 20 50 30 20 50 30 20 40 40
Nansei Islands 20 50 30 40 40
20 20 40 40 30 40 30 ( category -
below normal, 0 near normal, above normal,
Unit )
21
Seasonal forecast of U.S.
22
Seasonal forecast by I.R.I.
23
Examples of probabilistic forecast excluding
seasonal forecast at JMA JMA uses probabilistic
expression not only in seasonal forecast but in
many forecasts where there is uncertainty of
forecast. (1) Short-range forecast
Tokyo District Today
North-easterly wind, fine, occasionally
cloudy, Wave 0.5m
Probability of Precipitation
12-18 10
18-00 0 Temperature
forecast todays
maximum in Tokyo 14 degrees centigrade
24
(2) One-week forecast
10 / 16 11/15 7/15 9/16
9/17 11/18 70
50 40 30 30
40
Issued at 23,NOV
25
(3) Typhoon Forecast
The probability that a district will be in storm
warming area is also being issued.
26
Quality of Probabilistic Forecast What is good
probabilistic forecast? A good probabilistic
forecast must express the uncertainty of forecast
exactly and have large dispersion from climatic
proportion of frequency (a) Reliability (b)
Resolution
27
(a) Reliability Probability forecast P was
issued M(P) times for a event E. In
M(P) times, event E occurred N(P) times.
If probability forecast is reliable for
large number of M(P). Ex. Probability 30 was
issued 50times. Event E is expected
to occur about 15times in them.
28
The reliability diagram
Event E
Ideal reliability diagram
Relative frequency
Forecast probability
29
Reliability Diagram
Monthly mean surface temperature forecast in Japan
Relative frequency
Forecast probability
30
monthly mean Psea anomalies gt0
Reliability diagram Example of probability by
ensemble one month forecast
31
(b) Resolution Climatorogical relative frequency
for a event is perfectly reliable so far as there
is no climatic Change. ex. It is known that
relative frequency of rainy day is about 30 at
Tokyo from historical data set. Can we issue
probability of 30 as a tomorrow's probability
of precipitation at Tokyo every day? When the
reliability is perfect, dispersion of probability
from climatorogical relative frequency is
another measure of probabilistic forecast quality
resolution. The best resolution probabilistic
forecasts are those of 100 or 0 provided that
reliability is perfect. perfect forecast.
32
Monthly mean surface temperature forecast in Japan
Resolution
Relative frequency
Forecast probability
Frequency
33
Resolution of probability forecast
7day mean Z500 anomaliesgt0
1st week
Reliability diagram
Relative frequency(resolution)
1st week probabilistic forecast is better than
2nd week in resolution measure
2nd week
From one-month ensemble forecast
34
Quantitative evaluation of probabilistic
forecast -------The Brier score------- The Brier
score is defined as pi forecast
probability vi1 if event E occurred 0 if
not occurred N total number of forecast
b is mean square error of probability
35
After some algebraic transformations, b can be
rewritten as
brel
bres
bunc
where Nt frequency of forecast probability
pt Mt frequency of occurrence of event E within
Nt
36
represents Reliability
represents Resolution
represents Uncertainty (not depends on the
forecast. shows difficulty of forecast)
Murphys decomposition(1973)
37
Brier skill score The absolute value of brier
score is difficult to understand except perfect
reliability and resolution case (0). Then
usually following Brier skill score is
used where bc is the brier score of
climatorogical relative frequency( probabilistic
climate forecast).B1 when the forecast is
perfect. Additionally, following skill score is
also used.
BrelBres1 when the forecast is perfect.
38
Example of Brier skill score
Ensemble one-month forecast
7day mean Z500anomaliesgt0
2nd week
1st week
All scores are expressed in
39
(No Transcript)
40
(4) Economical value of forecast ----a
consideration with Cost/Loss model----- The
Cost/Loss model an event E ? If E occurs, the
loss is L due to a damage ? To protect form the
damage, a action costs C (ltL) ex. If the
temperature exceeds a threshold, a crop is
damaged by a pest. The damage is L. To protect
the crop from a pest , spraying agrochemical is
necessary and it costs C.
41
The cost/benefit model a Event E if the event E
occurs , a benefit is B. If not occurs, we lose
C. Example? If it is fine, a benefit B is
earned by selling lunch boxes, but if it rains,
no benefit is earned. The cost to make lunch
boxes is C.
The discussion of Cost/Loss is almost the same
as that of Cost/Benefit model.
42
The case without forecast (Cost/Loss
model) Considering D times operations. The
climatorogical proportion of occurrence of event
E is R. When always taking action to protect from
damage, the expense is , If no action is done,
the expense is Then, when RgtC/L If the event E
occurs frequently, it is better to take action
always.
43
The case with perfect forecast (Cost/Loss
model) If forecast is perfect, we take action
only when event E is forecasted. Then the expense
is of course,
Perfect forecast always reduces the expense
44
The case with actual deterministic
forecast Actual forecast sometimes fails. Then we
make following contingency matrix

Occurs
Forecast
( ) shows in case of perfect forecast
If we use these forecasts, the expenses
corresponding individual boxes above are,
Occurs
Take Action
45
The relationships with pre-defined variables are
Occurs
Forecast
We newly define following variables which express
forecast skill
Hit rate
False-alarm rate
The lager H is and the smaller F is , the
better forecast is. H1 and F0, for perfect
forecast
46
The expense with these forecasts is
If forecast is worse, Mp sometimes become lager
than Mclim2 or Mclim1 . IF H0 and F1(the worst
forecast!) MpDLRD(1-R)CgtMclim1,Mclim2
47
It should be also noted that Mp depends not only
on H and F but C/L and R .
An imperfect deterministic forecast is not always
useful for all users.
48
The case with probabilistic forecast (Cost/Loss
model) How do we use probabilistic
forecast? Consider to take action always when the
probabilistic forecast is P0 for the
occurrence of event E. The expense
is, Mp0DpC where Dp is total frequency that the
event E was forecasted with probability P0 If
probabilistic forecast is perfectly reliable, the
frequency that event E occurred is DpP0 and then
the expense without taking action is McDpP0L
49
Probabilistic forecast is useful when, Mp0DpC
lt McDpP0L Therefore, we should take action
when P0gtC/L This is the simplest way to use
probabilistic forecast for decision making. Note
the threshold probabilities are different for
individual users with different C/L and all
users can get some gain with their own
threshold. In case of Cost-benefit model, with
similar calculations, the criterion is, P0
gtC/(BC)
50
Ex. 1(Cost-Loss model) If the temperature
exceeds a threshold 33?, a crop is damaged by a
pest. The damage is L10,000 To protect the
crop from a pest , spraying agrochemical is
necessary and it costs C3,000 C/L3000/100000.3
Consider 10 times forecast of above 33? with
probability 20(30,40) . When we take
action the expense is 3000x1030,000
When we do not take action, the expense is
10000x(10x0.2)20,000 for probability 20
10000x(10x0.3)30,000 for probability 30
10000x(10x0.4)40,000 for probability 40 We
had better take action when Probabilitygt30C/L

51
Example for cost-benefit model If it is fine, a
benefit B is earned by selling lunch boxes, but
if it rains, no benefit is earned. The cost to
make lunch boxes is C ,which is the loss when it
rains. Price of lunch box10 and 100 lunch
boxes are sold in a fine day. The cost to make
one lunch box5 The benefit in a fine day is
B(10-5)x100500 The cost is C5x100500
,which is the loss when it rains C/(BC)0.5 10
times forecast of fine with probability
40(50,60) When we sell the lunch boxes, The
expected cost is 500x(10x(1-0.4))30,000
for probability 40 500x(10x(1-0.5))25,000
for probability 50 10000x(10x(1-0.6)))20,
000 for probability 60 The expected benefit
is 500x(10x0.4)20,000 for probability
40 500x(10x0.5)25,000 for probability
50 500x(10x0.6)30,000 for probability
60
We had better sell when Probabilitygt50C/(BC)
52
Verification of probabilistic forecast -----How
to use an actual probabilistic forecast---------

We used a important assumption to derive the
threshold probability to take action in the
previous section. Assumption probabilistic
forecast is perfectly reliable. Although this
condition would be satisfied approximately in
most practical probabilistic forecasts, there is
no guarantee that it is always satisfied. In
addition, the expense reduction (Mp-Mclim) with
probabilistic forecast also depends on the
resolution of probabilistic forecast and we
cannot know how much it is without verification.
Therefore verification is important to use
probabilistic forecast actually.
53
A Verification of probabilistic forecast from the
economical view point We assume E will occur
when PgtPt and E will not occur when PltPt where
Pt is a threshold probability. And again we make
contingency matrix as follows.
Occurs
Forecast
54
As similar to before, the expense is,
The expense for perfect forecast is,
That for climatic forecast is,
55
We define the Value of forecast as the
reduction in Mp over Mclim normalized by the
maximum possible reduction. That is,
V(Pt)1 for the perfect forecast and negative
for bad forecast.
56
We calculate V for various threshold
probabilities and C/L. For a given C/L, and a
event E, the optimal value is,
57
Probability of T850anomalies gt0
From Palmer(2000)
For a user with C/L0.6, V12 and best
threshold probability is 70, although
threshold 60(C/L) gives some benefit(7).
60 line
40
20
10
30

• From this graph,
• The user can choose the threshold which brings
  maximum benefit( Note reliability is not
  completely perfect).
• A user with a C/L can estimate maximum benefit

90
50
80
70
60
58
(No Transcript)
59
Vopt for Probability forecast
V for deterministic forecast
The graph of Vopt for probabilistic forecast is
higher and wider than that of V for the
deterministic forecast. Because, for
deterministic forecast, only one contingency
matrix is made and then only one graph of V is
drawn. On the other hand, Vopt is the maximum of
the graphs of V with various probability
threshold.
60
A verification example of monthly surface
temperature anomaly probability forecast in Japan
(Statistical down scaling from ensemble
forecast))
Surface temperature (above normal)
Surface temperature (below normal)
Surface temperature
Reliability diagram for above normal
Reliability diagram for below normal
61
A verification of dymamical one-month ensemble
forecast Anom.(Z500)gt0 28day mean ( winter
2001)
62
1st week
2nd week
63
3-4th week
64

• Conclusions
• Seasonal forecast has uncertainty due to chaos
  of atmospheric flow. The probabilistic forecast
  is the best method to express this uncertainty.
• In probabilistic forecast, a user can use his/her
  own threshold probability to take action
  depending on his/her own C/L in Cost-Loss model.
  For a sufficiently reliable probabilistic
  forecast, the threshold probability is equal to
  C/L. If reliability is not enough, user can
  know the best threshold probability by a
  verification.
• In general, the probabilistic forecast is
  superior to the deterministic forecast at least
  from the economical point of view so far as the
  forecast is not perfect although it seems
  difficult in some degree. More dissemination of
  probabilistic forecast is necessary.

65
Please remember the words uncertainty of
forecast and cost-loss ratio C/L. Thank you!

66
Monthly mean surface temperature forecast at Japan
Surface temperature above normal
Vopt