Experiences concerning fuzzy-verification and pattern recognition methods

1
Experiences concerning fuzzy-verification and
pattern recognition methods

• Ulrich Damrath

2
Outlook

• Results on operational verifcation for winter and
  summer month
• An approach concerning significance test of
  fuzzy-verification results
• Estimation of consistency of forecasts using a
  pattern recognition method (CRA method by Beth
  Ebert)

3
GME
COSMO-EU
COSMO-DE
Fractions skill score for forecasts of GME,
COSMO-EU and COSMO-DE for December 2008,
forecast time 06-18 hours
4
GME
COSMO-EU
COSMO-DE
ETS upscaling for forecasts of GME, COSMO-EU and
COSMO-DE for December 2008, forecast time 06-18
hours
5
Global
Europe
Germany
Fractions skill score for forecasts of GME,
COSMO-EU and COSMO-DE for August 2009, forecast
time 06-18 hours
6
Global
Europe
Germany
ETS upscaling for forecasts of GME, COSMO-EU and
COSMO-DE for August 2009, forecast time 06-18
hours
7
Examination of statistical significance of
fuzzy-verification results using bootstrapping

• Basic idea of bootstrapping
• Repeat a resampling all elements of a given in a
  sample of forecasts and observations as often as
  necessary (N times) and calculate the relevant
  score(s)
• Calculate from N scores statistical properties of
  the sample such as mean value standard deviation,
  confidence intervals and quantiles
• Application to fuzzy-verification
• Resampling is done using blocks.
• Blocks are defined as single days.
• Number of resampling cases NDays100
• Calculation scores from N samples for NT
  thesholds and NW windows
• Calculation of quantiles for each window and
  threshold

8
Values and quantiles 0.1 and 0.9 for Upscaling
ETS GME, period June - August 2009
Germany
9
Values and quantiles 0.1 and 0.9 for Upscaling
ETS COSMO-EU, period June - August 2009
Germany
10
Values and quantiles 0.1 and 0.9 for Upscaling
ETS COSMO-DE, period June -August 2009
Germany
11
Next step Evaluation of significance

• First impression Is the result of Model 1 better
  than the result of Model 2?
• Significance hypothesis checked using a
  Wilcoxon-test (IDL-code RS_TEST)

12
Differences between GME and COSMO-EU
Germany
ETS(COSMO-EU) - ETS(GME)
Significance test
COSMO-EU better than GME COSMO-EU worse than GME
13
Differences between GME and COSMO-DE
Germany
ETS(COSMO-DE) - ETS(GME)
Significance test
COSMO-DE better than GME COSMO-DE worse than GME
14
Differences between COSMO-DE and COSMO-EU
Germany
Significance test
ETS(COSMO-DE) - ETS(COSMO-EU)
COSMO-DE better than COSMO-EU COSMO-DE worse than
COSMO-EU
15
Differences between COSMO-DE and COSMO-EU
Significance test
ETS(COSMO-DE) - ETS(COSMO-EU)
COSMO-DE better than COSMO-EU COSMO-DE worse than
COSMO-EU
16
Example of good precipitation forecast of COSMO-DE
Zeigten die numerischen Modelle und die
statistischen Prognose- --------------------------
------------------------------------- verfahren
Signale für das Ereignis? ------------------------
----------- Die Numerik zeigte im Vorfeld
vermehrt Signale für kräftige Konvektion.
Während diese bei GME und COSMO-EU recht breit
gestreut und pauschal auftraten, signalisierten
mehrere COSMO- DE-Läufe eine linienartige
Struktur mit unwetterartigen Zellen (auf Basis
der 1- bzw. 3-stündigen RR-Prognosen) im
Grenzbereich von Hessen zu NRW und
Niedersachsen. Diese Linie trat dann in den
Mittags- und frühen Nachmittagsstunden
tatsächlich auf, wenn auch nicht 100ig
kongruent, aber doch in der Nähe, so dass in
diesem Fall von einer guten Prognose gesprochen
werden kann (mehr dazu siehe "Zentraler
UW- Sofortbericht" der VBZ).
17
Example of good precipitation forecast of COSMO-DE
3h-precipitation forecast of COSMO-DE valid
10.08.2009 12 UTC, left 03 UTC 9h, right 06 UTC
6h.
3h-precipitation observation 10.08.2009 09-12 UTC
18
Example of good precipitation forecast of
COSMO-DE compared to other models
19
Example of good precipitation forecast of
COSMO-DE compared to other models
20
About consistency and inconsistency

• Forecasters are interested in consistent model
  forecasts.
• But due to growing of errors during forecast time
  forecasts consistency cannot be expected
  concerning all properties of the forecasted
  fields!
• Inconsistency Differences between forecasts that
  are valid for the same time concerning different
  properties of the forecasted fields (properties
  of the pattern, values at special points of
  interest, extreme values, ...)
• Differences between the forecasted fields
  concerning
• phase,
• amplitude
• and the remaining part

21

• Entity-based QPF verification (rain blobs)
• by E. Ebert (BOM Melbourne)
• Verify the properties of the forecast rain system
  against the properties of the observed rain
  system
• location
• rain area
• rain intensity (mean, maximum)

CRA error decomposition The total mean squared
error (MSE) can be written as MSEtotal
MSEdisplacement MSEvolume MSEpattern
Configuration for the current study -
Observations forecasts 06-30 hours -
Forecasts forecasts 30-54 hours and
forecasts 54-78 hours
22
Dark forecasts 30-54 h Lightforecasts 54-78 h
23
Dark forecasts 30-54 h Lightforecasts 54-78 h
24
Dark forecasts 30-54 h Lightforecasts 54-78 h
25
Dark forecasts 30-54 h Lightforecasts 54-78 h
26
Summary

• Scores like Fractions skill score and ETS from
  upscaling show in general advantages of COSMO
  models compared to GME.
• This is true especially for summer months.
• For winter months all models have nearly the same
  quality for low precipitation amounts and large
  window sizes for averaging.
• Significance test lead to the results, that
• The advantages of COSMO models compared to GME
  are statistically significant for most window
  sizes and precipitation amounts.
• The differences between COSMO-EU and COSMO-DE are
  not significant altough there are systematical
  differences for different precipitation amounts
  and window sizes.
• There are some cases with very useful
  precipitation forecasts of COSMO-DE compared to
  COSMO-EU from the view of forecasters.
• A study about the consistency of precipitation
  forecasts showed - it could be expected , but now
  it is proved - that
• Forecasts of high precipitation amounts are less
  consistent than those for low precipitation
  amounts.
• Pattern errors contribute most to forecast
  errors.
• During winter months volume errors are higher
  than displacement errors.
• During summer months displacement errors are
  higher than volume errors.