Outline for Today

1
Outline for Today

• Brief Lecture on weather forecasting
• A comment on the upcoming Exam 4

2
Theoretical Models and Accuracy

• As we discussed previously, one of the most
  useful ways of evaluating a theoretical model is
  to compare the predictions of the model against
  empirical data
• Unfortunately, many industries do not provide you
  with information on the accuracy of their
  predictions
• how many people improve after taking a drug
• how many people actually lose weight by eating
  Subway sandwiches
• how many horoscopes accurately predict the future
• As a consequence, it is almost impossible for
  you, as a consumer, to evaluate the quality of
  the product

3

• In general, weather forecasters make their
  predictions based on either mathematical
  models/simulations of weather systems, past
  weather conditions, intuitive judgments based on
  the movement of current weather systems, or all
  of the above

4

• Unfortunately, weather forecasters do not tell
  usthe consumershow accurate their forecasts
  are. We dont know what the track record is
  for any weather/news team. (Moreover, they
  dont keep records of past forecasts on their
  websites, thereby preventing consumers from
  investigating the matter themselves.)

5
A scientific analysis of the accuracy of weather
forecasting

• Students recorded the 5-day forecasts (on Sunday
  nights) for the following sources
• ABC Ch. 7 Panagiota Sveronis
• CBS Ch. 2 Lisa Carlisle
• WGN Ch. 9 Debjit Sarkar
• NBC Ch. 5 Mary-Jennelle Guevara
• Fox Ch. 32 Greg Zykowski
• Weather.com Allison Vadasz
• Chicago Tribune Manuel Aponte
• Chicago Sun Times Kelly Meehan-Coussee
• I recorded the actual, end-of-the-day weather
  from an unrelated source Weather Underground.
• This allows us to compare the predictions made by
  the different teams with the actual weather.

6
Average temperature observed, as recorded by
Weather Underground, Chicago
100
80
Actual temperature
60
40
2
4
6
8
10
Day
7
Predictions made by NBC, Channel 5, on Sunday
nights
Average absolute discrepancy 4.7 degrees not
too bad, eh?
8
12
Errors tended to get larger as more days passed
between when the prediction was made (Sunday
evenings) and the day in question. Five days
later, the average prediction was off by about 10
degrees. (These numbers are based on averages
across NBC, ABC, CBS, WGN, Sun Times.)
10
Discrepancy between predicted and actual
temperature
8
6
4
1
2
3
4
5
Number of days since prediction
9
NBC Mean Discrepancy 4.7
ABC Mean Discrepancy 11
CBS Mean Discrepancy 6.8
100
100
100
80
80
80
Actual temperature
60
60
60
40
40
40
40
60
80
100
40
60
80
100
40
60
80
100
Predicted temperature
Predicted temperature
Predicted temperature
SUN TIMES Mean Discrepancy 5.5
WGN Mean Discrepancy 7.3
100
100
80
80
Actual temperature
60
60
40
40
40
60
80
100
40
60
80
100
Predicted temperature
Predicted temperature
10
TRUE.TEM NBC.TEMP ABC.TEMP CBS.TEMP
WGN.TEMP SUN.TEMP TRUE.TEM 1.00 0.85
0.72 0.89 0.58 0.74 NBC.TEMP 0.85
1.00 0.85 0.98 0.74
0.83 ABC.TEMP 0.72 0.85 1.00 0.90
0.79 0.95 CBS.TEMP 0.89 0.98
0.90 1.00 0.75 0.88 WGN.TEMP 0.58
0.74 0.79 0.75 1.00
0.75 SUN.TEMP 0.74 0.83 0.95 0.88
0.75 1.00
11
As an additional comparison, we can generate
predictions in another way based on day before
temperature In this case, were assuming that
the best bet for tomorrows temperature is
todays. Average discrepancy 8.3
12
Day before method
NBC
100
100
80
80
Actual temperature
Actual temperature
60
60
40
40
40
60
80
100
40
60
80
100
DAY.BEFORE
NBC.TEMP
Average discrepancy 11.8 degrees
Average discrepancy 4.7 degrees
13
What about broader weather conditions?

• Broader weather conditions
• (a) sunny/clear
• (b) partly cloudy/mostly cloudy
• (c) thunderstorm
• (d) rain/showers
• Accuracy in prediction
• NBC 55
• ABC 34
• CBS 55
• WGN 34
• SUN 34

day before accuracy 34
14
Summary

• It appears that the various weather teams do a
  pretty good job in predicting the temperature
• They were only off by approximately 4.7 11
  degrees.
• Their ability to make accurate forecasts was
  lower for extended forecasts
• In predicting weather conditions (e.g., rain,
  shine), the best outlets were only right half the
  time the worst outlets were right 34 of the
  timewhich is as accurate as simply guessing the
  weather today based on yesterdays weather.
• Nonetheless, you wouldnt know how good they are
  without doing the research yourself.