Theoretical Models and Accuracy

1
Theoretical Models and Accuracy

• As we discussed previously, one of the most
  useful ways of evaluating a theoretical model is
  to compare the predictions of the model against
  empirical data
• Unfortunately, many industries do not provide you
  with information on the accuracy of their
  predictions
• how many people improve after taking a drug
• how many people actually lose weight by eating
  Subway sandwiches
• how many horoscopes accurately predict the future
• As a consequence, it is almost impossible for
  you, as a consumer, to evaluate the quality of
  the product

2

• In general, weather forecasters make their
  predictions based on either mathematical
  models/simulations of weather systems, past
  weather conditions, intuitive judgments based on
  the movement of current weather systems, or all
  of the above

3

• Unfortunately, weather forecasters do not tell
  usthe consumershow accurate their forecasts
  are. We dont know what the track record is
  for any weather/news team. (Moreover, they
  dont keep records of past forecasts on their
  websites, thereby preventing consumers from
  investigating the matter themselves.)

4
A scientific analysis of the accuracy of weather
forecasting

• Students recorded the 5-day forecasts (on Sunday
  nights) for 3 weeks for the following sources
• WGNTV
• NBC
• CH. 7
• Fox News
• Intellicast
• Chicago Tribune
• Dr Fraley recorded the actual, end-of-the-day
  weather from an unrelated source Weather
  Underground.
• This allows us to compare the predictions made by
  the different teams with the actual weather.

5
Average temperature observed, as recorded by
Weather Underground, Chicago
April 5th April 23rd, 2004
6
Predictions made by WGN, Channel 9, on Sunday
nights
Average absolute discrepancy 7.7 degrees not
too bad, eh?
7
Errors tended to get larger as more days passed
between when the prediction was made (Sunday
evenings) and the day in question. Five days
later, the average prediction was off by about 10
degrees. (These numbers are based on WGN TV)
Discrepancy between predicted and actual
temperature
Number of days since prediction
8
WGNTV Mean Discrepancy 6.3
NBC Mean Discrepancy 4.2
CH. 7 Mean Discrepancy 7
Chicago Tribune Mean Discrepancy 6.2
Intellicast Mean Discrepancy 8.9
Fox Mean Discrepancy 3.9
9
As an additional comparison, we can generate
predictions in another way based on day before
temperature In this case, were assuming that
the best bet for tomorrows temperature is
todays. Average discrepancy 7.35
10
What about broader weather conditions?

• Broader weather conditions
• (a) sunny/clear
• (b) partly cloudy/mostly cloudy
• (c) thunderstorm
• (d) rain/showers
• Accuracy in prediction
• NBC 73
• Intellicast 80
• Chi. Trib 40
• Day before 71

11
Summary

• It appears that the various weather teams do a
  pretty good job in predicting the temperature
• They were only off by approximately 4 9
  degrees, on average.
• Their ability to make accurate forecasts was
  lower for extended forecasts.
• In predicting weather conditions (e.g., rain,
  shine), the various stations were right 40 (Chi.
  Tribune) 80 (Intellicast) of the time.
• Nonetheless, you wouldnt know how good they are
  without doing the research yourself.

12
Quiz

• True/False According to the graphs last week.
  You can predict a persons level of Extroversion
  based on their Astrological Sign.
• In our in-class exercise on horoscopes, which
  group felt that their horoscopes were most valid?
• the group with the true horoscopes
• the group with the true and false horoscopes
  provided fairly comparable responses
• the group with the Sun Times horoscopes
• Why should we care about the pseudosciences in
  health care?
• People will use them in stead of scientifically
  supported health care.
• They take money away from psycologists.
• They might harm peoples souls.