FORECAST PERFORMANCE AND DATA MINING

1
FORECAST PERFORMANCE AND DATA MINING Forecast
is the ultimate challenge for any econometric
model.Any model that fails to predict is
useless. Exceptions?History may not
repeat itself, but it rhythms.
2
Remarks on econometric forecast (1)Forecast
and decision- cost of forecast (forecast is
irrelevant per se) Forecast without properly
defined cost function means
nothing.(2)Forecast standard deviation
please(3)Statistical measures for prediction
accuracy can be problematic in the real
world.
3
(4)Out-of-sample monitoring Is yesterdays
model capable of explaining todays
data?(5)Correcting data mining bias in a
historically adequate model. Predictions
based on historically adequate model
resulting from extensive specification
search are subject to data mining bias.
4
Data MiningA general modeling process that
applies a sequence of statistical and
non-statistical tools to a data set in an attempt
to obtain a final model. ReferencesYe, J
(1998) On Measuring and Correcting the Effects
of Data Mining and Model Selection. JASA 93
p120-131White, H (2000) A Reality Check for
Data Snooping. Econometrica
5
Simulation Results Cost of Data
MiningGeneralized degree of freedom (GDF) is
defined as the sum of sensitivity of each fitted
value to perturbations in the corresponding
value.
6
Reality CheckWhenever a good forecasting model
is obtained by an extensive specification search,
there is always the danger that the observed good
performance results not from actual forecasting
ability, but instead of luck. Stock Investment
News letter Simulation using Stationary
bootstraps