Title: FORECAST PERFORMANCE AND DATA MINING
1FORECAST PERFORMANCE AND DATA MININGÂ Forecast
is the ultimate challenge for any econometric
model.Any model that fails to predict is
useless. Exceptions?History may not
repeat itself, but it rhythms.
2Remarks on econometric forecast (1)Forecast
and decision- cost of forecast (forecast is
irrelevant per se) Forecast without properly
defined cost function means
nothing.(2)Forecast standard deviation
please(3)Statistical measures for prediction
accuracy can be problematic in the real
world.
3(4)Out-of-sample monitoring Is yesterdays
model capable of explaining todays
data?(5)Correcting data mining bias in a
historically adequate model. Predictions
based on historically adequate model
resulting from extensive specification
search are subject to data mining bias.
4Data MiningA general modeling process that
applies a sequence of statistical and
non-statistical tools to a data set in an attempt
to obtain a final model. ReferencesYe, J
(1998) On Measuring and Correcting the Effects
of Data Mining and Model Selection. JASA 93
p120-131White, H (2000) A Reality Check for
Data Snooping. Econometrica
5Simulation Results Cost of Data
MiningGeneralized degree of freedom (GDF) is
defined as the sum of sensitivity of each fitted
value to perturbations in the corresponding
value.
6Reality CheckWhenever a good forecasting model
is obtained by an extensive specification search,
there is always the danger that the observed good
performance results not from actual forecasting
ability, but instead of luck. Stock Investment
News letter Simulation using Stationary
bootstraps