Nikolaos Kourentzes Dr. Sven F. Crone LUMS Department of Management Science

Transcription

1 Nikolaos Kourentzes Dr. Sven F. Crone LUMS Department of Management Science

2 Agenda ISF 2009 I. Motivation II. III. IV. i. Why Neural Networks? ii. Why focus on the input vector? iii. Why high frequency time series? Experimental Design I. The experiment II. The dataset Results I. Statistical similarities among models II. Performance against benchmarks Conclusions & Further Research

3 Motivation Why Neural Networks? Artificial Neural Networks in Forecasting Promising performance 77% (out of 84) articles found NN outperforming benchmarks (73% according to Adya & Collopy, 98) Large scale studies (100+ time series) NN at least as good as benchmarks [Hill et al., 96, Liao & Fildes, 05] Forecasting Competitions (M3) NN lower accuracy than statistical models [Makridakis & Hibbon, 00] NN produce unreliable forecasts criticised to offer little promise even after much research [Armstrong, 06] Why? What does ANN research suggest?

4 Motivation Focus on the input vector Modelling complexity gives rise to the problems Problem caused by inconsistent trial and error modelling approaches [Zhang et al., 98] Input variable selection The most important issue in forecasting with NN [Zhang, 01, Zhang et al., 01, Zhang et al., 98, Darbellay and Slama, 00 ] No widely accepted methodology on how to select the input variables [Anders and Korn, 99, Zhang et al., 98] 70.5% of papers are based on trial and error approaches to model the input vector! Problem in NN forecasting literature... Focus on the input vector!

5 Motivation High frequency data Low and High Frequency data present different modelling challenges Short time intervals; daily, hourly, minutes, etc Conventional statistical methods have problems [Granger, 98] Demand a different approach to forecasting [Taylor et al., 06] Transportation, NN3 & NN5 competitions Gains of ANN in high frequency domain in comparison to benchmarks Applicability of standard input identification methodologies for ANN challenged [Crone & Kourentzes, 08] Large gap in research 7.6% in forecasting literature 5.4% in ANN literature High frequency data are gaining importance in business practice!

6

7 Experimental Design Objectives & Setup Evaluate different methods to identify the input vector for ANN for high frequency data Compare with the naive model, exponential smoothing family and simple ANNs (naive ANNs) Forecast 7 future observations (1 week of daily data), evaluate using SMAPE Rolling origin evaluation (usually overlooked in ANN literature) Common ANN architecture except the input vector, 40 initialisations Variable input, 3 hyperbolic tangent hidden nodes, 1 linear output node The dataset?

8 NN5 competition dataset Experimental Design Time Series Initial set 111 time series K-means Clustering Test for trend (Regression/ Cox-Stuart test) Preference to time series with less missing values Final set 42 time series Increase homogeneity of the dataset and reduce number of time series for computational reasons What are the properties of the dataset?

9 Dataset properties Experimental Design Time Series Daily time series, 761 days 25 months data Double seasonality Day of the week & annual pattern Not enough data to capture the annual pattern Canova-Hansen Test Seasonality is deterministic for all time series Calendar effects Strong presence of Good Friday & Xmas Holiday bank holidays

10 Experimental Design Time Series 0.5 Training Validation 0 Test Training Validation 0 Test Training Validation 0 Test Validation set: 56 observations Test set: 56 observations

11 Experimental Design Objectives & Setup Input vector specification methodologies ACF/PACF ACF (acf) Nonlinear ACF (nlacf) PACF: Yule Walker (yule) PACF: OLS (ls) PACF: Burg (burg) Spectral Analysis (sa) All their combinations... Regression Linear forward regression (forw) Linear backward regression (back) Linear stepwise regression (auto) Naive models Only past lag (t-1) (naive) All lags (full season, all) 21 ANN models + Benchmarks

12 ACF/PACF models Experimental Design Input Vectors Input Vector Neural Network t -1 t -12 t -13 t For combinations of models all identified significant lags are used

13 Spectral Analysis Experimental Design Input Vectors Periodogram Input Vector t -6 t -12 t -4 t Neural Network SA is a mathematically equivalent to ACF [Box et. al., 1994]

14 Regression models Experimental Design Input Vectors x Time Series Regression Input Vector Neural Network t t t -35 t

15 3 different types of pre-processing Experimental Design Time Series No-Diff(erencing) Original domain Input vector and modelling in the original domain Season-Diff Remove Season Suggested by the literature [Zhang & Kline, 07] Input vector and modelling after seasonal differencing Input-Diff Remove season for the input vector Input vector after seasonal differencing, modelling on the original data BUT deterministic seasonality identified Dummy variables instead of differencing to encode seasonality [Ghysels & Osborn, 01] Shown to improve accuracy for ANN [Crone & Kourentzes, 09] 42 time series X 21 ANN models X 40 initialisations X 6 types of pre-processing = 211,680 networks...!

16

17 Worst Best Average SMAPE Results Differences across pre-processing Nonparametric Nemenyi test results Input-Diff- Dummy No-Diff- Dummy No-Diff Trn Val Tst No-Diff-Dummy Season-Diff-Dummy Input-Diff-Dummy No-Diff Season-Diff Input-Diff Input-Diff Season-Diff Season-Diff- Dummy Pre-processing type Important - Significant effect of seasonality dummy encoding

18 Results Statistical Differences among ANN models Rank -- Model -- NN-burg NN-naive NN-reg-forw NN-reg-auto NN-nlacf NN-fs NN-reg-back NN-nlacf+burg NN-ls NN-ywe NN-acf NN-acf+burg NN-acf+ls NN-acf+ywe NN-sa+burg NN-nlacf+ls NN-nlacf+ywe NN-sa+ls NN-sa+ywe NN-sa NN-all Statistically insignificant differences ACF or SA or PACF Regression ACF/SA and PACF Naive models Nemenyi test results across all models (Input-diff-Dummy) 1. Regression 2. ACF or SA or PACF 3. Heuristics 4. ACF/SA and PACF In contrast to low frequency equivalent experiments 1. Regression 2. ACF/SA and PACF 3. ACF or SA or PACF 4. Heuristics Different results from low frequency equivalent experiments [Kourentzes & Crone, 08, 09]

19 Results Input vector lengths Naive models ACF or SA or PACF ACF/SA and PACF NN5 dataset Regression + dummies ANN_burg ANN_naive ANN_reg_auto ANN_reg_forw ANN_nlacf ANN_fs ANN_reg_back ANN_nlacf+burg ANN_ls ANN_ywe ANN_acf ANN_acf+burg ANN_acf+ywe ANN_acf+ls ANN_sa+burg ANN_nlacf+ls ANN_nlacf+ywe ANN_sa+ls ANN_sa+ywe ANN_sa ANN_all Input-Diff No-Diff Season-Diff Ordered by mean rank (Nemenyi test result) Seasonal elements captured by dummies Short input vectors adequate to capture nonlinearities in the level

20 NN-naive NN-all NN-fs NN-ywe NN-ls NN-burg NN-acf NN-nlacf NN-sa NN-acf+ywe NN-acf+ls NN-acf+burg NN-nlacf+ywe NN-nlacf+ls NN-nlacf+burg NN-sa+ywe NN-sa+ls NN-sa+burg NN-reg-auto NN-reg-forw NN-reg-back EXSM Naive 1 Naive 7 Naive % 22.8% 22.4% 22.2% 22.2% 22.0% 22.1% 22.5% 23.1% 22.4% 22.4% 22.4% 23.4% 23.4% 22.5% 22.8% 22.8% 22.4% 22.0% 22.0% 22.0% 22.8% 48.86% 30.3% 29.3% Results MAPE & Benchmarks 24.0% SMAPE - Input-Diff + Dummies 23.5% 23.0% 22.5% 22.0% 21.5% 21.0% 20.5% Naive models ACF/SA and PACF ACF or SA or PACF Regression Benchmark models Worse than benchmark ANN outperform benchmark (significantly) when modelled properly

21 Conclusions & Further Research Pre-processing is important. Deterministing or stochastic seasonality modelling is important for high frequency data. Significant differences between competing input vector selection methodologies Differences in the behaviour of the input vector methodologies between low and high frequency data modelling Many low frequency methods tend to overpopulate the input vector Better than the include everything strategy! What would be the effect of stochastic seasonality? How robust are the results to different datasets?

22 Nikolaos Kourentzes Lancaster University Management School Centre for Forecasting Lancaster, LA1 4YX, UK Tel. +44 (0)