Kalman Filter and SVR Combinations in Forecasting US Unemployment

Transcription

1 Kalman Flter and SVR Combnatons n Forecastng US Unemployment Georgos Sermpns, Charalampos Stasnaks, Andreas Karathanasopoulos To cte ths verson: Georgos Sermpns, Charalampos Stasnaks, Andreas Karathanasopoulos. Kalman Flter and SVR Combnatons n Forecastng US Unemployment. Harrs Papadopoulos; Andreas S. Andreou; Lazaros Ilads; Ilas Magloganns. 9th Artfcal Intellgence Applcatons and Innovatons (AIAI), Sep 2013, Paphos, Greece. Sprnger, IFIP Advances n Informaton and Communcaton Technology, AICT-412, pp , 2013, Artfcal Intellgence Applcatons and Innovatons. < / _51>. <hal > HAL Id: hal Submtted on 7 Feb 2017 HAL s a mult-dscplnary open access archve for the depost and dssemnaton of scentfc research documents, whether they are publshed or not. The documents may come from teachng and research nsttutons n France or abroad, or from publc or prvate research centers. L archve ouverte plurdscplnare HAL, est destnée au dépôt et à la dffuson de documents scentfques de nveau recherche, publés ou non, émanant des établssements d ensegnement et de recherche franças ou étrangers, des laboratores publcs ou prvés. Dstrbuted under a Creatve Commons Attrbuton 4.0 Internatonal Lcense

2 Kalman Flter and SVR combnatons n forecastng US Unemployment Georgos Sermpns 1, Charalampos Stasnaks 2 and Andreas Karathanasopoulos 3 1 Unversty of Glasgow Busness School (E-mal: georgos.sermpns@glasgow.ac.uk) 2 Unversty of Glasgow Busness School (E-mal: c.stasnaks.1@research.gla.ac.uk) 3 Unversty of East London Busness School (E-mal: a.karatahnasopoulos@uel.ac.uk) Abstract. The motvaton for ths paper s to nvestgate the effcency of a Neural Network (NN) archtecture, the Ps Sgma Network (PSN), n forecastng US unemployment and compare the utlty of Kalman Flter and Support Vector Regresson (SVR) n combnng NN forecasts. An Autoregressve Movng Average model (ARMA) and two dfferent NN archtectures, a Mult-Layer Perceptron (MLP) and a Recurrent Network (RNN), are used as benchmarks. The statstcal performance of our models s estmated throughout the perod of , usng the last seven years for out-of-sample testng. The results show that the PSN statstcally outperforms all models ndvdual performances. Both forecast combnaton approaches mprove the statstcal accuracy, but SVR outperforms substantally the Kalman Flter. Keywords: Forecast Combnatons, Kalman Flter, Support Vector Regresson, Unemployment. 1 Introducton Many applcatons n the macroeconomc lterature am to derve and compare nformaton from econometrc models forecasts. For that reason, forecastng compettons of lnear and non-lnear archtectures are common and focus on numerous tme seres, such as unemployment, nflaton, ndustral producton, gross domestc product etc. The artfcal NNs are computaton models that researchers nclude n such macroeconomc forecastng schemes, because they embody promsng data-adaptve learnng and clusterng abltes. The motvaton for ths paper s to nvestgate the effcency of a Neural Network (NN) archtecture, the Ps Sgma Network (PSN), n forecastng US unemployment and compare the utlty of Kalman Flter and Support Vector Regresson (SVR) n combnng NN forecasts. An Autoregressve Movng Average model (ARMA) and two dfferent NN archtectures, a Mult-Layer Perceptron (MLP) and a Recurrent Network (RNN), are used as benchmarks. The statstcal performance of our models s estmated throughout the perod of , usng the last seven years for out-ofsample testng. The results show that the PSN statstcally outperforms all models ndvdual performances. Both forecast combnaton approaches mprove the statstcal accuracy, but SVR s substantally better than the Kalman Flter.

3 Secton 2 s a short lterature revew and Secton 3 follows wth the descrpton of the dataset used n ths applcaton. Sectons 4 and 5 gve an overvew of the forecastng models and the forecast combnaton methods mplemented respectvely. The statstcal performance of our models s presented n Secton 6. Fnally, some concludng remarks are summarzed n Secton 7. 2 Lterature Revew Forecastng unemployment rates s a very popular and well documented case study n the lterature (see amongst others Rothman [16], Montgomery et al. [14] and Koop and Potter [11]). Swanson and Whte [20] forecast several macroeconomc tme seres, ncludng US unemployment, wth lnear models and NNs. In ther approach, NN archtectures present promsng emprcal evdence aganst the lnear VAR models. Johnes [9] reports the results of another forecastng competton between lnear autoregressve, GARCH, threshold autoregressve and NNs models, appled to the UK monthly unemployment rate seres. In hs applcaton, NNs are superor when the forecastng horzon s 18 and 24 months ahead, but fal to outperform the other models n shorter forecastng horzons. Lang [12] apples Bayesan NNs n forecastng unemployment n West Germany and shows that they present sgnfcantly better forecasts than tradtonal autoregressve models. Teräsvrta et al. [22] examne the forecast accuracy of lnear autoregressve, smooth transton autoregressve and NN models for 47 monthly macroeconomc varables, ncludng unemployment rates, of the G7 economes. The emprcal results of ther study pont out the rsk for mplausble NN forecasts at long forecastng horzons. Nonetheless, ther forecastng ablty s much mproved when they are combned wth autoregressve models. Ths dea of combnng forecasts orgnates from Bates and Granger [1]. Newbold and Granger [15] also suggested combnng rules based on varances-covarances of the ndvdual forecasts, whle Deutsch et al. [3] acheved substantally smaller squared forecasts errors combnng forecasts wth changng weghts. Harvey [8] and Hamlton [7] both propose usng state space modelng, such as Kalman Flter, for representng dynamc systems where unobserved varables (socalled state varables) can be ntegrated wthn an observable model. Fnally, Teru and Van Djk [23] also suggest that the combned forecasts perform well, especally wth tme varyng coeffcents. 3 US Unemployment dataset In ths applcaton, we forecast the monthly percentage change of the US unemployment rate (UNEMP), as provded by the onlne Federal Reserve Economc Data (FRED) database of the Federal Reserve Bank of St. Lous 1. The forecastng perfor- 1 Based on the descrpton gven by FRED, the US unemployment rate or cvlan unemployment rate represents the number of unemployed as a percentage of the labour force. Labour force data are restrcted to people 16 years of age and older, who currently resde n 1 of the

4 US Unemployment Rate 1-Jan-72 1-Jan-75 1-Jan-78 1-Jan-81 1-Jan-84 1-Jan-87 1-Jan-90 1-Jan-93 1-Jan-96 1-Jan-99 1-Jan-02 1-Jan-05 1-Jan-08 1-Jan-11 mance of our models s explored over the perod of 1972 to 2012, usng the last seven years for out-of-sample evaluaton. The tme seres s seasonally adjusted and t s dvded nto three sub-perods as summarzed n Table 1 below: Table 1. The US Unemployment Dataset - Neural Networks Tranng Dataset PERIODS MONTHS START DATE END DATE Total Dataset //01/ /12/2012 Tranng Dataset (In-sample) //01/ /12/1998 Test Dataset (In-sample) 84 01/01/ /12/2005 Valdaton Dataset (Out-of-sample) 84 01/01/ /12/2012 The followng graph presents the US unemployment rate for the perod under study: st January st December 2012 Fg. 1. The US Unemployment Rate In the absence of any formal theory behnd the selecton of the nputs of a NN, we conduct some NN experments and a senstvty analyss on a pool of potental nputs n the n-sample dataset n order to help our decson. Fnally, we select as nputs sets of autoregressve terms of UNEMP that provde the best statstcal performance for each network n the test sub-perod. These sets are presented n Table 2 below: 50 states or the Dstrct of Columba, who do not resde n nsttutons (e.g., penal and mental facltes, homes for the aged) and who are not on actve duty n the Armed Forces. 3

5 Table 2. Neural Networks Inputs MLP RNN PSN UNEMP (1)* UNEMP (1) UNEMP (1) UNEMP (2) UNEMP (3) UNEMP (2) UNEMP (4) UNEMP (4) UNEMP (3) UNEMP (5) UNEMP (6) UNEMP (6) UNEMP (7) UNEMP (7) UNEMP (8) UNEMP (10) UNEMP (9) UNEMP (10) UNEMP (11) UNEMP (11) UNEMP (11) UNEMP (12) UNEMP (12) UNEMP (12) *UNEMPL UNEMP (1) s the frst autoregressve term of the UNEMP seres 4 Forecastng Models 4.1 Auto-Regressve Movng Average Model (ARMA) In ths paper an ARMA model s used to benchmark the effcency of the NNs statstcal performance. Usng as a gude the correlogram and the nformaton crtera n the n-sample subset, we have chosen a restrcted ARMA (7, 7) model, where all the coeffcents are sgnfcant at the 95% confdence nterval. The selected ARMA model s presented n equaton (1) below: Yˆ Y 0.293Y 0.511Y 0.321Y (1) t t1 t2 t4 t7 t1 t2 t4 t7 where Yˆt s the forecasted percentage change of the US unemployment rate. 4.2 Neural Networks (NNs) Neural networks exst n several forms n the lterature. The most popular archtecture s the Mult-Layer Perceptron (MLP). A standard neural network has at least three layers. The frst layer s called the nput layer (the number of ts nodes corresponds to the number of explanatory varables). The last layer s called the output layer (the number of ts nodes corresponds to the number of response varables). An ntermedary layer of nodes, the hdden layer, separates the nput from the output layer. Its number of nodes defnes the amount of complexty the model s capable of fttng. In addton, the nput and hdden layer contan an extra node called the bas node. Ths node has a fxed value of one and has the same functon as the ntercept n tradtonal regresson models. Normally, each node of one layer has connectons to all the other nodes of the next layer.

6 The network processes nformaton as follows: the nput nodes contan the value of the explanatory varables. Snce each node connecton represents a weght factor, the nformaton reaches a sngle hdden layer node as the weghted sum of ts nputs. Each node of the hdden layer passes the nformaton through a nonlnear actvaton functon and passes t on to the output layer f the calculated value s above a threshold. The tranng of the network (whch s the adjustment of ts weghts n the way that the network maps the nput value of the tranng data to the correspondng output value) starts wth randomly chosen weghts and proceeds by applyng a learnng algorthm called backpropagaton of errors [18].The learnng algorthm smply tres to fnd those weghts whch mnmze an error functon (normally the sum of all squared dfferences between target and actual values). Snce networks wth suffcent hdden nodes are able to learn the tranng data (as well as ther outlers and ther nose) by heart, t s crucal to stop the tranng procedure at the rght tme to prevent overfttng (ths s called early stoppng ). Ths can be acheved by dvdng the dataset nto three subsets respectvely called the tranng and test sets used for smulatng the data currently avalable to ft and tune the model and the valdaton set used for smulatng future values. The tranng of a network s stopped when the mean squared forecasted error s at mnmum n the test-sub perod. The network parameters are then estmated by fttng the tranng data usng the above mentoned teratve procedure (backpropagaton of errors). The teraton length s optmsed by maxmsng the forecastng accuracy for the test dataset. Then the predctve value of the model s evaluated applyng t to the valdaton dataset (out-of-sample dataset) The Mult-Layer Perceptron Model (MLP) MLPs are feed-forward layered NN, traned wth a back-propagaton algorthm. Accordng to Kaastra and Boyd [10], they are the most commonly used types of artfcal networks n fnancal tme-seres forecastng. The tranng of the MLP network s processed on a three-layered archtecture, as descrbed above The Recurrent Neural Network (RNN) The next NN archtecture used n ths paper s the RNN. For an exact specfcaton of recurrent networks, see Elman [5]. A smple recurrent network has an actvaton feedback whch embodes short-term memory. The advantages of usng recurrent networks over feed-forward networks for modelng non-lnear tme seres have been well documented n the past. However, as mentoned by Tent [21], the man dsadvantage of RNNs s that they requre substantally more connectons, and more memory n smulaton than standard back-propagaton networks (p. 569), thus resultng n a substantal ncrease n computatonal tme. 5

7 4.2.3 The Ps-Sgma Neural Network (PSN) The PSNs are a class of Hgher Order Neural Networks wth a fully connected feedforward structure. Ghosh and Shn [6] were the frst to ntroduce the PSN, tryng to reduce the numbers of weghts and connectons of a Hgher Order Neural Network. Ther goal was to combne the fast learnng property of sngle-layer networks wth the mappng ablty of Hgher Order Neural Networks and avod ncreasng the requred number of weghts. The prce for the flexblty and speed of Ps Sgma networks s that they are not unversal approxmators. We need to choose a sutable order of approxmaton (or else the number of hdden unts) by consderng the estmated functon complexty, amount of data and amount of nose present. To overcome ths, our code runs smulatons for orders two to sx and then t presents the best network. The evaluaton of the PSN model selected comes n terms of tradng performance. 2 5 Forecast Combnaton Technques 5.1 Kalman Flter Kalman Flter s an effcent recursve flter that estmates the state of a dynamc system from a seres of ncomplete and nosy measurements. The tme-varyng coeffcent combnaton forecast suggested n ths paper s shown below: 3 Measurement Equaton: t t t 2 f a f t, t ~ NID 0, c NNs (2) 1 t t1 2 State Equaton: a a n, n ~ NID(0, ) (3) t t n Where: t f s the dependent varable (combnaton forecast) at tme t c NNs t f ( 1, 2,3) are the ndependent varables (ndvdual forecasts) at tme t t a ( 1, 2,3) are the tme-varyng coeffcents at tme t for each NN ε t,n t are the uncorrelated error terms (nose) 0 The alphas are calculated by a smple random walk and we ntalzed 1. Based on the above, our Kalman Flter model has as a fnal state the followng: t t t t f f f f cnns MLP RNN PSN t (4) 2 For a complete descrpton of all the neural network models we used and ther complete specfcatons see Sermpns et al. [17].

8 From the above equaton we note that the Kalman flterng process also favors the PSN model, whch s the model that performs best ndvdually. 5.1 Support Vector Regresson (SVR) Vapnk [24] establshed Support Vector Regresson (SVR) as a robust technque for constructng data-drven and non-lnear emprcal regresson models. They provde global and unque solutons and do not suffer from multple local mnma (Suykens [19]). They also present a remarkable ablty of balancng model accuracy and model complexty (and Lu et al.[13]). The SVR functon can be specfed as: f ( x) w T ( x) b (5) where w and b are the regresson parameter vectors of the functon and φ(x) s the non-lnear functon that maps the nput data vector x nto a feature space where the tranng data exhbt lnearty. The ε-senstve loss Lε functon fnds the predcted ponts that le wthn the tube created by two slack varables, 0 f y f ( x) L ( x ), y f ( x) f other (6) In other words ε s the degree of model nose nsenstvty and L ε fnds the predcted values that have at most ε devatons from the actual obtaned values y. The goal s to solve the followng argument 3 : n 0 2 Mnmze * 1 C( ) w subject to * 0 and 1 2 C 0 T y w ( x ) b T * (7) w ( x ) b y The above quadratc optmzaton problem s transformed n a dual problem and ts * a, soluton s based on the ntroducton of two Lagrange multplers a and mappng wth a kernel functon K( x, x ) : n * ( ) ( ) (, ) 1 f x a a K x x b * where 0 a, a C (8) Support Vectors (SVs) are called all the x that contrbute to equaton (8), thus they le outsde the ε-tube, whereas non-svs le wthn the ε-tube. Increasng ε leads * : 3 For a detaled mathematcal analyss of the SVR soluton see Vapnk [24]. 7

9 to less SVs selecton, whereas decreasng t results to more flat estmates. The norm term w 2 characterzes the complexty (flatness) of the model and the term n * ( ) 1 s the tranng error, as specfed by the slack varables. Consequently the ntroducton of the parameter C satsfes the need to trade model complexty for tranng error and vce versa (Cherkassky and Ma [2]). In our applcaton, the NN forecasts are used as nputs for a ε-svr smulaton. A RBF kernel4 s selected and the parameters have been optmzed based on cross-valdaton n the n-sample dataset (ε=0.06, γ= 2.47 and C=0.103). 6 Statstcal Performance As t s standard n lterature, n order to evaluate statstcally our forecasts, the RMSE, the MAE, the MAPE and the Thel-U statstcs are computed (see Duns and Wllams [4]). For all four of the error statstcs retaned the lower the output, the better the forecastng accuracy of the model concerned. In Table 3 we present the statstcal performance of all our models n the n-sample perod. Table 3. Summary of the In-Sample Statstcal Performance ARMA MLP RNN PSN Kalman Flter We note that from our ndvdual forecasts, the PSN statstcally outperforms all other models. Both forecast combnaton technques mprove the forecastng accuracy, but SVR s the superor model regardng all four statstcal crtera. Table 4 below summarzes the statstcal performance of our models n the out-of-sample perod. SVR MAE MAPE 65.25% 52.78% 50.17% 47.73% 45.76% 41.52% RMSE Thel-U The RBF kernel equaton s K( x, x) exp( x x ), 0. 2

10 Table 4. Summary of the Out-of-sample Statstcal Performance ARMA MLP RNN PSN Kalman Flter SVR MAE MAPE 67.45% 50.17% 48.97% 44.38% 40.21% 34.33% RMSE Thel-U From the results above, t s obvous that the statstcal performance of the models n the out-of-sample perod s consstent wth the n-sample one and ther rankng remans the same. All NN models outperform the tradtonal ARMA model. In addton, the PSN outperforms sgnfcantly the MLP and RNN n terms of statstcal accuracy. The dea of combnng NN unemployment forecasts seems ndeed very promsng, snce both Kalman Flter and SVR present mproved statstcal accuracy also n the out-of-sample perod. Moreover, SVR confrms ts forecastng superorty over the ndvdual archtectures and the Kalman Flter technque. In other words, SVR s adaptve trade-off between model complexty and tranng error seems more effectve than the recursve ablty of Kalman Flter to estmate the state of our process. 7 Concludng Remarks The motvaton for ths paper s to nvestgate the effcency of a Neural Network (NN) archtecture, the Ps Sgma Network (PSN), n forecastng US unemployment and compare the utlty of Kalman Flter and Support Vector Regresson (SVR) n combnng NN forecasts. An Autoregressve Movng Average model (ARMA) and two dfferent NN archtectures, a Mult-Layer Perceptron (MLP) and a Recurrent Network (RNN), are used as benchmarks. The statstcal performance of our models s estmated throughout the perod of , usng the last seven years for out-ofsample testng. As t turns out, the PSN outperforms ts benchmark models n terms of statstcal accuracy. It s also shown that all the forecast combnaton approaches outperform n the out-of-sample perod all our sngle models. All NN models beat the tradtonal ARMA model. In addton, the PSN outperforms sgnfcantly the MLP and RNN n terms of statstcal accuracy. The dea of combnng NN unemployment forecasts seems ndeed very promsng, snce both Kalman Flter and SVR present mproved statstcal accuracy also n the out-of-sample perod. SVR confrms ts forecastng superorty over the ndvdual archtectures and the Kalman Flter technque. In other, SVR s adaptve trade-off between model complexty and tranng error seems more effectve than the recursve ablty of Kalman Flter to estmate the state of our process. The remarkable statstcal performance of SVR allows us to conclude that t can be consdered as an optmal forecast combnaton for the models and tme seres under 9

11 study. Fnally, the results confrm the exstng lterature, whch suggests that nonlnear models, such as NNs, can be used to model macroeconomc seres. References 1. Bates, J. M., Granger, C. W. J.: The Combnaton of Forecasts. Operatonal Research Socety. 20, (1969) 2. Cherkassky, V., Ma, Y.: Practcal selecton of SVM parameters and nose estmaton for SVM regresson. Neural Networks. 17, (2004) 3. Deutsch, M., Granger, C.W. J., Teräsvrta, T.: The combnaton of forecasts usng changng weghts. Internatonal Journal of Forecastng. 10, (1994) 4. Duns, C. L., Wllams, M.: Modellng and Tradng the EUR/USD Exchange Rate: Do Neural Network Models Perform Better?. Dervatves Use, Tradng and Regulaton. 8, (2002) 5. Elman, J. L.: Fndng Structure n Tme. Cogntve Scence. 14, (1990) 6. Ghosh, J., Shn, Y.: The P-Sgma Network: An effcent Hgher-order Neural Networks for Pattern Classfcaton and Functon Approxmaton. In: Proceedngs of Internatonal Jont Conference of Neural Networks, pp (1991) 7. Hamlton, J. D.: Tme seres analyss. Prnceton Unversty Press, Prnceton (1994) 8. Harvey, A. C.: Forecastng, structural tme seres models and the Kalman flter. Cambrdge Unversty Press, Cambrdge (1990) 9. Johnes, G.: Forecastng unemployment. Appled Economcs Letters. 6, (1999) 10. Kaastra, I., Boyd, M.: Desgnng a Neural Network for Forecastng Fnancal and Economc Tme Seres. Neurocomputng. 10, (1996) 11. Koop, G., Potter, S.M.: Dynamc Asymmetres n U.S. Unemployment. Journal of Busness & Economc Statstcs. 17, (1999) 12. Lang, F.: Bayesan neural networks for nonlnear tme seres forecastng. Statstcs and Computng. 15, (2005) 13. Lu, C.J., Lee, T.S., Chu, C.C.: Fnancal tme seres forecastng usng ndependent component analyss and support vector regresson. Decson Support Systems. 47, (2009) 14. Montgomery, A.L., Zarnowtz, V., Tsay, R.S., Tao, G.C.: Forecastng the U.S. Unemployment Rate. Journal of the Amercan Statstcal Assocaton. 93, (1998). 15. Newbold, P., Granger, C. W. J.: Experence wth Forecastng Unvarate Tme Seres and the Combnaton of Forecasts. Journal of the Royal Statstcal Socety. 137, (1974). 16. Rothman, P.: Forecastng Asymmetrc Unemployment Rates. The Revew of Economcs and Statstcs. 80, (1998) 17. Sermpns, G., Laws, J., Duns, C.L.: Modellng and tradng the realsed volatlty of the FTSE100 futures wth hgher order neural networks. European Journal of Fnance (2012) 18. Shapro, A. F.: A Htchhker's gude to the technques of adaptve nonlnear models. Insurance: Mathematcs and Economcs. 26, (2000) 19. Suykens, J. A. K., Brabanter, J. D., Lukas, L., Vandewalle, L.: Weghted least squares support vector machnes: robustness and sparse approxmaton. Neurocomputng. 48, (2002)

12 20. Swanson, N. R., Whte, H.: A Model Selecton Approach to Real-Tme Macroeconomc Forecastng Usng Lnear Models and Artfcal Neural Networks. Revew of Economcs and Statstcs. 79, (1997) 21. Tent, P. : Forecastng foregn exchange rates usng recurrent neural networks. Appled Artfcal Intellgence. 10, (1996) 22. Teräsvrta, T., Van Djk, D., Mederos, M. C.: Lnear models, smooth transton autoregressons, and neural networks for forecastng macroeconomc tme seres: A reexamnaton. Internatonal Journal of Forecastng. 21, (2005) 23. Teru, N., Van Djk, H. K.: Combned forecasts from lnear and nonlnear tme seres models. Internatonal Journal of Forecastng. 18, (2002). 24. Vapnk, V. N.: The nature of statstcal learnng theory. Sprnger (1995) 11