GP Algorithm versus Hybrid and Mixed Neural Networks

Transcription

1 CENTRE FOR EMEA BANKING, FINANCE & ECONOMICS GP Algorihm versus Hybrid and Mixed Neural Neworks Andreas Karahanasopoulos Working Paper Series No 16/11

2 GP Algorihm versus Hybrid and Mixed Neural Neworks Andreas Karahanasopoulos London Meropolian Universiy Absrac In he curren paper we presen an inegraed geneic programming environmen, called java GP Modelling. The java GP Modelling environmen is an implemenaion of he seady-sae geneic programming algorihm. Tha algorihm evolves ree based srucures ha represen models of inpu oupu relaion of a sysem. The moivaion of his paper is o compare he GP algorihm wih neural nework archiecures when applied o he ask of forecasing and rading he ASE 20 Greek Index using only auoregressive erms as inpus. This is done by benchmarking he forecasing performance of he GP algorihm and 6 differen ARMA-Neural Nework combinaion designs represening a Hybrid, Mixed Higher Order Neural Nework (HONN), a Hybrid, Mixed Recurren Nework (RNN), a Hybrid, Mixed classic Mulilayer Percepron (MLP) wih some radiional echniques, eiher saisical such as a an auoregressive moving average model (ARMA), or echnical such as a moving average convergence/divergence model (MACD), plus a naïve rading sraegy. More specifically, he rading performance of all models is invesigaed in a forecas and rading simulaion on ASE 20 ime series closing prices over he period using he las one and a half years for ou-of-sample esing. We use he ASE 20 daily series as many financial insiuions are ready o rade a his level and i is herefore possible o leave orders wih a bank for business o be ransaced on ha basis. As i urns ou, he GP model does remarkably well and ouperforms all oher models in a simple rading simulaion exercise. This is also he case when more sophisicaed rading sraegies using confirmaion filers and leverage are applied, as he GP model sill produces beer resuls and ouperforms all oher neural nework and radiional saisical models in erms of annualised reurn.

3 1. INTRODUCTION The use of arificial inelligence for he purpose of forecasing marke movemens has been widely reviewed in academia. This sudy is a comparaive analysis of he resuls yielded from uilizing a Geneic Programming Algorihm and various radiional Neural Nework compuing echniques when forecasing he Greek sock marke. Addiionally, we endeavour o develop more accurae and sophisicaed echniques in order o increase he performance of our rading simulaion. Due o he convergence and unificaion of global financial markes in recen years, his forecasing ask has become increasingly challenging. Furhermore, radiional economeric mehods on which forecasers have previously been relian no longer saisfy he demands of marke paricipans as hey sruggle o capure inegraing feaures associaed wih oday s markes. As discussed by Lisboa e al (2000), neural neworks are an emergen echnology wih an increasing number of real-world applicaions offering a unique aspec o he world of financial forecasing. Neverheless, some praciioners have ained he virues of neural neworks wih scepicism criicising heir capaciy o forecas and highlighing heir limiaions. Hence, his paper invesigaes a new, conemporary and more proficien mehod of forecasing ha is capable of idenifying and dealing wih disconinuiies, nonlineariies and high frequency muli-polynomial componens which are all prevalen in financial series of oday s markes. This model is mos commonly known as he Geneic Programming (GP) algorihm. GP Algorihms are domain-independen problem-solving echniques ha are run in various environmens. These environmens are srucured in a manner which approximaes problems in order o produce forecass a a high level of accuracy. GP can be caegorized in he forecasing bracke known in he finance world as Evoluionary Algorihms. The basis for his ype of problem solving echnique derives from he Darwinian principle of reproducion and survival of he fies. Addiionally, GP is also similar o he biological geneic operaions such as crossover and muaion. More imporanly, Koza (1990, 1992) sress ha GP addresses and quanifies complex issues as an auomaed process via programming, which enables compuers o process and solve problems. The Darwinian aspec of GP applies he heory of evoluion o a populaion of compuer programs of varying sizes and shapes. For insance, GP sars wih an iniial populaion of housands or even millions of randomly generaed compuer programs. These programs comprise of programmaic elemens buil o apply he fundamenal principles of biological evoluion in order o creae a new (and ofen improved) populaion of programs. As menioned previously, he creaion of his new populaion is generaed in a domainindependen sysem applying he Darwinian heory of naural selecion under he principal known as survival of he fies. An analogue of he naurallyoccurring geneic operaion of sexual recombinaion (crossover), and occasional muaion, he crossover operaion is designed o creae synacically valid offspring programs (given closure amongs he se of programmaic ingrediens). GP combines he expressive high-level symbolic represenaions of compuer programs wih he near-opimal efficiency of

4 learning of Holland s (1975) geneic algorihm in order o produce highly accurae oupus. Koza (1998) menioned ha a compuer program ha solves or a he very leas approximaes a given problem ofen emerges from his process. Dissimilar o oher models such as neural neworks, GP does no require any prior knowledge of a model s srucure for he purpose of sysem modelling. Alernaively, GP evolves a sysem model wih parameer values ha bes fi specific daa wihou manipulaing he daa o fi predefined model srucures as many oher preceding forecasing mehods end o do. In oher words, GP creaes an iniial populaion of models and evolves using geneic operaors in order o calculae he mahemaical expression which bes fis he specified daa inpu ino he sysem. Furhermore, GP simulaneously searches for and refines a model s parameers and ulimaely is srucure. The moivaion for his paper is o invesigae he use of GP algorihm and several neural neworks echniques combined wih ARMA models in order o improve he forecasing performance using auoregressive erms as inpus. This is achieved by comparing six benchmark neural nework combined archiecures wih a forecas produced by he GP Algorihm. Mos noably, classic neural neworks as Mulilayer Percepron (MLP), Higher Order Neural Nework (HONN), Recurren Neural Nework (RNN), auoregressive moving average model (ARMA), or echnical models such as a moving average convergence/divergence model (MACD), plus a naïve rading sraegy are all reviewed as benchmark mehods. From he analysis i emerges ha he GP algorihm demonsraes a remarkable performance and ouperforms all oher models in a simple rading simulaion exercise. This is also rue when more sophisicaed rading sraegies are uilized wih he applicaion of confirmaion filers and leverages as GP sill demonsraes superior forecasing abiliy in erms of annualised reurn. I is worh menioning he second bes performance of he Hybrid HONNs and he Mixed HONNs. Dunis e al. (2010a, b) sress ha he combinaion of neural neworks can produce beer forecass compared wih alernaive echniques. Furhermore he Hybrid MLP and he Mixed MLP also perform well. Also, he RNNs which hisorically have performed remarkably well display less impressive forecasing poenial in his research. I is observed ha his migh be due o he fac ha hey have an inabiliy o provide accurae resuls when only auoregressive erms are used as inpus. The remainder of he paper is organised as follows. In secion 2, we presen he lieraure relevan o GP modelling, he Hybrid, Mixed Neural Neworks, he Recurren Neural Nework, he Higher Order Neural Neworks and he Mulilayer Percepon. Secion 3 describes he daase used for his research and is characerisics. An overview of he differen neural nework models, Geneic Programming algorihm and saisical echniques is given in secion 4. Secion 5 displays he empirical resuls of all he models considered and invesigaes he possibiliy of improving heir performance wih he applicaion of more sophisicaed rading sraegies. Ulimaely, Secion 6 provides some concluding remarks.

5 2. LITERATURE REVIEW The purpose of his invesigaion is o apply he GP algorihm o he ASE 20 Greek Sock marke daa comparing is resuls wih he mos promising new neural neworks archiecures combining hem wih auoregressive models (in our case he ARMA model) which have been developed recenly wih he purpose o overcome he numerous limiaions of he more classic neural archiecures and o assess wheher hey can achieve a higher performance in a rading simulaion GP was firs developed by Barricelli (1954) as evoluionary algorihms. Progressively ino he 1960 s and 1970 s hese evoluionary algorihms became more commonly known and recognized as opimizaion mehods. In paricular, Rechenberg (1971) and his research eam were able o solve complex engineering problems hrough he applicaion of opimizaion mehods as documened in his 1971 PhD hesis. Furhermore Holland (1975) was anoher influenial figure in he 1970 s However, Fogel e al. (1964) are among he earlies praciioners pioneering in GP mehodology. They apply evoluionary algorihms o he problem of discovering finie-sae auomaa. In he developmen of GP mehodology i was laer adaped o he Markov decision making process. More imporanly he firs evidence of GP as he ree based mehod ha we are familiar wih in modern financial forecasing was provided by Cramer (1985). More recenly, Cramer s work has been expanded furher by John R. Koza (1990), Koza (1992), Koza (1994), Koza (1998) and Koza e al. (1999, 2003) who apply hese mehodologies o complex opimizaion and search problems. Alhough GP has now been esablished as a credible and respeced echnique his was no always he case. For example in he 1990 s GP was considered incomprehensible. Ener he 2000 s and he heory of GP has seen progressive and formidable growh. This has paricularly been he case in he area of probabilisic models as GP has been incorporaed wih schema heories and Markov chain models. A variey of Geneic Programming applicaions is shown in he papers below: Wincler (2004), Wincler e al. (2004a, b), Madar e al. (2004, 2005), Willis e al. (1997), Tsang e al. (1998), Fukunaga and Secher (1998) and Werner and Fogary (2001). On he oher hand combining differen models can increase he chance o capure differen paerns in he daa and improve forecasing performance. Several empirical sudies have already suggesed ha by combining several differen models, forecasing accuracy can ofen be improved over he individual model. Using hybrid models or combining several models has become a common pracice o improve he forecasing accuracy since he well-known M-compeiion (Makridakis e al.(1982)) in which combinaion of forecass from more han one model ofen led o improved forecasing performance. The basic idea of he model combinaion in forecasing is o use each model s unique feaure o capure differen paerns in he daa. Boh heoreical and empirical findings sugges ha combining differen mehods can be an effecive and efficien way o improve forecass (Makridakis (1989), Newbold e al. (1974), Palm e al. (1992)). Research in ime series forecasing argues ha predicive performance improves in combined models. (Bishop

6 (1994), Clemen (1989), Hansen e al. (2003), Hibber e al. (2000), Terui e al. (2002), Tseng e al. (2002), Zhang, (2003), Zhang e al. (2005). RNNs have an acivaion feedback which embodies shor-erm memory allowing hem o learn exremely complex emporal paerns. Their superioriy agains feedfoward neworks when performing nonlinear ime series predicion is well documened in Connor e al. (1993) and Adam e al. (1994). In financial applicaions, Kamijo e al. (1990) applied hem successfully o he recogniion of sock paerns of he Tokyo sock exchange while Teni (1996) achieved remarkable resuls using RNNs o forecas he exchange rae of he Deusche Mark. Tino e al. (2001) use hem o rade successfully he volailiy of he DAX and he FTSE 100 using sraddles while Dunis and Huang (2002), using coninuous implied volailiy daa from he currency opions marke, obain remarkable resuls for heir GBP/USD and USD/JPY exchange rae volailiy rading simulaion. HONNs were firs inroduced by inroduced by Giles and Maxwell (1987) as a fas learning nework wih increased learning capabiliies. Alhough heir funcion approximaion superioriy over he more radiional archiecures is well documened in he lieraure (see among ohers Redding e al. (1993), Kosmaopoulos e al. (1995) and Psalis e al. (1998)), heir use in finance so far has been limied. This has changed when scieniss sared o invesigae no only he benefis of Neural Neworks (NNs) agains he more radiional saisical echniques bu also he differences beween he differen NNs model archiecures. Pracical applicaions have now verified he heoreical advanages of HONNs by demonsraing heir superior forecasing abiliy and pu hem in he fron line of research in financial forecasing. For example Dunis e al. (2006b) use hem o forecas successfully he gasoline crack spread while Fulcher e al. (2006) apply HONNs o forecas he AUD/USD exchange rae, achieving a 90% accuracy. However, Dunis e al. (2006a) show ha, in he case of he fuures spreads and for he period under review, he MLPs performed beer compared wih HONNs and recurren neural neworks. Moreover, Dunis e al. (2008a), who also sudy he EUR/USD series for a period of 10 years, demonsrae ha when mulivariae series are used as inpus he HONNs, RNN and MLP neworks have a similar forecasing power. Finally, Dunis e al. (2008b) in a paper wih a mehodology idenical o ha used in his research, demonsrae ha HONN and he MLP neworks are superior in forecasing he EUR/USD ECB fixing unil he end of 2007, compared o he RNN neworks, an ARMA model, a MACD and a naïve sraegy. 3. THE ASE 20 GREEK INDEX AND RELATED FINANCIAL DATA For Fuures on he FTSE/ASE-20 ha are raded in derivaives markes he underlying asse is he blue chip index FTSE/ASE-20. The FTSE/ASE-20 index is based on he 20 larges ASE socks. I was developed in 1997 by he parnership of ASE wih FTSE Inernaional and is he esablished benchmark.

7 I represens over 50% of ASE's oal capialisaion and currenly has a heavier weigh on banking, elecommunicaion and energy socks. The FTSE/ASE 20 index is raded as a fuures conrac ha is cash seled upon mauriy of he conrac wih he value of he index flucuaing on a daily basis. The cash selemen of his index is simply deermined by calculaing he difference beween he raded price and he closing price of he index on he expiraion day of he conrac. Furhermore, selemen is reached beween each of he paricipaing counerparies. Whils he fuures conrac is raded in index poins he moneary value of he conrac is calculaed by muliplying he fuures price by he muliple of 5 euros per poin. For example, a conrac rading a 1,400 poins is valued a 7,000 EUR. As a resul, our applicaion is deemed more realisic and specific o he series ha we invesigae in his paper 1. Name of Period Toal Daase Training Daase Ou- of- sample Daase(Validaion Se) Trading Days Beginning 21 January January Augus /2007 Table 1: The ASE 20 daase End 31 December Augus December 2008 Fig. 1: ASE 20 fixing prices (oal daase). The observed ASE 20 ime series is non-normal (Jarque-Bera saisics confirms his a he 99% confidence inerval) conaining sligh skewness and high kurosis. I is also non-saionary and we decided o ransform he ASE 20 series ino a saionary series of raes of reurn 2. Given he price level P 1, P 2,,P, he rae of reurn a ime is formed by: 1 We examine he ASE 20 since is firs rading day on 21 January 2001 (Greece s enrance in he European Moneary Zone), and unil 31 December 2008, using he coninuous daa available from DaaSream. 2 The percenage reurn is linearly addiive bu he log reurn is no linearly addiive across porfolio componens.

8 P = P 1 1 R [1] Series: RETURNS Sample Observaions 2087 Mean Median Maximum Minimum Sd. Dev Skewness Kurosis Jarque-Bera Probabiliy Fig. 2: ASE 20 reurns summary saisics (oal daase) As inpus o our GP algorihm and our neworks, based on he auocorrelaion funcion and some ARMA experimens we seleced 3 ses of auoregressive and moving average erms of he ASE 20 reurns and he 1-day Risk Merics volailiy series. Number Variable Lag 1 Ahens Composie all share reurn 1 2 Ahens Composie all share reurn 3 3 Ahens Composie all share reurn 6 4 Ahens Composie all share reurn 8 5 Ahens Composie all share reurn 10 6 Ahens Composie all share reurn 13 7 Ahens Composie all share reurn 14 8 Moving Average of he Ahens Composie all share reurn 15 9 Ahens Composie all share reurn Ahens Composie all share reurn Moving Average of he Ahens Composie all share reurn 19 Table 2: Explanaory variables for radiional Neural Neworks and he GP algorihm Number Variable Lag 1 Ahens Composie all share reurn 1 2 Ahens Composie all share reurn 3 3 Ahens Composie all share reurn 5 4 Ahens Composie all share reurn 7 5 Ahens Composie all share reurn 8 6 Ahens Composie all share reurn 9 7 Ahens Composie all share reurn 12 8 Ahens Composie all share reurn 13 9 Moving Average of he Ahens Composie all share reurn Ahens Composie all share reurn Ahens Composie all share reurn Moving Average of he Ahens Composie all share reurn day Riskmerics Volailiy 1 Table 3: Explanaory variables for he hybrid neural neworks

9 Number Variable Lag 1 Ahens Composie all share reurn 1 2 Ahens Composie all share reurn 2 3 Ahens Composie all share reurn 4 4 Ahens Composie all share reurn 5 5 Ahens Composie all share reurn 7 6 Ahens Composie all share reurn 9 7 Moving Average of he Ahens Composie all share reurn 10 8 Ahens Composie all share reurn 13 9 Ahens Composie all share reurn Ahens Composie all share reurn Moving Average of he Ahens Composie all share reurn Ahens Composie all share reurn 17 Table 4: Explanaory variables for he mixed neural neworks In order o rain he neural neworks and he GP algorihm we furher divided our daase as follows: Name of Period Trading Days Beginning End Toal Daase January December 2008 Training Daase January May2006 Tes Daase May Augus 2007 Ou-of- sample Daase (Validaion Se) Augus December 2008 Table 5: The neural neworks and GP algorihm daases 4. FORECASTING MODELS 4.1 Benchmark Models In his paper, we benchmark our neural nework models wih 3 radiional sraegies, namely an auoregressive moving average model (ARMA), a moving average convergence/divergence echnical model (MACD) and a naïve sraegy Naïve sraegy The naïve sraegy simply akes he mos recen period change as he bes predicion of he fuure change. The model is defined by: Y ˆ = [2] +1 Y Where Y is he acual rae of reurn a period Y ˆ +1 is he forecas rae of reurn for he nex period The performance of he sraegy is evaluaed in erms of rading performance via a simulaed rading sraegy Moving Average The moving average model is defined as:

10 M ( Y + Y + Y + Y ) n+ 1 = [3] n + Where M n Y is he moving average a ime is he number of erms in he moving average is he acual rae of reurn a period The MACD sraegy used is quie simple. Two moving average series are creaed wih differen moving average lenghs. The decision rule for aking posiions in he marke is sraighforward: If he shor-erm moving average inersecs he long-erm moving average from below a long posiion is aken. Conversely, if he long-erm moving average is inerseced from above a shor posiion is aken 3. The forecaser mus use judgemen when deermining he number of periods n on which o base he moving averages. The combinaion ha performed bes over he in-sample sub-period was reained for ou-of-sample evaluaion. The model seleced was a combinaion of he ASE 20 and is 7-day moving average, namely n = 1 and 7 respecively or a (1, 7) combinaion. The performance of his sraegy is evaluaed solely in erms of rading performance ARMA Model Auoregressive moving average models (ARMA) assume ha he value of a ime series depends on is previous values (he auoregressive componen) and on previous residual values (he moving average componen) 4. The ARMA model akes he form: Y = φ + φ Y + φ Y φ Y + ε w ε w ε 2... w ε p p q q [4] where Y Y 1, 2 is he dependen variable a ime Y, and Y p are he lagged dependen variable φ 0, φ 1, φ 2, and φ p are regression coefficiens ε ε 1, 2 is he residual erm ε, and ε p are previous values of he residual w 1, w 2, and w q are weighs. Using as a guide he correlogram in he raining and he es sub periods we have chosen a resriced ARMA (7, 7) model. All of is coefficiens are significan a he 99% confidence inerval. The null hypohesis ha all coefficiens (excep he consan) are no significanly differen from zero is rejeced a he 99% confidence inerval (see Appendix A1). The seleced ARMA model akes he form: 3 A long ASE 20 posiion means buying he index a he curren price, while a shor posiion means selling he index a he curren price. 4 For a full discussion on he procedure, refer o Box e al. (1994) or Pindyck and Rubinfeld (1998).

11 Y = Y Y Y ε ε ε -7 The model seleced was reained for ou-of-sample esimaion. The performance of he sraegy is evaluaed in erms of radiional forecasing accuracy and in erms of rading performance Various Neural Nework Archiecures Neural neworks exis in several forms in he lieraure. The mos popular archiecure is he Muli-Layer Percepron (MLP). A sandard neural nework has a leas hree layers. The firs layer is called he inpu layer (he number of is nodes corresponds o he number of explanaory variables). The las layer is called he oupu layer (he number of is nodes corresponds o he number of response variables). An inermediary layer of nodes, he hidden layer, separaes he inpu from he oupu layer. Is number of nodes defines he amoun of complexiy he model is capable of fiing. In addiion, he inpu and hidden layer conain an exra node, called he bias node. This node has a fixed value of one and has he same funcion as he inercep in radiional regression models. Normally, each node of one layer has connecions o all he oher nodes of he nex layer. The nework processes informaion as follows: he inpu nodes conain he value of he explanaory variables. Since each node connecion represens a weigh facor, he informaion reaches a single hidden layer node as he weighed sum of is inpus. Each node of he hidden layer passes he informaion hrough a nonlinear acivaion funcion and passes i on o he oupu layer if he calculaed value is above a hreshold. The raining of he nework (which is he adjusmen of is weighs in he way ha he nework maps he inpu value of he raining daa o he corresponding oupu value) sars wih randomly chosen weighs and proceeds by applying a learning algorihm called backpropagaion of errors 6 (Shapiro (2000)). The learning algorihm simply ries o find hose weighs which minimize an error funcion (normally he sum of all squared differences beween arge and acual values). Since neworks wih sufficien hidden nodes are able o learn he raining daa (as well as heir ouliers and heir noise) by hear, i is crucial o sop he raining procedure a he righ ime o preven overfiing (his is called early sopping ). This can be achieved by dividing he daase ino 3 subses respecively called he raining and es ses used for simulaing he daa currenly available o fi and une he model and he validaion se used for simulaing fuure values. The nework parameers are hen esimaed by fiing he raining daa using he above menioned ieraive procedure (backpropagaion of errors). The ieraion lengh is opimised by maximising he forecasing accuracy for he es daase. Our neworks, which are specially designed for financial purposes, will sop raining when he profi of our forecass in he es sub-period is [6] 5 Saisical measures are given in secion below. 6 Backpropagaion neworks are he mos common muli-layer neworks and are he mos commonly used ype in financial ime series forecasing (Kaasra and Boyd (1996)).

12 maximized. Then he predicive value of he model is evaluaed applying i o he validaion daase (ou-of-sample daase). There is a range of combinaion echniques ha can be applied o forecasing he aemp o overcome some deficiencies of single models. The combining mehod aims a reducing he risk of using an inappropriae model by combining several o reduce he risk of failure. Typically his is done because he underlying process canno easily be deermined (Hibon and Evgeniou (2005)). Combining mehods involves using several redundan models designed for he same funcion, where he diversiy of he componens is hough imporan (Brown e al. 2005). The procedure of making a hybrid or a mixed forecasing ime series model can be achieved by combining an ARMA process in order o learn he linear componen of he condiional mean paern hrough an arificial neural nework process designed o learn is nonlinear elemens. In summary, he proposed mehodologies of he hybrid and mixed sysem will be explained in he nex secion in figures 6 and The Muli-Layer Percepron Model Archiecure The nework archiecure of a sandard MLP looks as presened in figure 4 7 : [k ] x [ j] h y~ u jk w j MLP Fig. 4: A single oupu, fully conneced MLP model Where: [n] x ( n = 1,2, L, k + 1) are he model inpus (including he inpu bias node) a ime [m] h ( m = 1,2,..., j + 1 ) node) are he hidden nodes oupus (including he hidden bias y~ is he MLP model oupu 7 The bias nodes are no shown here for he sake of simpliciy.

13 u jk and w j are he nework weighs is he ransfer sigmoid funcion: S( x) x 1 = 1 + e, [6] The error funcion o be minimised is: E T 1 ( u jk wj ) = ( y ~ y ( u jk, wj ) = 1 is a linear funcion: ( x) = 2 F [7],, wih y being he arge value [8] T i x i The Recurren Nework Archiecure Our nex model is he recurren neural nework. While a complee explanaion of RNN models is beyond he scope of his paper, we presen below a brief explanaion of he significan differences beween RNN and MLP archiecures. For an exac specificaion of he recurren nework, see Elman (1990). A simple recurren nework has acivaion feedback, which embodies shorerm memory. The advanages of using recurren neworks over feedforward neworks, for modelling non-linear ime series, has been well documened in he pas. However as described in Teni (1996) he main disadvanage of RNNs is ha hey require subsanially more connecions, and more memory in simulaion, han sandard backpropagaion neworks, hus resuling in a subsanial increase in compuaional ime. However having said his RNNs can yield beer resuls in comparison o simple MLPs due o he addiional memory inpus. A simple illusraion of he archiecure of an Elman RNN is presened below.

14 [1] x j [2] x j [3] x j [1] U j y~ [1] U j 1 [2] U j 1 [2] U j Fig. 5: Elman recurren neural nework archiecure wih wo nodes on he hidden layer Where: x ( = 1,2,, k + 1) [n] y~ [ f ] n L, d ( f = 1,2) and [ f ] [1] [2], u u are he model inpus (including he inpu bias node) a ime is he recurren model oupu [n] w ( = 1,2,, k + 1) n L are he nework weighs U ( f = 1,2) is he oupu of he hidden nodes a ime is he ransfer sigmoid funcion: S( x) x [9] 1 = 1 + e, The error funcion o be minimised is: is he linear oupu funcion: ( x) = F [10] T 1 2 E( d, w ) = ( y ~ y ( d, w )) [11] T = 1 In shor, he RNN archiecure can provide more accurae oupus because he inpus are (poenially) aken from all previous values (see inpus U and [2] Uj 1 in he figure above) The Higher Order Neural Nework Archiecure Higher Order Neural Neworks (HONNs) were firs inroduced by Giles and Maxwell (1987) and were called Tensor Neworks. Alhough he exen of heir use in finance has so far been limied, Knowles e al. (2009 page 52) i x i [1] j 1

15 show ha, wih shorer compuaional imes and limied inpu variables, he bes HONN models show a profi increase over he MLP of around 8% on he EUR/USD ime series (p. 7). For Zhang e al. (2002), a significan advanage of HONNs is ha HONN models are able o provide some raionale for he simulaions hey produce and hus can be regarded as open box raher han black box. HONNs are able o simulae higher frequency, higher order nonlinear daa, and consequenly provide superior simulaions compared o hose produced by ANNs (Arificial Neural Neworks) (p. 188). Furhermore HONNs clearly ouperform in erms of annualised reurn and his enables Dunis e al. (2008a) o conclude wih confidence over heir forecasing superioriy and heir sabiliy and robusness hrough ime. While hey have already experienced some success in he field of paern recogniion and associaive recall 8, HONNs have only sared recenly o be used in finance. The archiecure of a hree inpu second order HONN is shown below: Fig. 6: Lef, MLP wih hree inpus and wo hidden nodes; righ, second order HONN wih hree inpus Where: [n] x ( n = 1,2, L, k + 1) are he model inpus (including he inpu bias node) a ime y~ is he HONNs model oupu u are he nework weighs jk are he model inpus. 8 Associaive recall is he ac of associaing wo seemingly unrelaed eniies, such as smell and colour. For more informaion see Karayiannis e al. (1994).

16 S x = 1 + e ( ) x, [12] is he ransfer sigmoid funcion: The error funcion o be minimised is: E T 1 ( u jk, w j ) = ( y ~ y ( u jk )) T = 1 is a linear funcion: F( x) = 2,, wih y being he arge value HONNs use join acivaion funcions; his echnique reduces he need o esablish he relaionships beween inpus when raining. Furhermore his reduces he number of free weighs and means ha HONNS are faser o rain han even MLPs. However because he number of inpus can be very large for higher order archiecures, orders of 4 and over are rarely used. Anoher advanage of he reducion of free weighs means ha he problems of overfiing and local opima affecing he resuls of neural neworks can be largely avoided. For a complee descripion of HONNs see Knowles e al. (2005- page 52) THE HYBRID HONN, MLP AND RNN ARCHITECTURES i x [13] i [14] Fig. 7: The archiecure of a Hybrid ARMA - Neural Nework Model The mehodology we follow o consruc he Hybrid Neural Nework is divided ino 3 seps. In he firs sep we ake he residual from an ARMA model. In a second sep we forecas he ARMA residual wih our neural nework. In a hird sep we creae he hybrid model by adding he forecased reurns from he ARMA model wih he forecased residuals from he second sep THE MIXED HONN, MLP AND RNN ARCHITECTURES Original or ransformed daa ARMA model o exrac linear elemens in DGP* ARMA Forecased Reurns

17 Oupu Mixed ARMA-NNR forecas Neural Nework Regression Model Oher inpus Saved ARMA forecased reurns as NNR model inpu *DGP= Daa Generaing Process Fig. 8: The archiecure of a Mixed Neural Nework Model The mehodology we follow o consruc he mixed ARMA-NNR model is divided ino 2 seps. In he firs sep he ASE 20 index is modelled wih a radiional ARMA model. In he second sep he forecased reurns of he ARMA model are used as an inpu o he neural nework for forecasing he seleced ime series. 4.3 The Geneic Programming Algorihm For he purpose of our research, he GP applicaion is coded and implemened o evolve ree based srucures ha presen models (sub rees) of inpu oupu. In he design phase of our GP applicaion we focused primarily on execuion ime opimizaion as well as limiing he bloa effec. The bloa effec is similar o he issue of overfiing experienced in Neural Neworks however in our case we run a risk of coninuously increasing and expanding he ree size. This algorihm is run in a seady sae in ha a single member of he populaion is replaced a a ime. Furhermore, our GP applicaion reproduces newer models replacing he weaker ones in he populaion according o heir finess. Reasoning behind he decision o use a seady sae algorihm is jusified as hey hold a greaer selecion srengh and geneic drif over oher algorihms such as a ypical generaional GAs. Addiionally, seady sae algorihms also offer excepional muliprocessing capabiliies. In our applicaion of he geneic programming we uilize formulas o evolve algebraic expressions ha enable he analysis / opimizaion of resuls in a ree like srucure. This geneic ree srucure consiss of nodes (depiced as circles in he diagram below) which are essenially funcions ha perform acions wihin his srucure. Furhermore, hese funcions are in place o generae oupu signals. On he oher hand, he squares in he ree signify erminal funcions represening he end of a funcion once he mos superior sub ree (model) is achieved. For example, he below ree srucure (model) is characerized by he algebraic expression 4.0/x 1 (-1) + ln(x2(-2)). In his case here is one oupu and he erminal nodes are consan a 4. Addiionally, he

18 oupus are expressed by x 1 (-1) and x 2 (-2). In he execuion of he geneic algorihm i has o be undersood ha each individual in he populaion correspond o a single sub ree srucure. Each of hese sub rees are limied by he predefined maximum ree size se o 6 in our applicaion. Fig. 9, Example of a ree srucure Koza (1998) summarises he funcionaliy aspec of he GP algorihm in he following seps: (1) The generaion of an iniial populaion of randomly consruced models is developed wih each model being represened in a ree like srucure as discussed previously. Addiionally, he evoluionary algorihm represens each chromosome of he populaion as a ree of variable lengh (i.e. oal number of funcions and erminals) or a maximum deph of he model ree. The process of randomly reproducing each variable of he populaion is compleed once all of hese funcions of he ree are erminal symbols. However, unil he process is haled by hese erminal symbols hen he ree like srucure of chromosomes coninues o muliply (grow) wih each generaion as he populaion expands o no only include he parens bu also heir offspring. This is achieved by crossover and muaion operaors. On he whole, i also has o be undersood ha he majoriy of hese models produced in he iniial populaion are, in mos cases, unsaisfacory when esed for heir performance wih some individual models fiing beer han ohers. However, one of he virues offered by Geneic Programming is ha hey exploi and manipulae hese differences unil he bes fiing models, in erms of leas error, are produced. (2) Following his iniial generaion of randomly seleced models a random subse (sub ree) of he populaion is hen seleced for a ournamen. Hence his process is known as a ournamen selecion phase. This process (ournamen procedure) is essenially a selecion mechanism o decipher which individuals from he populaion are o be seleced for reproducion o develop he nex generaion.

19 (3) An evaluaion of he members of his subse is hen carried ou and assigned a finess value. As saed by Koza (1998) he finess cases are eiher seleced a random or in some srucured manner (e.g. a regular inervals). In our applicaion, as menioned briefly in he firs sep, he finess value is defined as he mean squared error (MSE) wih he lowes MSE being argeed as he bes. Furhermore, he finess may be measured in erms of he sum of he absolue value of he differences beween he oupu produced by he model and/or he desired oupu (i.e. he Minkowski disance) or, alernaively, he square roo of he sum of he squared errors (i.e. he Euclidean disance). (4) Following he esablishmen of finess values he ournamen winners are hen deermined. To reierae, he winners of his scenario are he models wih he lower MSE. (5) Having idenified he ournamen winners in he previous sep we hen proceed by exposing he models o wo geneic operaors known as muaions and crossovers. Boh operaors are discussed in more deail below: Muaion: This is he creaion of a new model ha is muaed randomly from an exising one as circled in he diagram below (1*). This one muaion poin is indiscriminaely chosen as an independen poin and he resuling sub-ree is o be omied. From his resuling sub-ree, anoher new sub-ree (2*) is hen reproduced using he same procedure ha was iniially implemened o creae he original random populaion. Alhough his was he procedure we implemened for muaion here are also a number of alernaive mehods ha are explored in oher research. Fig 10, Muaion ree srucure example Crossover: This operaor creaes wo new models from exising models by geneically recombining randomly chosen pars of hem. This is achieved by using he crossover operaion applied a a randomly chosen crossover poin wihin each model. Due o he fac ha enire sub-rees are swapped (from poin 1* o poin 2* and from poins 3* o 4*), he crossover operaion produces models as offsprings. Furhermore, he models are seleced based on heir finess and he crossover allocaes fuure rials o regions of he

20 search space whose models conain pars from superior models. As a full explanaion of crossovers is beyond he scope of his paper please refer o Koza (1992) for more deails. Fig. 11, Crossover family ree like srucure example (6) The populaion is hen alered wih he ournamen losers being replaced by he winners (superior) offspring. (7) Provided he erminaion crierion (depiced as he symbol? in he following flow of sages) is no reached, he algorihm reurns o sep 2 and hese seps are repeaed unil he predefined erminaion crierion for geneic programming is saisfied. In our sudy we have se he erminaion crierion o 100,000 a which poin he cycles are sopped and forecased resuls can be obained. (8) Ulimaely, his proocol produces he bes individual (model) of he populaion as a resul.

21 The generaion of an iniial populaion Evaluaion *? yes No Selecion Crossover Reproducion Muaion New Generaion End *noe: he symbol? is he erminaion crierion which ieraes or erminaes he procedure of GP. Fig. 12: The archiecure of Geneic Programming Algorihm 4.4 Seings for Geneic Programming Parameers (See Appendix A.4) The parameers used for he opimizaion of our individual models are defined in order o yield beer resuls and are specified as follows: 1. Populaion size 200. The populaion size is he oal number of randomly chosen models in our experimen. This number can be alered however in our specific case we found ha i was more beneficial (in erms of annualised reurns) o se he populaion o 200 individuals. Each individual model has a ree srucure composed of a se of funcions and erminals. In summary, every model is a mahemaical equaion which paricipaes in he program unil he GP produces he bes individual program.

22 2. Maximum ree deph 6. The maximum ree deph is he maximum lengh of each model (of each ree srucure). In neural neworks his is commonly known as hidden nodes. The deph depends on he funcions and erminals of each individual model. 3. Tournamen size 4. Tournamen size is he size of models in he subse. Through rial and error we found his o be he mos appropriae size. 4. Crossover rial 1. Crossover rial means he number of generaions ha we le he geneic programming algorihm o run. Crossover is achieved by creaing wo new offspring models for he new populaion by randomly recombining pars from wo seleced parens. In his experimen we have one crossover rial per generaion. 5. Muaion probabiliy The muaion probabiliy is he probabiliy ha can muae pars of individual models from an exising one. Specifically muaion is performed by randomly selecing a paren wih a probabiliy relaed o is finess, afer ha muaion randomly changes one or more genes represening par of he soluion i encodes. Due o he fac ha he populaion is 200 models, we use a relaively large probabiliy. The muaion probabiliy exends from an iniial 0.1 and finishes a EMPIRICAL TRADING SIMULATION RESULTS The rading performance of all he models considered in he validaion subse is presened in he able below. We choose he nework wih he higher profi in he es sub-period. Our rading sraegy applied is simple and idenical for all he models: go or say long when he forecas reurn is above zero and go or say shor when he forecas reurn is below zero. Appendix A.3 provides he performance of all he NNs and he GP Algorihm in he raining and he es sub-periods while Appendix A.5, A.4 and A.2 provide he characerisics of our models and he performance measures. The Hybrid-RNNs, Mixed- RNNs are rained wih gradien descen as for he Hybrid-MLPs and Mixed- MLPs. However, he increase in he number of weighs, as menioned before, makes he raining process exremely slow: o derive our resuls, we needed abou en imes he ime needed wih he Hybrid-MLPs and Mixed-MLPs. As shown in able 6 below, he Mixed-RNN, Hybrid-RNN have a slighly lower performance compared o he Hybrid-MLP, Hybrid-HONN, Mixed-MLP, Mixed- HONN and GP algorihm. NAIVE MACD ARMA MLP RNN HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 36.70% 38.12% 38.13% 38.11% 38.11% 38.10% Annualised Reurn (excluding coss) 11.42% 17.63% 7.68% 22.99% 22.51% 26.75% Maximum Drawdown % % % % % %

23 (excluding coss) Posiions Taken (annualised) Hybrid MLP Hybrid RNN HybridHONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 38.08% 38.09% 38.07% Annualised Reurn (excluding coss) 32.80% 30.72% 35.67% Maximum Drawdown (excluding coss) % % % Posiions Taken (annualised) Mixed MLP Mixed RNN Mixed HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 38.08% 38.09% 38.07% Annualised Reurn (excluding coss) 31.79% 29.63% 34.75% Maximum Drawdown (excluding coss) % % % Posiions Taken (annualised) GP Algorihm Informaion Raio (excluding coss) 1.03 Annualised Volailiy (excluding coss) 38.04% Annualised Reurn (excluding coss) 39.33% Maximum Drawdown (excluding coss) % Posiions Taken (annualised) 67 Table 6: Trading performance resuls We can see ha he GP algorihm performs significanly beer han he Hybrid-HONNs, Hybrid-MLPs, Mixed HONNs, Mixed MLPs Hybrid-RNNs, and he Mixed-RNNs wih similar sors of drawdowns, and significanly beer han he sandard neural nework archiecures. Up o now, we have presened he rading resuls of all our models wihou considering ransacion coss. Since some of our models rade quie ofen, aking ransacion coss ino accoun migh change he whole picure. Following Dunis e al. (2008a), we check for poenial improvemens o our models hrough he applicaion of confirmaion filers. Confirmaion filers are rading sraegies devised o filer ou hose rades wih expeced reurns below a hreshold d around zero. They sugges o go long when he forecas is above d and o go shor when he forecas is below d. I jus so happens ha he Mixed ARMA-Neural Nework models perform bes wihou any filer. This is also he case of he MLP and HONN models. Sill, he applicaion of confirmaion filers o he benchmark models and he RNN model could have led o hese models ouperforming he Mixed, MLP HONN models. This is no

24 he case however bu, in order o conserve space, hese resuls are no shown here bu hey are available from he auhors. 5.1 Transacion Coss According o he Ahens Sock Exchange, ransacion coss for financial insiuions and fund managers dealing a minimum of 143 conracs or 1 million Euros is 10 Euros per conrac (round rip). Dividing his ransacion cos of he 143 conracs by average size deal (1 million Euros) gives us an average ransacion cos for large players of 14 basis poins (1 basis poin=1/100 of 1%) or 0.14% per posiion. NAIVE MACD ARMA MLP RNN HONN Informaion Raio (excluding coss) Annualised Volailiy 36.70% 38.12% 38.13% 38.11% 38.11% 38.10% (excluding coss) Annualised Reurn 11.42% 17.63% 7.68% 22.99% 22.51% 26.75% (excluding coss) Maximum Drawdown % % % % % % (excluding coss) Posiions Taken (annualised) Transacion coss 15.47% 4.94% 9.36% 13.65% 19.11% 12.74% Annualised Reurn (including coss) -4.05% 12.69% -1.68% 9.35% 3.40% 14.01% Hybrid MLP Hybrid RNN Hybrid HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 38.08% 38.09% 38.07% Annualised Reurn (excluding coss) 32.80% 30.72% 35.67% Maximum Drawdown (excluding coss) % % % Posiions Taken (annualised) Transacion coss 12.22% 12.09% 12.22% Annualised Reurn (including coss) 20.58% 18.63% 23.45% Mixed MLP Mixed RNN Mixed HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 38.08% 38.09% 38.07% Annualised Reurn (excluding coss) 31.79% 29.63% 34.75% Maximum Drawdown (excluding coss) % % % Posiions Taken (annualised) Transacion coss 5.74% 7.98% 9.10%

25 Annualised Reurn (including coss) 26.05% 21.65% 25.65% GP Algorihm Informaion Raio (excluding coss) 1.03 Annualised Volailiy (excluding coss) 38.04% Annualised Reurn (excluding coss) 39.33% Maximum Drawdown (excluding coss) % Posiions Taken (annualised) 67 Transacion coss 9.40% Annualised Reurn (including coss) 29.93% Table 7: Ou-of-sample resuls wih ransacion coss We can see ha, afer ransacion coss, he GP algorihm model ouperforms all he oher sraegies based on he annualized ne reurn closely followed by he Mixed-MLP, he Mixed HONN and he Hybrid HONNs sraegy. On he oher hand, he naïve sraegy and he ARMA model produce negaive resuls afer ransacion coss are aken ino accoun. The HONN and MACD achieve decen reurns, ye well below hose produced by our bes models. 5.2 Leverage o Exploi High Sharpe Raios In order o furher improve he rading performance of our models we inroduce a level of confidence o our forecass, i.e. a leverage based on he es sub-period. For he naïve model, which presens a negaive reurn we do no apply leverage. The leverage facors applied are calculaed in such a way ha each model has a common volailiy of 20% 9 on he es daa se. The ransacion coss are calculaed by aking 0.14% per posiion ino accoun, while he cos of leverage (ineres paymens for he addiional capial) is calculaed a 4% p.a. (ha is 0.016% per rading day 10 ). Our final resuls are presened in able 8 below. NAIVE MACD ARMA MLP RNN HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) % 40.03% 38.13% 40.28% 40.21% 40.31% 9 Since mos of he models have a volailiy of abou 20%, we have chosen his level as our basis. The leverage facors reained are given in able The ineres coss are calculaed by considering a 4% ineres rae p.a. divided by 252 rading days. In realiy, leverage coss also apply during non-rading days so ha we should calculae he ineres coss using 360 days per year. Bu for he sake of simpliciy, we use he approximaion of 252 rading days o spread he leverage coss of non-rading days equally over he rading days. This approximaion prevens us from keeping rack of how many nonrading days we hold a posiion.

26 Annualised Reurn 11.42% 18.51% 7.68% 24.30% 23.75% 28.30% (excluding coss) Maximum Drawdown % % % % % % (excluding coss) Leverage Facor Posiions Taken (annualised) Transacion and leverage coss Annualised Reurn (including coss) % 4.94% 9.36% 13.65% 19.11% 12.74% -4.05% 13.57% -1.68% 10.65% 4.64% 15.56% Hybrid-MLP Hybrid-RNN Hybrid-HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 40.14% 40.30% 40.24% Annualised Reurn (excluding coss) 34.57% 32.50% 37.71% Maximum Drawdown (excluding coss) % % % Leverage Facor Posiions Taken (annualised) Transacion and leverage coss 12.22% 12.1% 12.22% Annualised Reurn (including coss) 12.35% 20.4% Mixed-MLP Mixed-RNN Mixed-HONN Informaion Raio (excluding coss) Annualised Volailiy (excluding coss) 40.22% 40.22% 40.17% Annualised Reurn (excluding coss) 33.57% 31.29% 36.67% Maximum Drawdown (excluding coss) % % % Leverage Facor Posiions Taken (annualised) Transacion and leverage coss 6.052% 8.30% 9.40% Annualised Reurn (including coss) 27.51% 23.00% 27.27% GP Algorihm Informaion Raio (excluding coss) 1.03 Annualised Volailiy (excluding coss) 41.84% Annualised Reurn (excluding coss) 43.26% Maximum Drawdown (excluding coss) % Leverage Facor 1.10

27 Posiions Taken (annualised) 67 Transacion and leverage coss 9.95% Annualised Reurn (including coss) 33.34% Table 8: Trading performance - final resuls As can be seen from able 8, he GP algorihm coninues o demonsrae a superior rading performance despie significan drawdowns. The Mixed HONN, he Mixed MLP and he Hybrid HONN sraegies also perform well and presens high annualised reurns. In general, we observe ha all models are able o gain exra profis from he leverage as he increased ransacion coss counered by increased performance Again i is worh menioning, ha he ime needed o rain he HONN, he Hybrid-HONN and he Mixed-HONN nework was considerably shorer compared wih ha needed for he MLP, Hybrid-MLP, Mixed-MLP, RNN, Mixed-RNN and he Hybrid-RNN neworks. 6. CONCLUDING REMARKS In his paper, we apply a Geneic Programming algorihm, Muli-layer Percepron, Recurren, Higher Order, Mixed-Mulilayer Percepron, Mixed- Recurren, Mixed-Higher Order neural neworks, Hybrid-Mulilayer Percepron, Hybrid-Recurren, Hybrid-Higher Order neural neworks o a one-day-ahead forecasing and rading ask of he ASE 20 fixing series wih only auoregressive erms as inpus. We use a naïve sraegy, a MACD and an ARMA model as benchmarks. We develop hese differen predicion models over he period January Augus 2007 and validae heir ou-of-sample rading efficiency over he following period from Sepember 2007 hrough December The GP algorihm demonsraes a higher rading performance in erms of annualised reurn and informaion raio before ransacion coss. When more elaborae rading sraegies are applied and ransacion coss are considered he GP algorihm again coninues o ouperform all oher models achieving he highes annualised reurn. The Mixed-HONNs, he Mixed-RNNs and he Hybrid-HONNs models perform remarkably as well and seem o have abiliy in providing good forecass when auoregressive series are only used as inpus. I is also imporan o noe ha he Mixed-MLP nework which presens a very close second bes performance needs less raining ime han he GP algorihm, a much desirable feaure in a real-life quaniaive invesmen and rading environmen. In he circumsances, our resuls should go some way owards convincing a growing number of quaniaive fund managers o experimen beyond he bounds of radiional saisical and neural nework models. In paricular, he sraegies consising of modelling in a firs sage he linear componen of a financial ime series and hen applying a neural nework o learn is nonlinear elemens and he use of Geneic Programming appear quie promising.