Wavelet Multi-Layer Perceptron Neural Network for Time-Series Prediction

Transcription

1 Wavelet Mult-Layer Perceptron Neural Network for Tme-Seres Predcton Kok Keong Teo, Lpo Wang* and Zhpng Ln School of Electrcal and Electronc Engneerng Nanyang Technologcal Unversty Block S2, Nanyang Avenue Sngapore {p e, elpwang, *To whom all correspondence should be addressed Abstract. In ths paper, we nvestgate the effectveness of wavelet Mult-Layer Perceptrons (MLP) neural network for temporal sequence predcton. It s essentally a neural network wth nput sgnal decomposed to varous resolutons usng wavelet transform. Wavelet transform can expose the tme-frequency nformaton that s normally hdden. We show that wavelet MLP network provdes predcton performance comparable to the conventonal MLP. After the less mportant nputs are elmnated, the wavelet MLP shows more consstent performance for dfferent weght ntalzaton n comparson to the conventonal MLP. 1. Introducton In many nstances, the desre to predct the future s the drvng force behnd the search for laws to explan certan phenomena. Example range from forecastng weather and Newton s laws of moton. The oldest and most studed method, a lnear autoregresson (AR) s to ft the data usng the followng [1]: where y(k) a() e(k) ^ y ( k ) y( k) = T = 1 a( ) y( k ) + e( k) ^ ( = y k) + e( k). actual value of the tme seres weghtage predcton error predcted value of y(k) Ths AR model forms y(k) as a weghted sum of past values of the sequence. Ths model can provde good performance only when the system under nvestgaton s lnear or nearly lnear. However the performance may be very poor for cases n whch system dynamcs s hghly nonlnear. Neural network has demonstrated great potental for tme-seres predcton where system dynamcs s nonlnear. Lapedes and Farber [2] frst proposed usng MLP for nonlnear sgnal predcton. It led to an explosve ncrease n research actvtes n examnng the approxmaton capabltes of MLP [3]-[6]. (1)

2 Neural networks are developed to emulate the human bran that s powerful, flexble and effcent. However, conventonal networks only process the sgnal on ts fnest resoluton. It s however not the case for human bran. For example, the retnal mage s lkely to be processed n separate frequency channels [8]. The ntroducton of wavelet decomposton [7]-[11] provdes a new tool for approxmaton. Inspred by both the MLP and wavelet decomposton, Zhang and Benvenste [12] nvented a new type of network, call a wavelet network. Ths has caused rapd development of a new bred of neural network model ntegrated wth wavelets. Most researchers used wavelets as radal bass functons that allow herarchcal, mult-resoluton learnng of nput-output maps from expermental data [13]-[16]. Lang and Page [17] proposed a new learnng concept and paradgm for neural network, called multresoluton learnng based on multresoluton analyss n wavelet theory. In ths paper, we use wavelets to break the sgnal down nto ts multresoluton components before feedng them nto a MLP. We show that the wavelet MLP neural network s capable of utlzng the tme-frequency nformaton to mprove ts consstency n performance. 2. WAVELET Wavelet theory provdes a unfed framework for a number of technques that had been developed ndependently for varous sgnals processng applcaton, e.g., multresoluton sgnal processng used n computer vson; subband codng, developed for speech and mage compresson; and wavelet seres expansons, developed n appled mathematcs. In ths secton, we wll concentrate on the multresoluton approxmaton that s utlzed n the presented model. 2.1 Multresoluton [10][11] Wavelet ψ can be constructed such that the dlated and translated famly j j { j, ( t) = 2 ψ (2 ( t )) } 2 ψ (2) ( j, ) Ζ where (mother wavelet) s an orthonormal bass of L 2 (R), where L 2 (R) denote the ψ vector space of square-ntegrable, one-dmensonal functon f(x), and let V j denote a close subspace n L 2 (R). Orthogonal wavelets dlated by 2 j carry sgnal varatons at the resoluton 2 j. Thus wavelet can be used to compute the approxmaton of sgnal at varous resolutons wth orthogonal projectons on dfferent spaces {V j } j Z. Each subspace contans the approxmaton of all functon f(x) at resoluton 2 j. The approxmaton of sgnal at resoluton 2 j+1 contans all nformaton necessary to compute the sgnal at the lower resoluton. Thus they are a set of nested vector subspace, V V V (3) j j + 1 j + 2 Therefore when computng the approxmaton of functon f at resoluton 2 j, some nformaton about f s lost. As the resoluton ncreases to nfnty, the approxmate

3 sgnal converges to the orgnal sgnal. When the resoluton approaches zero, the sgnal vanshes. If P vj denotes the orthogonal projecton operator from L 2 (R) onto V j lm P f = j V j On the other hand when the resoluton 2 j approaches +, the sgnal approxmaton converges to the orgnal sgnal: lm f P f = j + V J (5) guarantee that the orgnal sgnal can be reconstructed usng decomposed sgnals at lower resoluton. 0 0 (4) (5) 2.2 Sgnal Decomposton A tree algorthm can be used for computng wavelet transform by usng the wavelet coeffcents as flter coeffcents. Assume that vector s m represents the sampled sgnal f at fnest resoluton 2 m. Lowpass flter L s employed to produce a coarser approxmaton at resoluton 2 m-1. Thus s j-1 =Ls j j=1,2,.,m (6) The detal sgnal d j at resoluton 2 j s obtaned by applyng a hghpass flter H to s j. That s d j-1 =Hs j j=1,2,,m (7) Thus the process can be repeated to produce sgnals at any desred resoluton (Fg.1). Fg. 1. Decomposton Process The sgnal can be reconstructed usng two synthess flter L * and H * (the transposed matrces of L and H, respectvely). Thus reconstructon s gven by (Fg. 2). s j =L * s j-1 +H * d j-1 (8) Fg. 2. Reconstructon Therefore, any orgnal sgnal can be represented as m m 1 m f = s = s + d + d + L + d + d (9)

4 3. WAVELET MLP NEURAL NETWORK Fg.3 shows the Wavelet MLP Neural Network used n ths paper. The nput sgnal s passed through a tapped delay lne to create short-term memory that retans aspect of the nput sequence relevant to makng predctons. Ths s smlar to tme lagged MLP except that the delayed data s not sent drectly nto the network. Instead t s decomposed by wavelet transform to form the nput of the MLP. Fg. 4 shows an example of two level decomposton of the tapped delay data x. Data x s decomposed to coarser (CA1) and detaled (CD1) approxmaton. The coarser (CA1) s further decomposed nto t coarser (CA2) and detaled (CD2) approxmatons. Fg. 3. Model of Network (WD=Wavelet Decomposton Furthermore, we are lookng nto possblty of dscardng certan waveletdecomposed data that s of lttle use n the mappng of nput to output. The mappng s expected to be hghly nonlnear and dependent on the characterstc of ndvdual sgnal. Let n (10) s = w j j represent the mportance of nput x where w j weghtage of the of nput to neuron j n number of hdden neurons ' ' s (11) s = max( s ) serves as ndcator of the relatve mportance of nupt x. where ' s normalzed nput strength max(s ) maxmum of s 1, s 2,. s I, I s the number of nputs

5 ' Input pont havng small s wll be consdered to be trval and maybe dscarded wthout affectng the predcton performance. Fg. 4. The two level decomposton to form nput to the neural network (a) Fg. 5. (a) the frst 8 data ponts of Mackey-Glass (b) decomposed by Daubeches1 wavelet (b) 4. Smulaton The Mackey-Glass tme-seres predcton s frequently used as a benchmark n tmeseres studes. The Mackey-Glass tme-delay dfferental equaton s defned by dx( t) 0.2x( t π ) (12) = 0.1x( t) 10 dt (1 + x( t π )) The MLP used n our smulatons conssts of a nput layer, a hdden layer of two neurons and one output neuron, and s tran by backpropagaton algorthm usng a Levenberg-Marquardt for fast optmzaton [18]. All neurons use conventonal sgmod actvaton functon; however, the output neuron employed a lnear actvaton functon as frequently used n forecastng applcatons. In order to compare our result, the normalzed mean squared error (NMSE) s used to assess forecastng performance. The NMSE s computed as where NMSE N 1 1 ^ (13) 2 = [ x( t) x( t)] 2 σ N t= 1

6 x(t) actual value of the tme seres ^ x ( t ) predcted value of x(t); 2 σ varance of the tme seres over the predctng duraton. N s the number of elements The data s dvded nto three sectons, the tranng, valdaton and testng. The tranng data s of length 220, follow by valdaton and testng data, each of length 30. Valdaton NMSE s evaluated every 20 epochs. When there s an ncrease n the valdaton NMSE, tranng stops. Test data s used to test the generalzaton performance of the network and has not been seen by the network durng tranng or valdaton. Early stoppng by montorng valdaton error often shows multple mnma as a functon of tranng tme and results are also senstve to the weght ntalzaton [6]. In order to have a far comparson; smulaton s carred out for each network wth dfferent random weght ntalzaton over 100 trals. The 50 lowest NMSE s kept for calculatons of mean and standard devaton, whch are then used for comparsons. Fg. 6. Dstrbuton of relatve mportance of 20 nputs for the wavelet MLP network wth decomposton level one n one of the smulatons, whch s smlar to the results n other smulatons The smulatons ndcate that the nput ponts 1,4 and 5 s consstently less mportant than other nputs (Fg.6). Smulatons are re-run after these less mportant nputs are elmnated. Ths results a network of sze 17:2:1 (17 nputs, 2 hdden neurons and 1 output neuron). We denote ths wavelet MLP Neural Network (FNN) by (17:2:1) = (20:2:1) - [1,4,5]. Smulatons are done on other network szes to reduce the number of nputs when possble. Table 1. Result of the three networks on dfferent archtecture of network Normalzed Mean Square Error (NMSE) Archtecture Type Mean Standard Mnmum Devaton Conventonal MLP :2:1 Wavelet MLP Conventonal MLP :2:1 Wavelet MLP x (20:2:1)-[1,4,5] x

7 Conventonal MLP :2:1 Wavelet MLP Conventonal MLP :2:1 Wavelet MLP (12:2:1)-[7] Table 1 shows that wavelet MLP network provdes predcton performance comparable to the conventonal MLP. After less mportant nputs are elmnated, the wavelet MLP shows more consstent performance for dfferent weght ntalzaton to the conventonal MLP. (a) (b) Fg. 7. Result for network sze 17:2:1 on Mackey Glass tme seres predcton (a) MLP wth test and valdaton NMSE= and , respectvely. (b) Wavelet MLP wth decomposton level one, verfcaton and test NMSE= and , respectvely. (c) Wavelet MLP (17:2:1) = (20:2:1) -[1,4,5], test and valdaton NMSE= and , respectvely. Dotted lne s the actual data, whereas contnuous lne s the predcted data. (c) 5. CONCLUSION In ths paper, we used a wavelet MLP, consstng of a wavelet decomposton layer and a conventonal MLP, for tme seres predcton. We analyzed the relatve

8 mportance among the nput wavelets. After less mportant wavelets are elmnated, the modfed wavelet MLP network provdes a more consstent and stable network that s evdent n ts low mean and standard devaton for NMSE. Ths s n contrast to the conventonal MLP network that has large performance swng and s senstve to weght ntalzaton. However the wavelet MLP wthout nput elmnaton dd not show sgnfcance mprovement over the conventonal MLP. It s suspected that dfferent sgnals have dfferent tme-frequency compostons. Thus the decomposton level, type of wavelet or decomposton type may vary sgnfcantly wth sgnal. Therefore more work s requred to equp the network wth the ablty to adapt to dfferent sgnals wthout human nterventon. REFERENCE [1] C. Chatfeld, The Analyss of Tme Seres, An Introducton, Chapman & Hall 1989 Fourth Edton. [2] A. Lapedes and R. Farber, "Nonlnear sgnal processng usng neural network: Predcton and system modelng," Los Alamos Nat. Lab, Tech. Rep, LA-UR , 1987 [3] Smon Haykn; Neural Networks: A Comprehensve Foundaton, Macmllan College Publshng Company, Inc [4] Erc A. Wan, "Tme Seres Predcton by usng a connectonst network wth nternal delay lnes," n Tme Seres Predcton: Forecastng the Future and Understandng the Past, SFI Studes n the scences of complexty, Proc Vol XV, pp: , Addson-Wesley, 1993 [5] Mchael C. Mozer, "Neural Net Archtecture for Temporal Sequence Processng," n Tme Seres Predcton: Forecastng the Future and Understandng the Past, SFI Studes n the scences of complexty, Proc Vol XV, pp: ,Addson-Wesley,1993 [6] Andreas S. Wegend, Bernardo A. Huberman and Davd E. Rumlhart, " Predctng Sunspot and Exchange Rates wth Connectonst Networks," n Nonlnear Modelng and Forecastng, SFI studes n scences of complexty, Proc. Vol. XII, pp: , Addson-Wesley,1992 [7] Mallat, S.G, A Theory for Multresoluton Sgnal Decomposton: The Wavelet Representaton, IEEE Transactons on Pattern Recognton Vol: 11 No. 7, pp ,, July [8] Mallat, S.G, Multfrequency channel decompostons of mages and wavelet models, IEEE Transactons on Acoustcs, Speech and Sgnal Processng [see also IEEE Transactons on Sgnal Processng] Vol: 37 No. 12, pp ,, Dec [9] Vetterl, M.; Herley, C, Wavelets and flter banks: theory and desgn, IEEE Transactons on Sgnal Processng, Vol: 40 No.9, pp, Sept [10] Mallat, S.G, A Wavelet Tour of Sgnal Processng, Academc Press, 1998 [11] Glbert Strang and Truong Nguyen, Wavelet and Flter Banks, Wellesly-Cambrdge Press,1996 [12] Zhang, Q.; Benvenste, A. Wavelet networks," IEEE Transactons on, Neural Networks, Vol: 3 6, pp , Nov [13] Baksh R Baksh and George Stephanopoulos, "Wavelets as Bass for Localzed Learnng n a Mult- Resoluton Herarchy," IJCNN., Internatonal Jont Conference Neural Networks on Vol 2:, pp: , 1992 [14]Cuca, I.; Ware, J.A., Wavelet networks as an alternatve to neural networks, 6th Internatonal Conference on Emergng Technologes and Factory Automaton Proceedngs ETFA '97,pp: [15] Felpe Mguel Aparco Acosta and Jean-Marc Vesn, "Nonlnear Predcton of Tme Seres usng Radal Wavelet Networks," Tme-Frequency and Tme-Scale Analyss, Proceedngs of the IEEE-SP Internatonal Symposum, 1992, pp: [16] Chun-Ta Chen, Azm-Sadjad, M.R and Chunhua Yuan, " Sgnal Representaton Usng Adaptve Wavelet- Net," Internatonal Conference on Neural Networks 1997, Vol: 4,pp: [17] Yao Lang and Edward W. Page, "Multresoluton Learnng Paradgm and Sgnal Predcton," IEEE Transactons on Sgnal Processng, Vol: 45, No. 11, 1997 [18] Martn T. Hagan and Mohammad B. Menhaj, "Tranng Feedforward Networks wth the Marquardt Algorthm," IEEE Transactons on Neural Networks, Vol:5, No. 6, 1994