An Evolving Cascade System Based on A Set Of Neo-Fuzzy Nodes

Transcription

1 An Evolving Cascade System Based on A Set Of Neo-Fuzzy Nodes Zengbing Hu Scool of Educational Information Tecnology, Central Cina Normal University, Wuan, Cina zb@mail.ccnu.edu.cn Yevgeniy V. Bodyanskiy Karkiv National University of Radio Electronics, Karkiv, Ukraine, yevgeniy.bodyanskiy@nure.ua Oleksii K. Tyscenko and Olena O. Boiko Karkiv National University of Radio Electronics, Karkiv, Ukraine, leatis@gmail.com, olena.boiko@ukr.net Abstract Neo-fuzzy elements are used as nodes for an evolving cascade system. Te proposed system can tune bot its parameters and arcitecture in an online mode. It can be used for solving a wide range of Data Mining tasks (namely time series forecasting). Te evolving cascade system wit neo-fuzzy nodes can process rater large data sets wit ig speed and effectiveness. Index Terms Computational Intelligence, Macine Learning, Cascade System, Data Stream Processing, Neuro-Fuzzy System. I. INTRODUCTION Te task of time series forecasting (data sequences forecasting) is well studied nowadays. Tere are many matematical metods of different complexity tat can be used for solving tis task: spectral analysis, exponential smooting, regression, advanced intellectual systems, etc. In many real-world cases, analyzed time series are nonstationary, nonlinear, and usually contain unknown beavior trends, stocastic or caotic components. Tis obstacle complicates time series forecasting and makes te above mentioned systems less effective. To solve tis problem, nonlinear models based on matematical metods of Computational Intelligence [1-3] can be used. It sould be especially mentioned tat neuro-fuzzy systems [4-6] are widely used for tis type of tasks due to teir approximating and extrapolating properties, results interpretability, and learning abilities. Te most appropriate coice for non-stationary data processing is evolving connectionist systems [7-10]. Tese systems adjust not only teir synaptic weigts and parameters of membersip functions, but also teir arcitectures. Tere are many evolving systems tat are able to process data sets in an online mode. Most of tem are based on multilayer neuro-fuzzy systems. Te Takagi-Sugeno-Kang (TSK) fuzzy systems [11-12] and adaptive neuro-fuzzy inference systems (ANFIS) are te most popular and effective systems tat are used to solve suc tasks. But in some cases (e.g. wen a size of a data set is not sufficient for training) tey cannot rapidly tune teir parameters, so teir effectiveness can decrease. Te first solution for tis problem is to decompose an initial task into a set of simpler tasks, so tat te obtained system can solve a problem wit a data set at and regardless to its size. One of te most studied approaces based on tis principle is te Group Metod of Data Handling (GMDH) [13-14]. But in case of online data processing, te GMDH systems are not sufficiently effective. Tis problem can be solved by an evolving cascade model tat tunes bot its parameters as well as its arcitecture in an online mode. Generally speaking, one can use different types of neurons or oter more complicated systems as nodes in an evolving cascade system. For example, a compartmental R-neuron was introduced as a node of a cascade system [15, 16]. If a data set to be processed is large enoug, it seems reasonable to use neo-fuzzy neurons [17-19]. Te neo-fuzzy neuron is capable of finding a global minimum for a learning criterion in an online mode, it also as a ig learning speed and good approximating properties. It is also appropriate from te viewpoint of computational simplicity. Te remainder of tis paper is organized as follows: Section 2 describes an arcitecture of te evolving cascade system. Section 3 describes an arcitecture of te neo-fuzzy neuron as a node of te evolving cascade system. Section 4 presents several syntetic and real-world applications to be solved wit te elp of te proposed evolving cascade system. Conclusions and future work are given in te final section.

2 II. AN EVOLVING CASCADE MODEL An arcitecture of te evolving cascade model is sown in Fig. 1. Figure 1. An arcitecture of te evolving cascade model n -dimensional vector of input signals A 1 x k x1 k, x2 k,..., xn k is fed to te zero layer of te 2 system (ere k 1, 2,... is an index of te current discrete time). Te first idden layer contains c n nodes (eac one 1 as two inputs). Te outputs of te first idden layer form signals 2 yˆs k, s 1, 2,..., 0,5nn1 cn. Ten tese signals are fed to te selection block SB tat sorts te first layer nodes by some criteria (e.g. by te mean squared error ) so tat.... [1] ys k 1 Te outputs yˆ 1 * forms its output signal [1] [1] [1] yˆ1 * k yˆ 2 * k yˆ s * k 1 k and yˆ 2 * [2] ŷ k of te selection block are fed to a unique neuron of te second layer wic 1 k. Tis signal and te signal yˆ * k (an output signal of te selection block SB ) are fed to a node of te next layer. A process of te cascades increasing is continued until a required accuracy is obtained. III. THE NEO-FUZZY SYSTEM AS A NODE OF THE EVOLVING CASCADE SYSTEM Neo-fuzzy neurons were proposed by T. Yamakawa and co-autors [17-19]. Advantages of tis block are good approximating properties, computational simplicity, a ig learning speed, and ability of finding a global minimum for a learning criterion in an online mode. An arcitecture of te neo-fuzzy neuron as a node of te evolving cascade system is sown in Fig.2. 3 T Nonlinear synapses NS A and Figure 2. An arcitecture of te neo-fuzzy neuron NS B wic are structural elements of te neo-fuzzy neuron fulfill te Takagi Sugeno fuzzy inference of te zero order. A two-dimensional vector of input signals xk xak, xbk T is fed to te node s input. Te first layer of eac nonlinear synapse contains membersip functions. In [20], it was proposed to use te B-splines as membersip functions for te neo-fuzzy neuron. B-splines provide iger approximation quality. A B-spline of te q -t order as te form

3 q ia q ib x A x k B k 1if cia xa kci 1, A, for q 1 0oterwise xak cia q1 ia xa k ciq1, A cia ci q, A xak q1 i 1, AxAk for q 1 ciq, A ci 1, A i 1,..., q, 1if cib xb kc 1,, i B for q 1 0oterwise xbk cib q1 ib xk ciq1, B cib ci q, B xbk q1 i 1, BxBk for q 1 ciq, B ci 1, B i 1,..., q. were c ia, c ib are parameters tat define centers of te membersip functions. It sould be noticed tat wen q 2 one can get traditional triangular membersip functions, and wen q 4 one can get cubic splines, etc. Te B-splines meet te unity partitioning conditions pa xa k 1, p1 pb xb k 1 p1 tat allows to simplify te node s arcitecture excluding a normalization layer. So, te elements of te first layer compute membersip levels x k x k, p 1, 2,...,. pa A, pb B Te second layer contains synaptic weigts w ia, w ib tat are adjusted during a learning process. Te tird layer is formed by two summation units. It computes sums of te output signals of te second layer for eac nonlinear synapse NS and NS. Te outputs of te tird layer are values A B A pa pa A p1 B pb pb B p1 k w x k, k w x k Anoter summation unit sums up tese two signals in order to produce te output signal. ŷ k of te node

4 wpapa xa kwpbpb xb k ˆ A B y k k k p1 p1 Te expression (1) can be written in te form ŷ k T w k k were wk w1ak,..., wak, w1bk,..., wbk T (1), k 1A xa k,..., A xa k, 1B xb k,..., B xb k. To learn te neo-fuzzy neuron, we can use te procedure [21, 22] 1 T wk wk 1r k yk w k1kk (2) T rkrk1 kk,0 1 wic possesses bot filtering and tracking properties. It can be noticed tat wen 1 te procedure (2) coincides wit te Kaczmarz Widrow Hoff optimal gradient algoritm 1 T 1 T kk y k w k k w k w k k tat can be used if a training data set is non-stationary [23]. IV. EXPERIMENTS In order to prove te effectiveness of te proposed system, several simulation tests were implemented. Te system s effectiveness was analyzed by a value of te root mean square error () and data processing time. A. A nonlinear system A nonlinear system to be identified can be described by te equation yt 1 m i1 m were u sin(2 t/ 20), y 0, j 1,..., m, m 10. Te model is presented in te form t j i1 y ti y ti 2 u t1 yˆ f y, y,..., y, u t t1 t2 t10 t1 were y ˆt stands for a model s output. Te aim was to predict te next output using previous inputs and outputs. Tis data set contains 2500 points: 2000 points were selected for a training stage and 500 points were used for testing. To compare results, we used a multilayer perceptron (MLP), a radial-basis function neural network (RBFN), an adaptive neuro-fuzzy inference system (ANFIS), and te proposed evolving cascade system wit neo-fuzzy nodes. MLP was learnt during five epocs. A number of MLP s inputs was equal to 5 and a number of idden nodes was equal to 10. A total number of parameters to be tuned was equal to 51. In te RBFN s arcitecture, we used 3 inputs and 11 kernel functions, so a total number of parameters to be tuned in te RBFN s system was rougly equal to 56. One of te best results was sown by ANFIS, but it was processing data during 5 epocs and a little longer tan te proposed system, so it cannot process data in an online mode. ANFIS ad 3 inputs and 34 idden nodes. A total number of parameters to be adjusted was equal to 32. Te proposed evolving cascade system ad 3 inputs and 4 membersip functions in eac node. A total number of parameters to be adjusted was 54. Te proposed system demonstrated te best prediction results, and its data processing time was te best among te oters. A comparison of te systems results is sown in Table 1. TABLE I. COMPARISON OF THE SYSTEMS RESULTS T Systems Parameters to be tuned (training) (test) Time, s MLP RBFN

5 ANFIS Te proposed system Prediction results for te proposed system are in Fig.3 (a blue line represents signal s values, a magenta line represents prediction values, and a grey line represents prediction errors). Figure 3. Prediction results Figure 4. A pase portrait of te signal B. Internet traffic data (in bits) from an ISP Tis data set describes ourly traffic in te United Kingdom academic network backbone (taken from datamarket.com). It was collected between November 19 t, 2004, at 09:30 and January 27 t, 2005, at 11:11. Tis data set contains 1657 points: 1326 points were selected for a training stage and 331 points were used for testing. To compare results, a similar to te previous experiment set of systems was used but wit oter caracteristics. MLP was learnt during one epoc (tis case is similar to learning in an online mode). A number of MLP s inputs was equal to 3 and a number of idden nodes was equal to 8. A total number of parameters to be tuned was 31.

6 In te RBFN s arcitecture, 7 kernel functions and 3 inputs were used, so a total number of parameters to be adjusted in te RBFN system was rougly equal to tat of te proposed system. ANFIS ad 4 inputs and 55 idden nodes. It was processing data during 5 epocs. A total number of parameters to be tuned was 80. Tis system sowed te best prediction result, but it processed data longer tan te proposed system. Te proposed evolving cascade system ad 4 inputs and 4 membersip functions in eac node. A total number of parameters to be tuned was 36. Te proposed system sowed one of te best prediction results according to (te second result after ANFIS), and its data processing time was te best among te oters. A comparison of te systems results is sown in Table 2. TABLE II. COMPARISON OF THE SYSTEMS RESULTS Systems Parameters to be tuned (training) (test) Time, s MLP RBFN ANFIS Proposed system Prediction results for te proposed system are in Fig.5. Figure 5. Prediction results

7 Figure 6. A pase portrait of te signal C. Darwin Sea Level Pressures Tis data set was taken from researc.ics.aalto.fi. Tis data set describes montly values of te Darwin sea level pressure. It was collected between 1882 and Tis data set contains 1300 points: 1040 points were selected for a training stage and 260 points were used for testing. MLP was learnt during five epocs. It ad 3 inputs and 10 idden nodes. A total number of parameters to be adjusted was equal to 41. RBFN ad 7 kernel functions and 3 inputs, and a total number of parameters to be tuned in te RBFN system was equal to 36, i.e. it was very close to a number of parameters in te proposed system. A number of inputs for ANFIS was 4, a number of idden nodes was equal to 55. ANFIS was processing data during one epoc. A total number of parameters to be tuned was equal to 80. Tis system sowed te best result from te view point of accuracy, but it processed data longer tan te proposed system. Te proposed evolving cascade system ad 3 inputs and 4 membersip functions in eac node. A total number of parameters to be tuned was 40. Te proposed system sowed one of te best results, and its data processing time was te best. A comparison of te systems results is sown in Table 3. TABLE III. COMPARISON OF THE SYSTEMS RESULTS Systems Parameters to be tuned (training) (test) Time, s MLP RBFN ANFIS Proposed system Prediction results for te proposed system are in Fig.7.

8 Figure 7. Prediction results Figure 8. A pase portrait of te signal V. CONCLUSION An evolving cascade model based on te neo-fuzzy nodes is proposed. It can adjust bot its arcitecture in an online mode. Te proposed system as a rater simple computational implementation and can process data sets wit a ig speed. A number of experiments demonstrated tat tis evolving cascade system can forecast time series wit ig effectiveness. Te results may be successfully used in a wide class of Data Stream Mining and Dynamic Data Mining tasks. ACKNOWLEDGMENT Te autors would like to tank anonymous reviewers for teir careful reading of tis paper and for teir elpful comments. Tis scientific work was supported by RAMECS and CCNU16A REFERENCES [1] Du K-L, Swamy M N S. Neural Networks and Statistical Learning. Springer-Verlag, London, [2] Kruse R, Borgelt C, Klawonn F, Moewes C, Steinbrecer M, Held P. Computational Intelligence. Springer, Berlin, [3] Rutkowski L. Computational Intelligence. Metods and Tecniques. Springer-Verlag, Berlin-Heidelberg, [4] Jang J-S, Sun C-T, Mizutani E. Neuro-Fuzzy and Soft Computing: A Computational Approac to Learning and Macine Intelligence. Prentice Hall, Upper Saddle River, [5] Arora N, Saini J R. Estimation and Approximation Using Neuro-Fuzzy Systems. International Journal of Intelligent Systems and Applications(IJISA), 2016, 8(6): [6] Kartika B S, Deka P C. Modeling of Air Temperature using ANFIS by Wavelet Refined Parameters. International Journal of Intelligent Systems and Applications(IJISA), 2016, 8(1): [7] Lugofer E. Evolving Fuzzy Systems Metodologies, Advanced Concepts and Applications. Springer, Berlin, [8] Kasabov N. Evolving Connectionist Systems. Springer-Verlag, London, [9] Kasabov N. Evolving fuzzy neural networks: teory and applications for on-line adaptive prediction, decision making and control. Australian J. of Intelligent Information Processing Systems, 1998, 5(3):

9 [10] Kasabov N. Evolving fuzzy neural networks algoritms, applications and biological motivation. Proc. Metodologies for te Conception, Design and Application of Soft Computing, Singapore, 1998: [11] Sugeno M, Kang G T. Structure identification of fuzzy model. Fuzzy Sets and Systems, 1998, 28: [12] Takagi T, Sugeno M. Fuzzy identification of systems and its applications to modeling and control. IEEE Trans. on Systems, Man, and Cybernetics, 1985, 15: [13] Ivaknenko A G. Heuristic self-organization in problems of engineering cybernetics. Automatica, 1970, 6(2): [14] Ivaknenko A G. Polynomial teory of complex systems. IEEE Trans. on Systems, Man, and Cybernetics, 1971, 1(4): [15] Bodyanskiy Ye, Grimm P, Teslenko N. Evolving cascade neural network based on multidimensional Epanecnikov s kernels and its learning algoritm. Int. J. Information Tecnologies and Knowledge, 2011, 5(1): [16] Bodyanskiy Ye, Teslenko N, Grimm P. Hybrid evolving neural network using kernel activation functions. Proc. 17t Zittau East-West Fuzzy Colloquium, Zittau/Goerlitz, Germany, 2010: [17] Miki T, Yamakawa T. Analog implementation of neo-fuzzy neuron and its on-board learning. In: Computational Intelligence and Applications, 1999: [18] Yamakawa T, Ucino E, Miki T, Kusanagi H. A neo fuzzy neuron and its applications to system identification and prediction of te system beavior. Proc. 2nd Int. Conf. on Fuzzy Logic and Neural Networks IIZUKA-92, Iizuka, Japan, 1992: [19] Ucino E, Yamakawa T. Soft computing based signal prediction, restoration, and filtering. In: Intelligent Hybrid Systems: Fuzzy Logic, Neural Networks, and Genetic Algoritms, 1997: [20] Kolodyazniy V, Bodyanskiy Ye. Cascaded multiresolution spline-based fuzzy neural network. Proc. Int. Symp. on Evolving Intelligent Systems, Leicester, UK, 2010: [21] Otto P, Bodyanskiy Ye, Kolodyazniy V. A new learning algoritm for a forecasting neuro-fuzzy network. Integrated Computer-Aided Engineering, 2003, 10(4): [22] Bodyanskiy Ye, Koksenev I, Kolodyazniy V. An adaptive learning algoritm for a neo-fuzzy neuron. Proc. 3-rd Int. Conf. of European Union Soc. for Fuzzy Logic and Tecnology (EUSFLAT 03), Zittau, Germany, 2003: [23] Bodyanskiy Ye, Viktorov Ye. Te cascade neo-fuzzy arcitecture using cubic-spline activation functions. Int. J. Information Teories & Applications, 2009, 16(3):