Reduction of Neural Network Training Time Using an Adaptive Fuzzy Approach in Real Time Applications

Transcription

1 Internatonal Journal of Informaton and Electroncs Engneerng, Vol., No. 3, May Reducton of Neural Network Tranng Tme Usng an Adatve Fuzzy Aroach n Real Tme Alcatons Hamdreza Rashdy Kanan and Mahd Yousef Azar Khanan Abstract A maor roblem of neural network n real-tme alcatons s ther long tranng tme. We resent a modfcaton of the neural network (NN for reducton of tranng tme. Unlke tradtonal tranng tme reducton algorthms, we roose a new Adatve Fuzzy technque to create ensembles of neural network usng multle roectons of the same data obtaned from dfferent NNs. The urose of ths aer s to demonstrate the otmzaton of tranng that occurs wth the alcaton of fuzzy controller theory to neural network. A fuzzy system s emloyed to control the learnng arameters of a neural network to reduce the ossblty of overshootng durng the learnng rocess. Hence, the learnng tme can be shortened. Ths aer comares the tranng effcency and accuracy between a NN and a fuzzy controlled neural network, when they are requred to carry out the same assgnment. We ustfy the sutablty of the roosed method by some exerments n soccer robot traectory generaton tasks; the resultng fuzzy controlled neural network ndcates a sgnfcant reducton n the tranng tme by 3%. Index Terms Neural Network, Adatve Fuzzy Logc Controller, Backroagaton Learnng Algorthm, Moble Robot, Traectory Generaton. I. INTRODUCTION Hghlght Neural network erformance s drectly related to the sze and qualty of tranng samles. The ablty of a moble robot to generate real-tme traectory by neural network n unknown and unstructured envronments by relyng only on ts sensory system s regarded as the key ssue n an enormous number of research ublcatons. In order to acheve ts goal, the robot s usually requred to determne n real-tme a safe and smooth ath from a startng locaton to an end locaton (target. Among all soft-comutng methods, fuzzy logc based nference and neural network have been found to be the most attractve technques that can be utlzed for ths urose. Fuzzy system s tolerant to nose and error n the nformaton comng from the sensory system, and most mortantly t s a factual reflecton of the behavor of human exertse. Neural network have been roven to be very effcent at handlng a wde range of engneerng alcatons []. The evolvement of neural network aradgms have rovded a owerful tool to deal wth moble robot traectory generaton rocess, whch exhbts ncomlete and uncertan knowledge due to the naccuracy and mrecson nherent from the sensory system. A formal tranng rocedure s Manuscrt receved March, ; revsed Arl 7,. H. R. Kanan s wth the Deartment of Electrcal Engneerng, Bu-Al Sna Unversty, Hamedan, Iran (Tel ; fax: ; ; emal: h.rashdykanan@basu.ac.r. M. Y. A. Khanan s wth the Electrcal, Comuter and IT Engneerng Deartment, Qazvn Branch, Islamc Azad Unversty, Qazvn, Iran. requred by the neural network n order to learn. In general, t s useful only when the network archtecture (.e., model s chosen correctly. Too small network cannot learn the roblem well, but too large network wll lead to over-generalzaton and thus oor erformance []. Deendng on the nature of data, hdden layers wth many neurons may add a large amount of tme to tranng hase and not roduce any observable reducton n squared error. However, tranng a multlayer network wth more than one hdden layer s slow and further t wll not also gve good results [3]. Ths s due to ths fact that the weghts at dee hdden layers are hardly otmzed. The tme requred for tranng such a network s also extremely hgh. Tranng a system to make decson n the resence of uncertantes s a dffcult roblem, esecally when comutatonal resources are lmted. However, the desrablty of ast decsons can usually be assessed after that, outcomes of ther mlementatons are observed. Therefore, unsuervsed tranng methods that do not utlze those assessments cannot take full advantage of the avalable knowledge. Adatve Fuzzy controller systems have structured numercal knowledge that can exlot nosy or nexact stuatons [, 3]. By emloyng a fuzzy control system to adatvely vary the learnng arameters of the network, the tranng tme can be reduced sgnfcantly. The fuzzy control system uses exert knowledge to determne the secfc change n learnng arameters. The back-roagaton algorthm has a maor drawback: a long erod of tranng tme. That s, for any gven roblem, the network must have many solved examles resented reeatedly durng the learnng rocedure before the network learns the data wth an accetable degree of accuracy. Therefore, the learnng stage s a lengthy rocess. As far as we concerned, lmted study drectly has ever tred to combne the fuzzy set theory wth neural network learnng algorthms n order to control the learnng arameters and reduce the tranng tme n real-tme alcatons. Ths rocess can be short tme by usng fuzzy controller systems. Incororaton of fuzzy technques can mrove erformance n these cases. The method roosed n ths aer takes advantages of the concet of the adatve fuzzy controller systems to tune the learnng arameters and reduce the tranng tme by 3% n traectory generaton tasks for soccer robots. The rest of the aer s organzed as follows. Secton II ntroduces some relmnares about neural network basc tranng models, back-roagaton learnng algorthm, and standard tranng convergence tme. The novel ecewse adatve fuzzy logc control system s roosed n Secton III. In Secton IV, the case study on determnaton of learnng arameters wth adatve fuzzy controller s resent to demonstrate the effectveness of the roosed aroach. Ths 47

2 Internatonal Journal of Informaton and Electroncs Engneerng, Vol., No. 3, May secton descrbes how the learnng rocess s streamlned. In addton to the tranng effcency, the neural network stll obtans an mressve erformance, whch s evdenced by the exermental results, dscussed n Secton IV. F. Fnally, some conclusons are drawn n Secton V. II. BASIC TRAINING ALGORITHMS A neural network (NN, an nformaton-rocessng centre, mmcs the human bran wth resect to oeraton and rocessng ablty. Neural network can be successfully used as a forecastng tool due to caablty n dentfyng non-lnear relatons. In order to learn a soluton, a tranng strategy must be used. One of the most oular tranng methods s backroagaton [4]. A. Backroagaton Learnng Algorthm The Backroagaton algorthm s one of the most oular and versatle forms of neural network classfers used for the recognton of comlex atterns. In a tycal backroagaton tranng hase for a multlayer network, the suervsor resents nut data to the network and comares the network s actual outut, O, wth the target (or desred outut, T : T O Er ( Ths dfference, or error, s used to change the connecton weghts between neurons n the network: ( Er w( t ( w ( t w Wth the back-roagaton algorthm, neural network weghts are adusted n a gradent descent manner, whch means the mnmzaton of the error between the exected outut and the actual outut for a artcular nut. However, two maor roblems exst wth ths knd of the networks. Frst, slow convergence rate and second, the ossblty of settlng nto a local mnmum [5]. A otentally large number of teratons are requred to tran the network untl t learns the data wth an accetable degree of accuracy. B. Fnal Standard Tranng Convergence Tme A maor roblem of a back-roagaton learnng algorthm s ts slow convergence tme [6]. Durng tranng, a NN s tryng to converge to the global mnmum wth the shortest ath. Global mnmum s the maxmum erformance that the neural network can yeld gven the sze of the network. At the begnnng of a learnng rocess, the network learns radly, and ts value of RMS error decreases fast. In general, the value of RMS reflects how a neural network s erformng, and comuted by equaton (3. Error RMS o ( ( t k k x k (3 nn Fnally, the NN may comlete ts learnng, and ossbly takes thousands teratons of data resentatons. III. ADAPTIVE FUZZY LOGIC CONTROLLER An Adatve fuzzy logc control system conssts of four comonents as shown n Fgure. [7,8]. A fuzzy logc controller (FLC s an ntellgent control system that smoothly nterolates between rules,.e. rules fre to contnuous degrees and the multle resultant actons are combned nto an nterolated result. Degree of membersh Degree of membersh Fuzzfer Crs Value.5 Knowledge Base Inference Engne Fg.. A general scheme of a Fuzzy Logc Controller. NL.5 NS Fg.. An examle of lngustc varables. Defuzzfer Crs Value Drecton nut Drecton nut Imlementaton of a fuzzy controller requres assgnng membersh functons for nuts and oututs. Inuts to a fuzzy controller are usually measured varables, assocated wth the state of the controlled lant, that are fuzzfed (assgned membersh values before beng rocessed by an nference engne. The heart of the controller nference engne s a set of f-then rules whose antecedents and consequences are made u of lngustc varables and assocated fuzzy membersh functons. Consequences from fred rules are numercally aggregated by fuzzy set unon and then collased (defuzzfed to yeld a sngle crs outut as the control sgnal for the lant. For detaled ntroductons to fuzzy control, fuzzy set oeratons, and concets of fuzzfcaton, nference, aggregaton, and defuzzfcaton see one of [7, 9]. There are two reasons that fuzzy logc control systems are referred: the nfnte measurement resoluton, and mrecse lngustc descrtons. Due to usng fuzzy sets and lngustc varables, there s no lmted resoluton as t does n a conventonal control system. Forced effcency on the outut can be the result caused by low resoluton. An examle of the use of lngustc varables s shown n Fgure. In addton to use lngustc varables, fuzzy logc control systems can encasulate comlex nut/outut oeratons n the form of ZE 47

3 Internatonal Journal of Informaton and Electroncs Engneerng, Vol., No. 3, May rules. Also, the oeratons of most control systems are based on exert knowledge. IV. DETERMINATION OF LEARNING PARAMETERS The mlementaton of fuzzy logc control system to adatvely determne learnng arameters n neural network s dscussed n ths secton. The exermental results are resented, too. A. Tunng Membersh Functon The rocedure of the algorthm has several maor stes and conssts of clusterng the data nto classes. The membersh functons are then generated from the classes obtaned. The algorthm s descrbed for one arameter, and as follows. B. Fnd Dfference Between Adacent Values Sort the values of each attrbute of the tranng nstances n a descendng sequence and fnd the maxmum attrbute value and the mnmum attrbute value of each attrbute. Randomly choose n nstances as the tranng nstances, and let the remanng n nstances be the testng nstances [9]. Then, we can fnd the maxmum attrbute value and the mnmum attrbute value of each seces of the n tranng nstances. Gven a data set, there are n tranng samles. The values for the arameter n queston, X x, x,, x are sorted nto ascendng order, the values are sorted to fnd an assocaton between adacent values. Let the maxmum attrbute value and the mnmum attrbute value of any two attrbutes of each seces of the tranng nstances s the boundares to form a lane. The dfference between adacent values n the sorted data s determned. The dfference obtaned wll rovde a way to calculate the smlarty between adacent values. The dfference for a set of tranng set data s: X X,,, 3,..., n (4 where, X and X + are adacent values n the sorted data. C. Fnd Smlartes Between Adacent Values The followng equaton fnds the smlartes between adacent values and mas them nto real numbers between and. X for C s D C (5 s otherwse where, s the dfference between adacent data, s s the standard devaton of, and C s the control arameter. The control arameter s used to determne the shae of the membersh functon. D. Defne the Membersh Functon for Each Class One of the smlest membersh functons s the trangular membersh functon, and wll be used for the rest of the equatons. The trangular membersh functon for class conssts of three arameters: the central vertex ont, b, and the two endonts, a and. The central vertex ont s n determned for each class and s calculated by the followng equaton: x x x x b k k k k k k k k where reresents the th class, k reresents the endng data ndex for ths class,.e., data x through x k fall nto class, and s the smlarty between x and x +. The endonts of the membersh functon, a and c, are obtaned usng nterolaton. The followng equaton determnes the left and rght endonts. b x a b ( x (7 xk b b ( x k where, a s the left endont and c s the rght endont. (x and (x k s the membersh determned by usng equaton (8 : x x mn,,, (6 k k (8 where, k reresents the maxmum data ndex value wthn the class. The membersh functons for the classes determned by ths method are shown n Fgure 3. Membersh Chance of (Outut MF of "Traectory Proerty Intal Intal Fg. 3. Determnng the membersh functons. E. Fnd The Proosed Aroach Tranng Samles Neural Network Target Drecton (Inut Inut-Outut Surface Target Drecton (Outut Actual Outut Fg. 4. A basc archtecture of fuzzy logc controlled neural network. A fuzzy control system to adatvely determne learnng arameters (.e. learnng rate and momentum n neural network s the focus of ths study. As roosed n [], ths contrasts wth regular neural network, n whch the learnng arameters are fxed. The on-lne fuzzy logc controller s Target Outut Erro Fuzzy Logc Controller 47

4 Internatonal Journal of Informaton and Electroncs Engneerng, Vol., No. 3, May utlzed to adat the learnng arameters based on the RMS error generated by the neural network. A basc archtecture for such an ntegrated system s shown n Fgure 4. The urose of the FLC s to automatcally adust the learnng rate and momentum term accordng to the RMS error surface. To descrbe the error surface, there are four arameters used to create the rules for the FLC. They are relatve error (RE, change n relatve error (CRE, sgn change n error (SC and cumulatve sum of sgn change n error (CSC whch s descrbed as follows: RE( t E( t E( t CRE RE( t RE( t SC( t sgn( RE( t sgn( RE( t t CSC( t SC( m m t 4 The collecton of rules created (called the rule base reflects the use of these arameters to adatvely change the neural network s learnng rate and momentum. The rule base can be summarzed concsely n the form of a decson table. Two of the fuzzy rule bases (decson tables for the alcatons dscussed n secton 4. are shown. To demonstrate the dea, two of the fuzzy IF-THEN based rules that are emloyed n the FLC are lsted [9], []: Rule : If RE s small and CSC s less or equal to two, then the value of learnng rate and momentum term should be ncreased. Rule : If CSC s larger than three, then the value of learnng rate and momentum term should be decreased regardless the value of RE and CRE. Table I descrbes the rule base for the change n the learnng arameter,, and Table II descrbes the rule base for the change n momentum,. In these decson tables, the utlzaton of fve lngustc varables has been used. Secfcally, NL reresents a Negatve Large value, NS s Negatve Small, ZE s zero, PS s Postve Small, and PL s Postve Large. CRE CRE TABLE I: DECISION TABLE FOR, WHEN CSC. RE NL NS NS NS NS NS NS NS ZE PS ZE NS ZE ZE PS ZE NS ZE PS NS ZE PS ZE NS PL NS NS NS NS NS TABLE II: DECISION TABLE FOR, WHEN CSC. RE NL NS ZE... PS. PL.. (9 F. Exermental Results Here we descrbe a fuzzy MLP for rule generaton [] and resent ts effectveness n soccer robot traectory generaton roblems. At the end of the tranng hase the network s suosed to have encoded the nut outut nformaton dstrbuted among ts connecton weghts. Ths consttutes the knowledge base of the desred decson-makng system. Handlng of mrecse nuts s ossble and natural decson s obtaned assocated wth a certanty measure denotng the confdence n the decson. Moreover, we have ncluded both rule extracton and rule refnement n the broader ersectve of rule generaton. Outut Outut Tme, sec Desred traectory neural network redcton Fg. 5. Accuracy of neural network controller. Desred traectory Fuzzy controlled NN redcton Tme, sec Fg. 6. Accuracy of fuzzy controlled neural network. The system was tested on a classc alcaton area, to soccer robot traectory generaton. The fuzzy logc controlled neural network showed 3% average mrovement n tranng tme. That s, n comarson wth the erformance of a regular neural network wth dfferent value learnng rate, and momentum term traned n the normal fashon, the fuzzy logc controlled system requres about /3 of the teratons (measured to convergence by an error lmt, whch only takes less than 5 seconds n our PC wth 3. GHz CPU and Gbyte RAM. Futhermore, the fuzzy logc controlled system assesses erformance wth each teraton, and the erformance s sgnfcantly mroved. In a secfc case when both systems have a learnng rate of.8 and momentum term of.8, the regular NN fals to learn n about teratons. Fgures 5 and 6 llustrate the effcency and erformance of the roosed algorthm. Here the NN outut wth the roosed new algorthm s much closer to the orgnal ndex data whle the fuzzy logc controlled neural network system s able to contnue learnng. It should be mentoned that the "best" 473

5 Internatonal Journal of Informaton and Electroncs Engneerng, Vol., No. 3, May redctons were obtaned from our frst control fle n whch we utlzed two hdden layers of neurons. It also "learned" much faster than the frst examle. From Table III the roosed aroach has the best MSE and standard devaton. In addton, t s faster and t has best success 9% to reach the goal. TABLE III: THE COMPARISON BETWEEN THE AVERAGE OF THE TRAINING PERFORMANCE BETWEEN NN, AND THE FUZZY CONTROLLED NEURAL NETWORK. Algorthm MSE Standard devaton Success NN 5.63e e-3 89% The Proosed Aroach 5.694e e-3 9% V. CONCLUSION Ths study ndcated that the neural network s tranng tme s reduced sgnfcantly by combnng an adatve fuzzy control system wth a neural network n order to adatvely vary the learnng arameters and reduces the tranng tme n real-tme alcatons. The method roosed n ths aer takes advantages of the concet of the adatve fuzzy controller systems to tune the learnng arameters and reduce the tranng tme by 3% n traectory generaton tasks for soccer robots and t yelds 9% accuracy. In addton, ths technque can reduce the ossblty of overshootng and sometmes hel the network get out of a local mnmum.the network s ablty to converge durng tranng and the fnal erformance are deendent on the learnng arameters. Our study renforces ths fact, as our smulatons have resented that a wrong value of learnng rate can lead to oor accuracy. Furthermore, the methodology s flexble, and can be exercsed on other neural network alcatons. [5] D. Srnvasan, X. Jn, and R. L. Cheu, Adatve neural network models for automatc ncdent detecton on freeways, Neurocomutng, vol. 64, , March 5. [6] S. Zhou, Z. Q. Lu, and J. Y. Zhang, Fuzzy causal networks: general model, nference, and convergence, IEEE Trans. on Fuzzy Systems, vol. 4, ssue. 3, June 6. [7] J. Zhang, Modellng and otmal control of batch rocesses usng recurrent neuro-fuzzy networks, IEEE Trans. on Fuzzy Systems, vol. 3, no. 4, , August 5. [8] M. C. Choy, D. Srnvasan, and R. L. Cheu, Neural networks for contnuous onlne learnng and control, IEEE Trans. on Neural Networks, vol. 7, ssue. 6,. 5-53, Nov. 6. [9] D. A. Kaur, and K. Kaur, Fuzzy exert systems based on membersh functons and fuzzy rules, n Proc. Internatonal Conf. on Artfcal Intellgence and Comutatonal Intellgence (AICI, 9, [] S. M. Ba, and S. M. Chen, Automatcally constructng grade membersh functons of fuzzy rules for students' evaluaton, Exert Systems wth Alcatons, vol. 35, ssue. 3, , October 8. [] T. H. Yu, K. H. Huarng, A neural network-based fuzzy tme seres model to mrove forecastng, Exert Systems wth Alcatons, vol. 37, ssu. 4, , Arl. Hamdreza Rashdy Kanan was born n 978 n Hamedan. He receved B.Sc. degree n Electronc Engneerng from Tabrz Unversty n, M.Sc. degree n Electronc Engneerng from Tarbat Modarres Unversty wth the frst rank n and Ph.D. degree n Electronc Engneerng from Amrkabr Unversty of Technology (Tehran Polytechnc n 8. From 8 to 9, he was emloyed as a Research Scentst at Iran Telecommuncaton Research Center (ITRC, ON, where he was nvolved n research for multmodal bometrcs. In 9, he oned the Faculty of Engneerng at Bu-Al Sna Unversty, Hamedan, Iran, as an Assstant Professor. Hs maor research nterests nclude dgtal mage rocessng, attern recognton, comuter vson, face detecton and recognton, facal exresson recognton, and gender classfcaton. REFERENCES [] G. Leng, T. M. McGnnty, and G. Prasad, Desgn for self-organzng fuzzy neural networks based on genetc algorthms, IEEE Trans. on Fuzzy Systems, vol. 4, ssue. 6, , Dec. 6. [] G. E. Hnton, and R. R. Salakhutdnov, Reducng the dmensonalty of data wth neural networks, Scence, vol. 33, no. 5786, , July 6. [3] Y. Bengo, P. Lambln, D. Poovc, and H. Larochelle, Greedy layer-wse tranng of dee networks, n Advances n Neural Informaton Processng Systems, MIT Press, 7, ch. 9, [4] L. Huun, A nonlnear redctve model based on BP neural network, n Proc. Control and Decson Conference (CCDC, Chna,, Mahd Yousef Azar Khanan was born n 983 n Qazvn. He receved B.Sc. degree n Electronc Engneerng from Islamc Azad Unversty (Qazvn Branch n 7, M.Sc. degree n Comuter Engneerng (Artfcal Intellgence from Islamc Azad Unversty (Qazvn Branch n and He s a Ph.D. student n Bomedcal Engneerng n Amrkabr Unversty of Technology (Tehran Polytechnc snce. From 3 to 9, he was emloyed as a Researcher at Mechatronc Research Laboratory (MRL. Hs maor research nterests nclude Artfcal ntellgence and robotcs, Theory and alcaton of bologcal sgnal rocessng, human moton analyss, Comutatonal neuroscence and neural engneerng. 474