Adaptive modified backpropagation algorithm based on differential errors

Transcription

1 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Adptve modfed bckpropgton lgorthm bed on dfferentl error S.Jeyeel Subvth nd T.Kthrvlvkumr b Deprtment of Informton Technology, Sr Klwr College, Svk 62630, Tmlndu, Ind b Deprtment of Computer Scence, V.H.N.S.N. College, Vrudhungr 62600, Tmlndu, Ind Abtrct A new effcent modfed bck propgton lgorthm wth dptve lernng rte propoed to ncree the convergence peed nd to mnmze the error. The method elmnte ntl fxng of lernng rte through trl nd error nd replce by dptve lernng rte. In ech terton, dptve lernng rte for output nd hdden lyer re determned by clcultng dfferentl lner nd nonlner error of output lyer nd hdden lyer eprtely. In th method, ech lyer h dfferent lernng rte n ech terton. The performnce of the propoed lgorthm verfed by the multon reult. Keyword Adptve lernng rte, Dfferentl error, Lner error, Modfed tndrd bck propgton, Nonlner error.. Introducton The clcl method for trnng feedforwrd neurl network (FNN) the bckpropgton lgorthm (BP) [9] whch bed on the teepet decent optmzton technque. Trnng uully crred out by tertve updtng of weght bed on the error gnl. BP decent lgorthm whch ttempt to mnmze the error t ech terton. The weght of the network re duted by the lgorthm uch tht the error decreed long decent drecton [8]. Trdtonlly two prmeter clled lernng rte nd momentum fctor re ued for controllng the weght dutment long the decent drecton. Fndng ntl lernng rte nd fxed lernng rte mut be done wth gret cre. If the lernng rte very lrge, then the lernng my become untble. If t mll, then often t very low for prctcl pplcton whch led to fndng of ft lernng lgorthm [3]. DOI : 0.52/ce

2 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Mny technque hve been propoed to ncree the convergence peed. Abd et l. [] decrbed modfed BP lgorthm (MBP) bed on um of lner nd nonlner error of output neuron to mprove the peed of convergence n mnmum terton. The lgorthm converge fter thn the tndrd BP lgorthm. Some reercher focued on electon of better energy functon [2,4] nd electon of utble lernng rte nd momentum [6,9,6,7]. Lernng rte dptton by gn chnge wll dpt the tep ze by hvng eprte lernng rte for ech connecton [2] A problem wth ll of thee technque ther convergence to locl mnm. To olve th problem, globl erch lgorthm lke genetc lgorthm hve to be ppled [4]. But erchng for the globl mnmum my be trpped t locl mnm durng grdent decent. Alo f the network trned wth dturbnce n the nput, then globl mnmum pont cn not be found. So ft convergence nd trong robutne my not be gurnteed. To olve thee problem dptve lernng lgorthm hve been developed recently. Jeong nd Lee [7] hve propoed n dptve lgorthm bed on frt nd econd order dervtve of neurl ctvton t hdden lyer whch reult n hybrd lernng rule. Sh nd Bc [3] hve propoed n dptve lernng rte lgorthm for I/O dentfcton bed on two ANN ung convergence nly of the conventonl grdent decent method. Xe nd Zhng [5] hve propoed vrble lernng rte LMS lgorthm ung Lypunv method epeclly when there noe n the nput gnl. Beher et l. [3] hve decrbed new lernng lgorthm LFI nd LF II bed on Lypunov functon for the trnng of feeforwrd neurl network. In th lgorthm fxed lernng prmeter re replced wth dptve lernng prmeter ung convergence theorem bed on Lypunov tblty theory. ]. Zhhong Mn et l [9] propoed new dptve bckpropgton lgorthm bed on lypunov tblty theory for neurl network. They howed tht the cnddte of lypunov functon of the trckng error between the output of neurl network nd the dered reference gnl choen frt, nd the weght of neurl network re then updted from the output lyer to nput lyer. Our prevou work [8] decrbe modfed bckpropgton lgorthm n neghborhood bed network by replcng fxed lernng prmeter by dptve lernng prmeter. Here the prmeter re clculted ung convergence theorem bed on Lypunov tblty theory. Irnmneh nd Mhdv [] hve propoed lernng method ung dfferentl dptve lernng rte. In ech terton, the lernng rte updted ccordng to the error of the output lyer. The lernng rte of the output lyer computed by dfferenttng the error of the output lyer. The dfferentton of the gmodl functon of the um of multplcton of error of ech output lyer neuron wth correpondng weght dvded by the number of hdden neuron ued n dptve lernng rte of hdden lyer. We propoe new dptve lernng rte lgorthm to peed up the lernng proce of the neurl network. In the propoed lgorthm eprte dptve lernng rte ued n both hdden nd output lyer. In th, lner nd nonlner error for ech neuron n the output lyer re multpled wth dervtve of the correpondng neuron ctvton functon, dded nd then dfferentted to get the dptve lernng rte for the output lyer. Lner nd nonlner error of ech hdden neuron multpled wth t correpondng output lyer weght eprtely nd then dded. Then the vlue dvded by number of hdden neuron. The dfferentton of the gmodl functon of th vlue ued lernng rte for the hdden lyer. The effcency of the propoed lgorthm n term of tme nd epoch hown by multng the benchmrk problem uch XOR, 3-bt prty, nonlner functon pproxmton problem nd r dt et. 22

3 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 The remnng of the pper orgnzed follow: ecton 2 decrbe dptve lernng rte lgorthm, ecton 3 decrbe the propoed lgorthm nd ecton 4 dcue the multon reult. 2. Trnng of Neurl network Conder ngle hdden lyer feedforwrd neurl network hown n Fgure. A b node ncluded n the nput lyer. Let X = (x ) be the nput vector, Y = (y ) be the output vector nd w [] be the weght of the th unt n the (-) th lyer to the th unt n the th lyer. The ctvton functon of both hdden nd output lyer neuron re umed to be gmodl. Sequentl mode trnng ppled here. X 0 (B) -2 - X Y : : X n Fgure. Sngle hdden lyer neurl network. Stndrd BP (SBP) For ech nput pttern nonlner output of the th neuron of the output lyer network clculted follow: u = n ( ) = w y u ( + e ) () f ( u ) = = d (2) where n (-) repreent number of neuron n the (-) th lyer. SBP mnmze the followng crteron equl to the um of the qure of the error between the ctul y nd the dered d output for pttern p. n = ( ) E p e (3) = where the nonlner error gnl 2 e = ( y d ) (4) 23

4 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 The weght updte rule E p w = µ (5) where µ the fxed lernng rte elected by trl nd error. Subttutng (3) n (5), the weght updte rule become, w = µ e w = µ e y y u u w = µ e f '( u ) y (6) The etmted nonlner error of the hdden lyer (-) follow: e n f r= ( u ) e w = (7) r r The weght updte rule of the hdden lyer r E ( ) p w = µ (8) ( ) ( ) ( ) ( ) ( 2) ( u ) y w = µ e f ' (9) Now the weght of both hdden nd output lyer re updted ung w Modfed BP ( t ) = w ( t) + w (0) For ech nput pttern the lner nd nonlner output of the th neuron n output lyer of the network re clculted repectvely follow: u = n( ) = w y u ( + e ) () f ( u ) = = d (2) where n (-) repreent number of neuron n the (-) th lyer. The MBP pproch mnmze modfed form of crteron E p ued n tndrd BP lgorthm. The crter E p um of the lner nd nonlner qudrtc error of the output neuron for the current pttern p. 24

5 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 E p = = 2 2 ( e ) + ( e2 ) n = where the nonlner error gnl e nd the lner error gnl 2 λ (3) 2 = ( y d ) (4) e 2 = ( ly u ) (5) Here ly ( y ) = f (6) where y nd d repectvely re dered nd current output for th unt n the th lyer. p n (3) denote the p th pttern nd λ the weghtng coeffcent. In the output lyer the lner nd nonlner error re known []. So the weght updte rule [] for the output lyer E p w = µ (7) where µ the fxed lernng rte elected by trl nd error. Subttutng (3) n (7), the weght updte rule become, y u w = µ e + µλe 2 y u w = µ e + µλe u 2 y f ' ( ) u y + e y w = µ e µλ (8) 2 In the hdden lyer, the lner nd nonlner error re unknown nd mut be clculted []. The etmted nonlner nd lner error [] of the hdden lyer (-) re repectvely follow: e n f r= ( u ) e w = (9) = f r r r n ( u ) e2r wr e2 (20) r = The weght updte rule of the hdden lyer ( ) ( ) E ( ) p w = µ (2) ( ) ( ) ( 2) ( ) ( 2) ( u ) y + e y w = µ e f ' µλ (22) 2 25

6 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Now the weght of both hdden nd output lyer re updted ung w ( t ) = w ( t) + w (23) where t repreent terton. In order to ncree the convergence peed nd to mke the lernng rte µ dptve, we propoe new technque bed on dfferentl lner nd nonlner error of output lyer nd hdden lyer. Adptve Modfed BP In the propoed technque frt lner nd nonlner error of th neuron n the output lyer re clculted ung (4), (5) nd (6). Then ll the lner nd nonlner error of the neuron re multpled wth the dervtve of the correpondng neuron ctvton functon nd dded eprtely hown below: δ e f ( u ) (24) o n = = δ e f ( u ) (25) o2 n = = 2 Then δ o nd δ o2 re dded to get the totl error o o o2 δ = δ + δ (26) Now the totl error dvded by the totl number of output neuron known δ n δ = (27) nd the µ out of the output lyer computed follow: out ( ) µ = f δ (28) where f gmodl ctvton functon gven by wth property f ( δ ) δ ( + e ) = ( δ ) = f ( δ )( f ( δ ) (29) f (30) Then the chnge of weght re clculted ung w ( ) u y + µ λe y = µ e (3) out f ' out 2 δ Smlrly for the hdden lyer (-) the me procedure ppled to clculte dptve lernng 26

7 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 rte µ hd. Frt nonlner error e nd lner error ( ) ung (9) nd (20). Then nonlner error δ h nd δ h2 h n( ) n = = ( ) e of ll hdden neuron re clculted ( ) 2 δ = e w (32) h2 n( ) n = = ( ) 2 repectvely re δ = e w (33) nd then both δ h nd δ h2 re dded to get the totl error δ h below: δ = δ + δ (34) h h h2 Now the totl error dvded by the totl number of hdden neuron known b δ h δ = (35) n ( ) nd then µ hd computed follow: hd b ( ) µ = f δ (36) where f gmodl ctvton functon. Then the chnge of weght re clculted ung the followng equton. w ( ( ) ( 2) ( ) ( 2) ( u ) y + µ λe y = µ e f ' (37) ( ) hd Now the weght of both hdden nd output lyer re updted ung (23). 3. Algorthm hd 2 b δ. Defne network tructure nd gn ntl weght rndomly. 2. Select pttern to be proceed n the network. 3. For ech node n the hdden lyer, compute. Net vlue ung Eq (). b. Output vlue ung Eq (2). 4. For the output lyer, compute. Net vlue ung Eq () nd output vlue ung Eq (2). b. Non Lner nd lner error ung Eq (4), Eq (5) nd Eq (6). c. Adptve lernng rte µ out ung Eq (24) to Eq (30). d. Chnge of weght ung Eq (3). 5. For the hdden lyer, compute. Non Lner error ung Eq (9). b. Lner error ung Eq (20). 27

8 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 c. Adptve lernng rte µ hd ung Eq (32) to Eq (36). d. Chnge of weght ung Eq (37) 6. Updte weght of output nd hdden lyer ung Eq (23). 7. Repet the tep 2 to 6 for ll the pttern. 8. Evlute network error wth new weght. 9. Stop trnng f termnton condton reched. Otherwe repet the tep 3 to Smulton Reult nd dcuon The performnce of the propoed lgorthm verfed by multng the benchmrk problem uch XOR, 3-Bt prty, Nonlner functon pproxmton functon problem nd Ir dt et. All the problem re multed ung lnguge C on Pentum IV wth 2.40 GHz. The convergence property of the propoed lgorthm compred wth MBP [], Bckpropgton wth momentum (BPM) [9] nd bckpropgton lgorthm [0]. Ech tme ll the pttern n the problem hve been ued once n the network durng trnng clled n epoch. Men qured error (MSE) of the network clculted by dvdng the um of qured lner error n ech epoch by twce the number of pttern. Network tructure, prmeter vlue nd termnton condton re condered contnt for ll the lgorthm to hve better compron. Network weght re rndomly nd unformly generted from the rnge [-5, +5]. The weghtng coeffcent λ gned the vlue 3.7. The convergence of the propoed lgorthm hown by the lernng curve. XOR The network tructure condered n th problem h 3 nput neuron ncludng b, 4 hdden neuron nd one output neuron. The termnton condton fxed for convergence MSE The reult obtned re tbulted n Tble. Tble : Compron tble for XOR problem ALGORITHM PARAMETERS EPOCHS TRAINING TIME MSE IN MSECS BP µ= BPM µ=.5 α= MBP µ=0.25 λ= Propoed λ= It h been oberved tht the BP lgorthm tke 76 mec nd 754 epoch to rech the mnmum error. The propoed lgorthm converge fter even the lernng rte not fxed n the begnnng. Snce the lernng rte dpted bed on the error of output nd hdden lyer t tke mnmum tme of 49 mec nd mnmum epoch of 237 for convergence. The lernng 28

9 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 curve obtned hown n Fgure 2 for the propoed lgorthm. The dptve lernng rte obtned bed on the error of output lyer nd hdden lyer re hown n Fgure 3 nd Fgure 4. MSE Epoch Fgure 2. Lernng curve bed on MSE nd Epoch of XOR problem for the propoed lgorthm Muout Iterton Fgure 3. Adptve lernng rte of hdden lyer. Muhdden Iterton 3-bt prty Fgure 4. Adptve lernng rte of output lyer. We ued 4-9- ANN ncludng one b n nput lyer to multe the 3-bt prty problem. The reult obtned re tbulted n Tble 2. 29

10 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Tble 2. Compron tble for the 3-bt prty problem ALGORITHM PARAMETERS EPOCHS TRAINING TIME MSE IN MSECS BP µ= BPM µ=.5 α= MBP µ=0.25 λ= Propoed λ= From the tble t h been oberved tht the propoed lgorthm converge quckly wthn 77 mec n 298 epoch. But the lgorthm BP, BPM nd MBP requre 570, 450 nd 520 epoch for convergence repectvely. Alo they requre 379 mec, 364 mec nd 26 mec tme to rech the termnton condton MSE All the lgorthm except propoed lgorthm tke tme to fx the lernng rte. The bet performnce of the propoed lgorthm hown n Fgure 5. MSE Epoch Fgure 5. Lernng curve bed on MSE nd Epoch of 3-bt prty problem for the propoed Algorthm Nonlner functon pproxmton problem A nonlner functon pproxmton wth 8 nput vlue x defned n th problem. The three output quntte y re defned by the followng equton ( x x + x x + x x x ) 4 y + = x8 ( x + x + x + x + x + x + x ) 8 y + 2 = x8 ( ) 2 y3 = y 500 number of nput vlue x ε (0,) re rndomly generted nd the correpondng y re clculted ung the bove equton. All the lgorthm tken for compron re umed to hve the network tructure wth 9 neuron n the nput lyer ncludng b, 5 neuron n the hdden lyer nd 3 neuron n the output lyer. All the lgorthm ncludng propoed fxed wth the mnmum error of MSE The reult obtned re tbulted n Tble 3. It how tht the lgorthm BP nd BPM converge to MSE n 590 epoch nd 389 epoch wthn 62 mec nd 487 mec repectvely. But MBP converge to the termnton condton wth the mxmum 30

11 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 of 75 epoch wthn 89 mec. The propoed lgorthm converge quckly n 25 epoch wthn 5 mec. Tble 3. Compron tble for the nonlner functon pproxmton problem. ALGORITHM PARAMETERS EPOCHS TRAINING TESTING TIME MSE MSE IN MSECS BP µ= BPM µ=.5 α= MBP µ=0.25 λ= Propoed λ= The lernng curve of the propoed lgorthm hown n Fgure 6. Another et of 500 pttern re generted for tetng. The tetng MSE obtned for the propoed nd for the MBP MSE Epoch Fgure 6. Lernng curve bed on MSE nd Epoch of Non lner functon pproxmton problem for the propoed lgorthm Ir dt et The Ir dt [5], one of the bet known dtbe n the pttern recognton lterture. The dt et contn three cle. Ech cl h 50 ntnce, totlly 50 pttern re ued. Among 75 pttern re ued for trnng nd the remnng for tetng. All the vlue re normlzed by dvdng the vlue by 0. The network tructure condered 5-0- ncludng one b n the nput lyer. Tble 4 how the reult obtned for ll the lgorthm tken for compron. 3

12 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Tble 4. Compron tble for the Ir dt et problem. ALGORITHM PARAMETERS EPOCHS TRAINING TESTING TIME MSE MSE IN MSECS BP µ= BPM µ=.5 α= MBP µ=0.25 λ= Propoed λ= The propoed lgorthm nd MBP tke mnmum epoch of 95 nd 368 nd mnmum tme of 33 mec nd 43 mec repectvely. But BP nd BP wth momentum requre 49 nd 44 epoch nd 93 mec nd 65 mec repectvely to rech the termnton condton MSE Alo the tetng MSE obtned for the propoed lgorthm mnmum. The lernng curve drwn gnt epoch nd MSE for the propoed lgorthm hown n Fgure MSE Epoch Fgure 7. Lernng curve bed on MSE nd epoch of Ir dt et problem for the propoed lgorthm. 4. Concluon An effcent technque for dptng the lernng rte n modfed bckpropgton lgorthm for trnng equentl FNN propoed. Here, the lernng rte dpted bed on the dfferentl lner nd nonlner error of output nd hdden lyer. Seprte dptve lernng rte ued for both hdden nd output lyer n ech terton. The tme requred to fx the lernng rte by trl nd error ved. The propoed lgorthm mprove the convergence peed n term of tme nd epoch whch hown by multng four dfferent problem. The mn dvntge of the propoed lgorthm no need to put effort to tune the lernng prmeter to obtn optml convergence. The propoed lgorthm ey to mplement nd ey to compute lernng rte for both hdden nd output lyer whch modfe the vlue of weght nd ncree the convergence peed. The lernng curve how tht the convergence gurnteed. 32

13 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Reference [] Abd S, Fnech F, Nm M, (200), A ft feedforwrd trnng lgorthm ung modfed form of the tndrd bckpropgton lgorthm, IEEE Trn. Neurl Network [2] Ahmd M, Slm F.M, (992) Superved lernng ung cuchy energy functon, Proc. 2nd Int. Conf. Fuzzy logc neurl network, lzuk, Jpn, [3] Beher L, Kumr S, Ptnk A, (2006) On dptve lernng rte tht gurntee convergence n feedforwrd network, IEEE Trn. Neurl Network [4] Bengo S, Bengo Y, Clouter J, (994) Ue of genetc progrmmng for the erch of new lernng rule for neurl network, Proc.IEEE World Congr. Computtonl Intellgence nd Evolutonry, [5] Fher R.A, (936) The ue of multple meurement n txonomc problem, Annul Eugenc [6] Jcob R.A, (988) Increed rte of convergence through lernng rte dptton, Neurl network [7] Jeong S.Y, Lee S.Y, (2000) Adptve lernng lgorthm to ncorporte ddton functonl contrnt nto neurl network, Neurocomputng [8] Kthrvlvkumr T, Subvth S.J, (2009) Neghborhood bed modfed bckpropgton lgorthm ung dptve lernng prmeter for trnng feedforwrd neurl network, Neurocomputng [9] Ro R, (996) Neurl network : ytemtc ntroducton,. Berln. Sprnger verlg; [0] Rumelhrt DE, Hnton GE, Wllm RJ, (986) Lernng nternl repreentton by error propgton, Prllel dtrbuted proceng: explorton n the mcrotructure of cognton, Cmbrdge(MA): MIT Pre; [] Sed Irnmneh, Amn Mhdev M, (2009) Dfferentl dptve lernng rte method for bck propgton neurl network, World Acdemy of Scence, Engneerng nd Technology [2] Srkr D, (995) Method to peed up error bck propgton lernng lgorthm, ACM Comput. Surv., [3] Sh D, Bc V.B, (999) Adptve on-lne ANN lernng lgorthm nd pplcton to dentfcton of non-lner ytem, Informtc [4] Vn Ooyen A, Nenhu B, (992) Improvng the convergence of the bckpropgton lgorthm, Neurl network, [5] Xe S, Zhng C, (2006) Vrble lernng rte LMS bed lner dptve nvere control, Journl of nformton nd computng cence [6] Yu C.C, Lu B.D, (2002) A bckpropgton lgorthm wth dptve lernng rte nd momentum coeffcent, Proc.Int.Jont Conf. Neurl network(ijcnn'02), [7] Yu X.H, Chen G.A, Cheng S.X, (993) Accelerton of bckpropgton of lernng ung optmzed lernng rte nd momentum, Electron.Lett, 29(4) [8] Yhy H. Zwer, (2006) Optmzton of three term bckpropgton lgorthm ued for neurl network lernng, Interntonl ournl of Computtonl Intellgence [9] Zhhong Mn, Hong Ren Wu, Sophe Lu, Xnghuo Yu, (2006), A new dptve bckpropgton lgorthm bed on Lypunov tblty thory for neurl network, IEEE Trncton on Neurl Network

14 Interntonl Journl of Computer Scence, Engneerng nd Applcton (IJCSEA) Vol., No.5, October 20 Author T.Kthrvlvkumr receved M.Sc. degree n Mthemtc from Mdur Kmr Unverty n 986, Pot Grdute Dplomo n Computer Applcton from Bhrthdn Unverty n 987, M.Phl. degree n Computer Scence from Bhrthr Unverty n 994 nd the Ph.D. degree n Computer Scence from Unverty of Mdr n Snce 987 he h been workng Lecturer, currently Aocte Profeor n Computer Scence t V.H.N.Senthkumr Ndr College, Vrudhungr, Tmlndu, Ind. H reerch nteret nclude Neurl Network nd Applcton, Pttern recognton nd Dt Mnng. S.Jeyeel Subvth receved the MCA degree from Mdur Kmr Unverty, n 998 nd M.Phl degree n Computer Scence, from Mother Tere Women' Unverty n From Jnury 2000 to Aprl 2007 he worked Lecturer n Computer Applcton t SFR College, Ind. Snce July 2007 he h been workng Lecturer n Informton Technology, Sr Klwr College, Svk, Tmlndu, Ind. At preent he doctorl cnddte n the Deprtment of Computer Scence t Mdur Kmr Unverty, Ind. Her re of nteret nclude Neurl Network nd Dt tructure nd lgorthm. 34