Formulation for Second Derivative of Generalized Delta learning Rule in Feed Forward Neural Networks for Good Generalization

Transcription

1 Internton Journ of Scentfc & nneern Reerch, Voume 4, Iue 8, Auut 03 ISSN Formuton for Second Dervtve of Generzed Det ernn Rue n Feed Forwrd Neur Networ for Good Generzton 7 Shvpue Ntn Prpp DR Mnu Prtp Snh Shr Ventehwr Unvert, Inttute of Computer & Informton Scence Gru, AmrohUttr Prdeh, Dr BRAmedr Unvert, Ar-800 Ind Uttr Prdeh, Ind e-m: hvpuentn@mcom e-m: mnu_p_nh@hotmcom Atrct Generzed det ernn rue ver often ued n muter feed forwrd neur networ for ccomph the t of pttern mppn A cpropton ernn networ expected to enerze from the trnn et dt, o tht the networ cn e ued to determne the output for new tet nput Th networ ue the rdent decent technque to trn the networ for enerzton It evove the tertve procedure for mnmzton of n error functon, wth dutment to the weht en mde n equence of tep The frt dervte of the error wth repect to the weht dentfe the oc error urfce n decent drecton Therefore for ever dfferent preented pttern, the networ exht the dfferent oc error nd the weht modf n order to mnmze the current oc error In th pper, we re provdn the enerzed mthemtc formuton for the econd dervtve of the error functon for the rtrr feed forwrd neur networ topoo The new o error pont cn evute wth the hep of current o error nd the current mnmzed oc error The weht modfcton proce ccomphe two tme, one for the preent oc error nd econd tme for the current o error The propoed method ndcte tht the weht, thee re determne from the mnmzton of o error re more optm wth repect to the conventon rdent decent pproche Index Term Generzton, Pttern mppn networ, Bc propton ernn networ, decent rdent, nd Conute decent IJSR INTRODUCTION Generzton dfferent from nterpoton, nce n enerzton the networ expected to mode the unnown tem or the mut er perceptron It cn o e pped to the error functve, cn e pped to mn other nd of networ nd not ut functon from whch the trnn et dt h een otned The ton other thn ut the mpe um-of-qure, nd the evuton of other dervtve uch the Jcon nd Hen metrc proem of determnton of weht from the trnn et dt nown the odn proem The enerzton performnce depend on the ze nd compext of the trnn et, ecuted dervtve cn e tced un vert of optmzton nd o, the econd te of weht dutment wth the cde the rchtecture of the networ nd the compext of the cheme proem Therefore, f the performnce for the tet dt Depte the ener ucce of c-propton method n the ood for the trnn dt, then the networ d to hve ernn proce, ever mor defcence re t needed to e enerzed from the trnn dt The feed forwrd neur networ rchtecture more common ued to perform the t of of error, pon nd over trnn The converence rte of oved e converence urntee nd converence rte, nture enerzton wth pttern mppn networ In th pttern c-propton ver ow nd hence t ecome unute for mppn networ for enerzton the orthm for modfn re proem Furthermore, the converence ehvor of the the weht etween the dfferent nterconnected er uu c-propton orthm depend on the choce of nt vue of connecton weht nd other prmeter ued n the o- nown c propton ernn technque 3 Th orthm uperved ernn method for mut ered feed forwrd rthm uch the ernn rte nd momentum term There re neur networ It eent rdent decent oc optmzton technque whch nvove cwrd error correcton of neted dfferent reercher 5-7 for mprovn the trnn eff- vrou other enhncement nd modfcton were o preentwor weht It evute the dervtve of the error functon cenc of neur networ ed orthm ncorportn the wth repect to weht n weht pce for n ven preented eecton of dnmc ernn rte nd momentum 8-9 The mprovement n performnce of c propton further conder nput pttern from the ven trnn et 4 It nvove the tertve procedure for mnmzton of n error functon, wth dutment to the weht en mde n equence of tep In ech ven the dervtve of the networ output wth repect to wth the evuton of the Jcon mtrx, whoe eement re tep we cn dtnuh etween two dtnct te In the frt the nput The Jcon mtrx provde meure of the oc te, the dervtve of the error functon wth repect to the entvt of the output to chne n ech of the nput vre weht mut e evuted In the econd te, the dervtve In ener, the networ mppn repreented trned neur re then ued to compute the dutment to e mde to the networ w e non-ner, nd o the eement of the Jcon mtrx w not e contnt ut depend on the prtcur nput vec- weht 0 The frt te proce, nme the propton of error cwrd throuh the networ n order to evute dervtor ued IJSR 03

2 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN The c propted error further ued to evute the econd dervtve of the ntntneou qured error Thee dervtve form the eement of the Hen mtrx 3, whch nvove the of ft procedure for re-trnn feed forwrd networ foown m chne n the trnn dt Thu, due to the ever ppcton of the Hen mtrx, there vrou pproxmton cheme hve een ued to evute t e the don pproxmton 4 outer product pproxmton nd nvere Hen 5 The exct evuton of the Hen mtrx h o propoed, whch vd for networ of rtrr feed forwrd topoo Th method ed on n extenon of the technque of the c propton ued to evute the frt dervtve, nd hre the mn dere feture of t 6 The econd dervtve of the error wth repect to weht otned conute decent method 7, 8 The further mprovement n conute decent method o condered 9 nd n the fm of Qu Newton orthm 0 Further, the nfuence of n w tude few reercher 3 The n prmeter contro the teepne of the ctvton functon It h een hown tht rer n vue h n equvent effect of ncren the ernn rte Recent t h een ueted tht mpe modfcton to the nt erch drecton, e the rdent of error wth repect to weht, cn utnt mprove the trnn effcenc It w dcovered tht f the rdent ed erch drecton oc modfed n vue ued n the ctvton functon of the correpondn node, nfcnt mprovement n the converence rte cn e cheved 4 In th preent pper, we re condern muter fed forwrd neur networ wth trnn et of nh phet Th neur networ trned for ood enerzton wth enerzed econd dervtve det ernn rue for the tochtc error Th rndom error c propted mon the unt of hdden er, for the modfcton of the connecton weht n order to mnmze the error Th modfcton n the weht preformed wth enerzed econd dervtve of error wth repect to weht etween hdden er nd output er nd o n etween nput nd hdden er The neur networ trned for cpturn the enerzed mpct functon retonhp etween nput nd output pttern pr Thu, t expected from the dptve neur networ tht t coud e to reconze the ndvdu chrcter from the hndwrtten nh word of three etter Hence, the propoed method provdn the enerzed w for mnmzton of optm o error whch cont wth ntntneou unnown mnmum oc error FD FORARD NURAL NTOR ITH DLTA LARN- ING RUL IJSR n f Z A mut er feed forwrd neur networ norm cont wth the nput, output nd hdden er The procen unt n the output er nd hdden er uu contn non ner dfferente output functon nd the unt of nput er ue the ner output functon If the unt n the hdden er nd n the output er re non-ner, then the numer of unnown weht or connecton trenth depend on the numer of unt n the hdden er, ede the numer of unt n the nput nd the output er Ovou, the networ uppoe to ue for enerzton e pttern mppn Thu, the pttern mppn, proem IJSR 03 nvove determnn thee weht, for the ven trnn nputoutput pttern pr hown n fure Fure : Archtecture of the mut er feed forwrd Neur Networ So fr, n order to determne the weht n upervor mode t necer to now the error etween the derved or expected output nd the ctu output of the networ for ven trnn pttern e now the dered output on for the unt n the fn output er, not for the unt n the hdden er Thu, the me error of output er c propted for the hdden er to ude the updtn or modfcton n the weht Therefore, the ntntneou error cn mnmze updtn the weht etween the nput er to hdden er nd hdden to output er Thu, we ue the pproch of rdent decent on the error urfce n the weht pce to dut the weht to rrve t the optmum weht vector The error defned the qured dfference etween the dered output nd the ctu output otned t the output er of the networ due to ppcton of n nput pttern from the ven nput-output pttern pr The output h ccuted un the current ettn of the weht n the er foow: here f the output functon, the output of hdden er nd Z the weht etween hdden nd output er And, o for hdden er procen unt output ; Z here f n V X X the output of nput er nd V the weht etween nput nd hdden er The ntntneou qured error for the th pttern cn repreent : 5 t 3 0 here t dered output Thu, for ech nput-output pttern pr the networ h the dfferent error So, we hve the oc error for the ven nputoutput pttern pr The weht re updted to mnmze the current oc error or unnown ntntneou qure error for ech preented nputoutput pttern pr Thu, the optmum weht m e otned

3 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN f the weht re duted n uch w tht the rdent decent mde on the tot error urfce The error mnmzton cn e hown ; ' t f z 4 eht modfcton on the hdden er cn e defned ; ' V * t f z V v 5 Let, ' = t And we hve f ' V = * f z 6 Thu the weht updte for output unt cn e repreented ; t t t t 7 here t the tte of weht mtrx t terton t IJSR 03 t the tte of weht mtrx t next terton 3 GNRALIZD SCOND DRIVATIV FOR FD FORARD t the tte of weht mtrx t prevou terton IJSR MULTILAYR NURAL NTORS: In th ecton we preent enerzed method for otnn the econd dervtve of the decent rdent of o error wth repect to weht n weht pce for the ood enerzed ehvor t current chne/ modfcton n weht mtrx nd tndrd momentum vre to cceerte of the feed forwrd muter neur networ for the ven trnn et Therefore to otn the optm weht vector for the feed ernn proce Th vre depend on the ernn rte of the networ A the networ ed the et ernn rte the forwrd neur networ, the weht modfcton houd perform for the o mnmum pont mon the vrou oc error mnm pont Thu, the modfcton n the weht vector n ech momentum vre tend to cceerte the proce tep for mnmzn frt the oc ntntneou qure error In th proce the weht re updtn on ech tep for the ntntneou oc unnown qure error nted of et men qure error Now, we utrte the enerzed method for de- nd n econd tep to mnmze the current o or et men error or tot error or o error for the entre trnn et The termnn the econd dervtve of ntntneou error nd further determnton of the tot error urfce cnnot e nown, ecue for the current o error correpond to preented nput-output the et of nput-output pttern pr re nd contnuou pttern pr on ech tep The error cn otn for the feed forwrd neur networ hown n f for the th pttern from Thu, the rdent decent on ech tep otned on the ntntneou oc error urfce Hence the weht re now duted n mnner tht the networ edn towrd the mn- the equton 3 nd decent rdent for the ntntneou qure error otned from the equton 4 Now from thee m of oc error urfce for the preented nput-output pttern two equton we hve; pr Therefore to otn the optmum weht vector for the ven trnn et, the weht mut modf n mnner tht the networ houd ed towrd the mnmum of o error e the expected vue of the error functon for the trnn mpe = 3 Now, we conder the econd dervtve for the error 3; A we cn oerve from the fure, tht the output vector of dmenon nd o for we hve the -dmenon error urfce The networ w ed towrd the decent rdent of - dmenon error urfce Now, nother nput-output pttern pr e + repreent nd m hve nother error urfce whch dfferent from the frt one Therefore the weht of the networ w where J ctvton from the output er unt further modfed per equton 5 for the error of + pttern e + Hence, th proce w contnue for ever preented pttern pr nd we m hve the dfferent error urfce On or one error urfce t one tme w ctvte for the preented n- put pttern It e to nterpret tht, ever tme the networ tre to mnmze the current oc unnown error Now, t cer tht n decent rdent ernn rue weht of the networ re updtn on on the of oc error rther thn the o error nd n order to otn the enerzed ehvor the updtn n the weht re requred on the decent rdent of the o error or et men qure error for the entre trnn et Hence, t cn reze tht to otn the ood enerzton for the ven nput output pttern pr the weht updtn houd te pce for the o error rther thn the oc error urfce Therefore, we cn vuze - dmenon error urfce n whch we hve dfferent decent rdent correpondn to dfferent nput output pttern pr, ut on one decent rdent w ctvte t one tme So, t dffcut to eep the entre oc decent rdent nd to erch for the o one Inted of th, we cn eep the dfferent mnm pont of the error correpondn to dfferent nput output pttern pr Thee mnm pont w dtrute n the entre error urfce nd to trce the o mnm from thee oc mnmum pont w e nd convenent Thu, to ccomph the determnton of oc mnmum pont, we cn conder the econd dervtve of decent rdent of the oc error

4 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN IJSR 03 =, nce 0 Hence, 3 e further extend the dervtve term from the equton 3 ; ; here nd the non ner dfferente output n defned ; exp f So,we hve For the unt of the output er, where 33 So tht, 34 Thu, the weht dutment cn otn correpondn to the mnm pont of error ; 35 Correpondn the new weht etween the procen unt of hdden nd output er cn otn ; t t 36 Now, we determne the weht modfcton etween the procen unt of nput er nd hdden er of feed forwrd neur networ hown n f, n order to mnmze the me oc error nd to otn the oc mnm pont of the error An we w conder the econd dervtve of the error wth repect to the weht ; 37 So, on utrton of the term we hve; here I ctvton from the hdden er unt And f the pped nput on the th unt of nput er Hence,, nce 0 38 e further expnd the dervtve term from the equton 38 ; here exp f n output n wth = So tht, So, we hve = IJSR 74

5 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN IJSR 03 = Snce, 0 = = 39 here Or, nce = 0 nd = 0 Becue the output n of hdden er unt ndependent of the chne n the ctvton of output er unt Hence, 30 Now, from equton 38 we hve, 3 Hence, from equton 37 nd 3 we hve; 3 Thu, the new weht etween the procen unt of nput er nd hdden er cn otn ; t t 33 So tht, here we hve otned the weht modfcton for the feed forwrd neur networ n order to mnmze the oc error for the preented nput-output pttern pr The econd dervtve of oc error h een ccuted eprte wth repect to the weht etween hdden nd output er, nput nd hdden er Now, we re determnn the econd dervtve of the me oc error wth repect to the weht of nput nd hdden er nd hdden nd output er n comnton Thu, n we conder the oc error nd the rdent decent of error urfce n weht pce ; nd h 0 34 here 0 h repreent the one weht from ech hdden nd output er So tht, Hence, the weht chne ; ho 35 Hence, from the ove mentoned expreon we cn otn weht modfcton n term of econd dervtve of error wth repect to weht of the hdden-nput er nd output-hdden er The weht modfcton h een otned for the unt of hdden er n the term of c propted error nd for the unt of output er n the term of oc error enerted from the unt of output er The weht modfcton h o een otned for the one weht from the ech hdden nd output er Thu, for ech nput output pttern pr of the trnn et the weht vector w ncrement updte ccordn to the ernn IJSR 75

6 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN equton e t t 36 Th proce w contnue for the ech preented nput- output pttern pr Here we hve otned the mnmum pont of ech oc ntntneou qured error determnn t econd dervtve In th w, we hve the coecton of oc error mnmum pont n -dmenon error urfce Now we cn determne the o mnmum pont tn the qure men of current oc error pont wth the current qure men of the error pont e the current o or tot error pont Let, ntze the o error wth zero e mn = 0 nd determne the oc ntntneou error pont correpond to the preented nput output pttern pr e ' ', The current o error pont cn determne ; mn mn / 37 Now, the current weht of the networ w further updte per the equton 35, 3 & 35 to mnmze the current o error mn Th proce w contnue for the preented nput-output pttern pr of ven trnn et nd ever tme once the econd dervtve of ntntneou oc error for the preented pr h The foown m- mn otned the IJSR w modf Thu, the mnmum o error word t heht e thn hf of the w chne wth ever current unnown econd dervtve n pe depct the me phenomenon: decent drecton for ntntneou oc error, nd the ncrement weht updte w preformed to mnmze the current o error Thu the proce of updtn of weht w ccomph two tme Frt tme for the econd dervtve of oc error nd further the econd dervtve of current o error So tht, n th pproch the trnn h performed to mnmze the o error The o error h otned n tertve dnmc fhon wth the econd dervtve of ntntneou oc error Hence, we hve otned the optm weht vector for the mut er feed forwrd neur networ to cpture enerze mpct functon Here, we re condern the retonhp etween nput- output pttern pr of ven trnn et 4 SIMULATION DSIGN AND RSULT: IJSR 03 In th muton the performnce of mut er feed forwrd neur networ trned wth enerzed econd dervtve of o error ernn rue nzed the ood enerzton for the ven trnn et The trnn et cont wth nh phet n nr form nput pttern wth correpondn nr output pttern nformton The enerzed econd dervtve of ntntneou error ued to mnmze the current o error whch me the networ more converent nd how the remre enhncement n the performnce The 000 tet mpe word re preented to the vertc ementton prorm whch dened n MATLAB nd ed on porton of vere heht of the word Thee emented chrcter re cued toether fter nrzton to form trnn pttern for neur networ The propoed enerzed econd dervtve det ernn rue mnmzn the current o error for ech preented nput output pttern pr The networ dened to ern t ehvor preentn ech one of the 0 mpe 00 tme thu cheved 000 tr The reut ndcte the nfcnt mprovement n the performnce of the networ To ccomph the muton wor we conder the feed forwrd neur networ tem whch cont of 50x0x6 neuron n nput, hdden nd output er repectve 000 tr hve een conducted wth ppn dfferent nd of contrnt for the ementton The contrnt re ed on the heht of the word The emented chrcter re rezed onto 5x0 nr mtrxe nd re expoed to 50 nput neuron The 6 output neuron correpond to 6 etter of nh phet The foown tep hve een nvoved for the experment : 4 Preprocen: Th tep condered mndtor efore ementton nd nzn the optm performnce of the neur networ for reconton A hnd wrtten word re cnned nto r ce me ch word ftted nto rectne ox n order to e extrcted from the document nd th w the cn contrute nto the ccuton of heht nd wdth 4 The ementton Proce: The oerved vere heht H nd v v nd wdth me the for mpementton of ementton proce It we oerved tht n curve hnd wrtten text, the chrcter re connected to ech other to form H v Fure : Connecton etween the chrcter of the curve word * H v Avere of heht for decdn ementton pont ch word trced vertc fter convertn the r ce me nto nr mtrx Th nrzton done un oc operton on r ntent eve : I = I >= Leve Here 0<= Leve <= the threhod prmeter Th Leve ed on the r-ce ntent of the text n document More ntent ed to the more threhod vue The udment of ementton pont ed on foown orthm: Aorthm: VertcSement I : Repet for ech coumn n me mtrx I trtn from I 0, 0 poton : Repet for ech row eement n coumn 3 Chec f I, 0 c pxe nd row numer > Heht / then 3 Chec f coumn < 5 OR - t ementton coumn < 5 then Proce the next eement 3 e Store th ementton pont coumn numer 4 If no c pxe found n entre coumn then t emen-

7 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN tton pont 5 Cut the me n correpondence to ementton pont dentfed of ernn nd the men vue of the tr of ther performnce h een ued the fn reut for the repreentton The foown te nd rph re exhtn th performnce n: Fure 3: Vertc Sementton Technque nd Bnr chrcter 43 Rehpe nd Rezn of Chrcter for pttern creton: ver emented chrcter frt chned nto nr mtrx nd then rezed to 5x0 mtrxe un neret nehorhood nterpoton nd rehped to 50x oc vector o tht t cn e preented to the networ for ernn Such chrcter re cued toether n mtrx of ze 50, 6 to form the trnn pttern et 44 xperment Reut: To nze the performnce of feedforwrd neur networ wth conute decent for the pttern reconton proem the foown prmeter hve een ued n the experment: Sr Prmeter Nme Vue No Lernn/ Trnn Go for 0000 entre networ Accepte o rror Momentum Term IJSR Smpe Smpe Smpe3 Smpe4 Smpe5 Smpe 4 Mxmum poch Fure 4: The Compron chrt for rdent vue etween 5 Int eht nd ed Rndom enerted vue etween The reut n the te 3 re repreentn the epoch of trnn Cc method nd enerzed econd dervtve method term vue 0 nd Te : Prmeter nd ther vue ued n ernn procee After the ementton technque pecfed the orthm the foown numer of pttern hve een otned for the trnn Sementton Contrnt Correct Semented ord Out of 000 Incorrect Semented ord Out of 000 Succe Percente Heht / % Te : Reut of Vertc Sementton Technque Thu, out of 000 word mpe the 78 word hve een correct emented nd ued the pttern for the trnn of neur networ The neur networ h een trned wth conventon decent rdent method nd from the propoed enerzed econd dervtve of ntntneou error method wth dnmc men of the o error The performnce of the networ h een n The vue of rdent decent nd propoed enerzed econd dervtve method re computed for ech tr IJSR 03 Smpe Te 3: Compron of rdent vue nd rror of the networ poch poch Cc Method Cc Method Second dervtve method nd the preence of error n the networ for cc rdent decent method nd propoed enerzed econd dervtve decent rdent method The reut hown here re the men of the tr The propoed enerzed econd dervtve decent rdent method for the hndwrtten word reconton hown the remre enhncement n the performnce 5 CONCLUSION: Networ rror Second Cc Methoddervtve Method Second dervtve Method Smpe Smpe Smpe Smpe Smpe Th pper foown the pproch of enerzed econd dervtve of ntntneou qured error for t mnmzton n the weht pce correpond to the preented nput-output pttern pr to exht ood enerzed ehvor to the fed forwrd neur networ for the ven trnn et The weht modfcton hd conder for the hdden & output er nd nput & hdden er, ede th the econd dervtve of the oc error h otned wth repect to oth the weht e hdden nd output er The foown oervton were mde for the entre dcued procedure The enerzed econd dervtve rdent method enerte the mnmum of unnown ntntneou error n - dmenon error urfce The weht hve modfed for ech of th error Thee modfcton n the weht were otned wth

8 Internton Journ of Scentfc & nneern Reerch Voume 4, Iue 8, Auut-03 ISSN the enerzed econd dervtve of th ntntneou qure error The enerzed econd dervtve of the error h otn wth repect to the weht for output er & hdden er ndvdu nd o n comnton The propoed method for ood enerzton otnn the pont of mnmum oc error for ever nput pttern durn the trnn n ech tep Once the ntntneou oc error mnmum h otned, the qure men of th oc error mnmum wth current o error h otned Th exht the current o dnmc error on ech tep Further, the weht re n modfed wth enerzed econd dervtve for the current o error Thu, the networ h trned for the o ehvor rther thn the ndvdu oc ehvor whch repreent the ood enerzed ehvor of the neur networ Th tertve proce contnue t the o error doe not mnmze for the preented nput output pttern pr of trnn et 3 The more experment of compete pttern nd n re t needed for compete verf the method The compext of the orthm houd o nze nd compre wth the other method Thee cn conder the extended or future wor Reference IJSR A L Bum nd R Rvet, Trnn 3- node neur networ NP-compete, Neur Networ, D R Huh nd B G Horne, prore n uperved neur networ: ht new nce Lpmnn, I Sn Procen mzne, m, R J nd Zper, D 995 Grdent-ed ernn orthm for recurrent networ nd ther computton compext In Chuvn, Y nd Rumehrt, D, edtor, Bc-propton: Theor, Archtecture nd Appcton, chpter 3, pe Lwrence rum Puher, Hde, NJ 4 Prer,D B,985 Lernn oc MIT Spec report TR-47 MIT Centre for Reerch n Computton conomc nd Mnement Scence, Mchuett nttute of technoo, Cmrde, MA 5 FAHLMAN, S 988 An emprc tud of ernn peed n c-propton networ Tech Rep CMU-CS-88-6, Crnee Meon Unvert, Ptturh, PA 6 Roerto Bttt, Frt- nd econd-order method for ernn: etween teepet decent nd Newton' method, Neur Computton, v4 n, p4-66, Mrch 99 7 Jco, RA 988 Increed rte of converence throuh ernn rte dptton Neur Networ,, er M, A method for ef-determnton of dptve ernn rte n c propton, Neur Networ, 4 99, Yu X H, Chen G A, Chen S X, Acceerton of c propton ernn un optmzed ernn rte momentum ectronc Letter, Rumehrt, D, Hnton, G, & m, R J 986 Lernn ntern repreentton error propton In D Rumehrt, & J L McCend d, Pre dtruted procen: xporton n the mcrotructure of conton Vo : Foundton pp Cmrde, MA: MIT Pre Bhop, C M Neur Networ for Pttern Reconton New Yor: Oxford Unvert Pre 995 Jco et 99, Jco, R, Jordn, M, Nown, S, Hnton, G 99 Adptve mxture of oc expert Neur Computton, 3, Bhop, C M 99 xct ccuton of the Hen mtrx for the muter perceptron Neur Computton 44, S Bcer nd Y Le Cun, "Improvn the converence of c-propton ernn wth econd order method", Proceedn of the 988 IJSR 03 Connectont Mode Summer Schoo, pp 937, Sn Mteo, Morn ufmnn, H, B nd tor, DG econd order dervtve for networ prunn: optm rn ureon, n Advnce n Neur Informton procen tem, 564-7, 993, Sn Mteo, CA: Morn ufmn 6 Buntne, L nd weend AS Computn econd dervtve n feed-forwrd networ: revew I Trncton on Neur Networ, 53, Fetcher R nd Powe M J D, A rpd converent decent method for nnmzton Journ of Brth Computer, Fetcher R nd Reeve R M, Functon mnmzton conute rdent, Journ of Computer, Hetene M R nd Stefe, Method of conute rdent for ovn ner tem, Journ of reerch NBS Hun H Y, A unfed pproch to qudrtc converent orthm for functon mnmzton, Journ of Optm Theor, 5 970, Thmm G, Moernd F, nd Feer, The nterchnet of ernn rte n Gn n Bc Propton Neur Netwro, Neur Computton Dh V S nd Snh M P, Hndwrtten chrcter reconton un modfed rdent decent technque of neur networ nd repreentton of conute decent for trnn pttern, Internton Journ of nneern, IJ Trncton A: Bc, Snh M P, umr S, nd Shrm N, Mthemtc formuton for the econd dervtve of cpropton error wth non-ner output functon n feedforwrd networ, Internton Journ of Informton nd Decon Scence,