Naveen Kumar Sharma et al, / (IJCSIT) International Journal of Computer Science and Information Technologies, Vol. 2 (2), 2011,

Transcription

1 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, Performance Evauaton Anayss of MLP & DG-RBF Feed Forward Neura Networs for Pattern Cassfcaton of Handwrtten Engsh Curve Scrpts Naveen umar Sharma, S R Pande and Manu Pratap Sngh *. CET-IILM-AHL, nowedge Par II, Greater Noda, Inda,. Department of Computer Scence, SSES Amt's Scence Coege, Congressnagar, Nagpur, Inda. Department of Computer Scence, ICIS, Dr. B. R. A. Unversty, handar, Agra, Inda Abstract- The purpose of ths study s to evauate the performance anayss of mutayer feed forward neura networs traned wth bac propagaton agorthm & descent gradent Rada bass functon networ for the pattern cassfcaton of hand wrtten curve scrpt. Ths anayss has been done for handwrtten text of three etters and for the ndvdua Engsh vowes. Ths anayss n the performance has been evauated for the fve dfferent sampes of handwrtten Engsh vowes and handwrtten text of the three etters. Evauaton process s executed upon raw data n bnary form and data based on extracted features (tangent vaues & densty vaue for each word & vowes. These characters are presented to the neura networ for the tranng. Adustng the connecton strength and networ parameters perform the tranng process n the neura networ. The resuts of 600 experments ndcate that the feed forward MLP performs accuratey and exhaustvey wth mposed DG-RBF method. ey words- Pattern Cassfcaton, Feed forward neura networ, Bac propagaton agorthm, Rada Bass Functon Neura Networ. I. INTRODUCTION An artfca neura networ (ANN s a we-estabshed technque for creatng the artfca ntegence n the machne. Ths s an attempt to smuate the human behavor n the machne for the varous pattern recognton tass []. Neura networs consst of computer programmabe obects caed as neurons. These neurons are programmed to perform a smpe mathematca functon or to process a sma porton of data. A neuron s nterconnected wth other neurons wth the connecton strength nown as weght. These weghts of the neura networ are adustabe n nature to adept the behavor of nput pattern nformaton. Thus, by adustng the weghts of the networ, the behavor of the neura networ can be atered and controed. Ths mechansm n neura networ system s nown as earnng. Neura networs have been used n a number of appcatons such as pattern recognton & cassfcaton [,, 4, 5], remote sensng [6], dynamc modeng and medcne [7]. The ncreasng popuarty of the neura networs s party due to ther abty to earn and generazaton. Partcuary, feed forward neura networ maes no pror assumpton about the statstcs of nput data and can construct compex decson boundares [8]. Ths property maes neura networs, an attractve too to many pattern cassfcaton probems such as hand wrtten curve scrpts [9, 0, ]. The pattern cassfcaton of handwrtten text and numeras has been a doman of great mportance for the researchers due to the arge avaabty of data and to mae the processes wor faster [ 4]. The empoyment of few systems e Optca character recognton [5] for Prnted text and OMR [6] sheets for examnaton forms have reduced the cost of operaton and aso tme to process ths data. Such systems requre arge nowedge base and ntegent technoogy that can functon correcty even when dstorted/ modfed nput s present to t. For ths purpose Artfca Neura Networs have been used to mpement such systems. Snce ast 5 decades varous methods and technooges have been deveoped to use Artfca Neura Networs (ANN n the doman of handwrtng recognton [7]. The ncreasng popuarty of the neura networs s party due to ther abty to earn and generazaton n partcuar, the feed forward neura networ refers to pror assumpton about the statstcs of nput data and can construct compex decson boundares [8]. Many systems have been deveoped by researchers for handwrtng recognton usng mut ayer Perceptron (MLP [9-], Hdden Marov modes and feed forward neura networs [, 4, 5-7]. The terature s supped wth hgh precson recognton systems for aready segmented handwrtten numeras and characters [8-9]. However, research nto the recognton of characters extracted from cursve and touchng handwrtng does not have the same measure of achevement [0]. One of the maor probems faced when deang wth segmented, handwrtten Engsh character recognton s the ndstnctness and egbty of the characters. The accurate recognton of segmented characters s very mportant n the context of segmentaton based, word recognton []. There are dfferent types of archtectures and desgns for the neura networs, but here we dscuss the most common one,.e. feed forward manner. In a feed forward neura networ the nodes are organzed nto ayers; each "staced" on each other. The neura networ conssts of an nput ayer of nodes, one or more hdden ayers, and an output ayer []. Each node n the ayer has one correspondng node n the next ayer, thus creatng the stacng effect. The nput ayer's nodes conssts wth output functons those dever data to the frst hdden ayers nodes. The hdden ayer(s s the processng 9

2 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, ayer, where a of the actua computaton taes pace. Each node n a hdden ayer computes a sum based on ts nput from the prevous ayer (ether the nput ayer or another hdden ayer. The sum s then "compacted" by an output functon (sgmod functon, whch changes the sum down to more a mted and manageabe range. The output sum from the hdden ayers s passed to the output ayer, whch exhbts the fna networ resut. Feed-forward networs may contan any number of hdden ayers, but ony one nput and one output ayer. A snge-hdden ayer networ can earn any set of tranng data that a networ wth mutpe ayers can earn []. However, a snge hdden ayer may tae onger to tran. There are numerous agorthms have been proposed to mprove the bac propagaton earnng agorthm. Snce, the error surface may have severa fat regons; the bac propagaton agorthm wth fxed earnng rate may be neffcent. In order to overcome wth these probems, Voge et. a. [4] and Jacobs [5] proposed a number of usefu heurstc methods, ncudng the dynamc change of the earnng rate by a fxed factor and momentum based on the observaton of the error sgnas. Yu et. a. proposed dynamc optmzaton methods of the earnng rate usng dervatve Informaton [6]. Severa other varatons of bac propagaton agorthms based on second order methods have been proposed [7-4]. Artfca Neura Networs (ANNs have payed a wonderfu roe n achevng remunerabe resuts. The one essenta subset of ANN s Rada Bass Functon Networ and ts generazaton. The mportant aspect of the RBFN s the dstncton between the technques of updatng the frst and the second ayers weghts. Varous technques have been proposed n the terature for optmzng the Rada Bass functons such as unsupervsed methods e seecton of subsets of data ponts [4], orthogona east square method [4], custerng agorthm [44], Gaussan mxture modes [45] and wth the supervsed earnng method []. The RBF networ has one hdden ayer of Gaussan functons, whch are combned neary by the output nodes. In eary stage, the parameters of RBF networs were usuay estmated n two phases: Gaussan parameter estmaton by custerng and weght earnng by error mnmzaton. Snce the custerng procedure does not consder the dvsbty of patterns, the Gaussan parameters earned ths way do not ead to good cassfcaton performance. A substanta mprovement s to adust a the parameters smutaneousy by error mnmzaton [46]. Ths maes the RBF networ compettve wth the MLP n cassfcaton accuracy. In ths paper, we consder two neura networs archtectures (NN & NN. The NN s traned wth the conventona bac propagaton earnng agorthm wth ncorporaton of momentum terms & Doug s Momentum descent term [47]. The NN networ archtecture has been mpemented wth the Rada bass functon [48] n the snge hdden ayer. Ths networ ncorporates the steepest gradent descent for weght updates. The performance of these two networ archtectures has been anayzed aganst three types of nput patterns. These nput patterns are n the form of bnary mage matrx, tangent vaues matrx for the handwrtten curve scrpts of three etters and the densty vaues for the handwrtten Engsh vowes. The networs anayzed to fgure out the networ that exhbts hgher performance resuts wth greater effcency. Every networ s assessed based on the rate of convergence and speed of determnaton of the convergence weghts for the every pattern. The experments are conducted wth 600 sampes of Engsh words of three characters and fve sets of handwrtten characters of Engsh vowes. The sgnfcant mprovement n the performance of the networ has been acheved for the pattern cassfcaton of the handwrtten vowes as we as for the recognton of handwrtten words. The next secton presents the mpementaton of the neura networ archtecture wth Rada bass functon. The smuaton desgn and agorthmc steps of the probem are represented n secton. The expermenta resuts and dscusson are presented n secton 4. Secton 5 contents the concuson of ths paper and the future research drectons. II. IMPLEMENTATION OF THE RADIAL BASIS FUNCTION There are varous methods for cassfcaton probems [49] e the handwrtten Engsh characters recognton and each of them has pros and cons. The probem specfcaton n handwrtten recognton usng ANN s bascay the optmzaton probem. Methods of nonnear optmzaton n ANNs have been studed for hundreds of years, and there s a huge terature on the subect n feds such as numerca anayss, operaton research and statstca computng [50 5]. The mut ayer neura networs (MLP archtecture has used for the constrant free non near optmzaton for the varous optmzaton probem. The pattern cassfcaton s a good exampe of ths optmzaton. The MLP usuay suffers wth the convergence probem of oca error surface and the use of second order gradent descent term n weght update has sgnfcanty mproved the performance of MLP [5] but st there s no guarantee to fnd the goba optmum. Goba optmzaton for neura nets s especay dffcut because the number of dstnct oca optma can be astronomca. Another mportant consderaton n the choce of optmzaton agorthms s that neura nets are often -condton [5], especay when there are many hdden unts. The agorthms that use ony frst-order nformaton, such as steepest descent and standard bac-propagaton are notorousy sow for condtoned probems. Generay speang, the more use an agorthm maes of second-order nformaton, the better t w behave under -condtonng. The foowng methods are sted n order of ncreasng use of second-order nformaton: conugate gradents, quas-newton, Gauss-Newton and Newton-Raphson. Due to the -condtoned nature of ANN used for handwrtng recognton; t has aso been proposed to evauate the performance of the proposed networ wth the ntroducton of gradent descent of RBF. The Gradent Descent RBF methods have been proven to be most effectve and fast convergence methods n the probem doman of supervsed earnng. In ths secton, we nvestgate a networ structure reated to the mut ayer feed forward neura 9

3 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, networ (FFNN, mpemented usng the Rada Bass Functon (RBF-MLP whch suffces the need of ocay responsve neurons. If we nterpret ths rea fe phenomena n the doman of handwrtten recognton then we can ceary see that oca approxmaton based functon can pay mportant roe n adustng weghts of nter medate ayers of an MLP desgned for pattern recognton tas and n the process of optmzaton of these ocay responsve neurons pay the roe of budng boc of goba mnma. The archtecture and tranng methods of the RBF networ are we nown [54, 55, 56, 57, 58, 59] & we estabshed. The Rada bass functon networ (RBFN s a unversa approxmator wth a sod foundaton n the conventona approxmaton theory. The RBFN s a popuar aternatve to the MLP, snce t has a smpe structure and a much faster tranng process. The RBFN has ts orgn n performng exact nterpoaton of a set of data ponts n a mutdmensona space [60]. The RBFN s havng, networ archtecture smar to the cassca reguarzaton networ [55], where the bass functons are the Green s functons of the Gram operator assocated wth the stabzer. If the stabzer exhbts rada symmetry, the bass functons are raday symmetrc as we and an RBFN s obtaned. From the approxmaton theory vewpont, the reguarzaton networ has three foowng desrabe propertes [6, 6]:. It can approxmate any mutvarate contnuous functon on a compact doman to an arbtrary accuracy, gven a suffcent number of unts.. The approxmaton has the best-approxmaton property snce the unnown coeffcents are near.. The souton s optma n a way that t mnmzes a functon that measures how much t oscates. An RBFN s a three ayer feed forward networ that conssts of one nput ayer, one hdden ayer and one output ayer as shown n Fgure, each nput neuron corresponds to a component of an nput vector x. The hdden ayer conssts of neurons and one bas neuron. Each node n the hdden ayer uses an RBF denoted wth (r, as ts non-near actvaton functon. The hdden ayer performs a non-near transform of the nput and the output ayer ths ayer s a near combner whch maps the nonnearty nto a new space. The bases of the output ayer neurons can be modeed by an addtona neuron n the hdden ayer, whch has a constant actvaton functon 0 ( r. The RBFN can acheve a goba optma souton to the adustabe weghts n the mnmum MSE range by usng the near optmzaton method. Thus, for an nput pattern x, the output of the th node of the output ayer can defne as; y ( x w ( x w0 ( for (,,...,, M where y (x s the output of the th processng eement of the output ayer for the RBFN, w s the connecton weght from the th hdden unt to the th output unt, s the prototype or centre of the th hdden unt. x x x N Fg. Archtecture of the RBFN. The nput ayer has N nodes; the hdden and the output ayer have and M neurons, respectvey. 0 ( x, corresponds to the bas. The Rada Bass Functon (. s typcay seected as the Gaussan functon that can be represented as: x ( x exp( for (,,...,, ( and for 0 (bas neuron Where x s the N- dmensona nput vector, s the vector determnng the centre of the bass functon and represents the wdth of the neuron. The weght vector between the nput ayer and the th hdden ayer neuron can consder as the centre for the feed forward RBF neura networ. Hence, for a set of L pattern pars {( x, y }, ( can be expressed n the matrx form as Y w T ( where W [ w,... w m ] s a xm weght matrx, T w ( w 0,... w,,... ] s a x L [ 0 T, [,,..., ] matrx, s the output of the hdden ayer for the th sampe,, ( x c, Y [ y, y,... ym] s a M x L matrx and T y,... y. m 0 W y y y M 94

4 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, The mportant aspect of the RBFN s the dstncton between the rues of the frst and second ayers weghts. It can be seen [47] that, the bass functons can be nterpreted n a way, whch aows the frst ayer weghts (the parameters governng the bass functon, to be determned by unsupervsed earnng. Ths eads to the two stage tranng procedure for RBFN. In the frst stage the nput data set {x n } s used to determne the parameters of the bass functons. The bass functons are then eep fxed whe the second ayer weghts are found n the second phase of tranng. There are varous technques have been proposed n the terature for optmzng the bass functons such as unsupervsed methods e seecton of subsets of data ponts [6], orthogona east square method [64], custerng agorthm [55], Gaussan mxture modes [65] and wth the supervsed earnng method. It has been observed [46] that the use of unsupervsed technques to determne the bass functon parameters s not n genera an optma procedure so far as the subsequent supervsed tranng s concerned. The dffcuty wth the unsupervsed technques arses due to the settng up of the bass functons, usng densty estmaton on the nput data and taes no consderaton for the target abes assocated wth the data. Thus, t s obvous that to set the parameters of the bass functons for the optma performance, the target data shoud ncude n the tranng procedure and t refects the supervsed tranng. Hence, the bass functon parameters for regresson can be found by treatng the bass functon centers and wdths aong wth the second ayer weghts, as adaptve parameters to be determned by mnmzaton of an error functon. The error functon has consdered n equaton ( as the east mean square error (LMS. Ths error w mnmze aong the decent gradent of error surface n the weght space between hdden ayer and the output ayer. The same error w mnmze wth respect to the Gaussan bass functon s parameter as defned n equaton (.. Thus, we obtan the expressons for the dervatves of the error functon wth respect to the weghts and bass functon parameters for the set of L pattern pars ( x, y as; where to L. w E (4 w E (5 E and (6 here, and y E M ( d y w ( x (7 x and ( x exp( Hence, from the equaton (4 we have, E E y E w. x w y y E s. w ( x or..exp( w s y M ( x = ( d y. s. exp( So, M ( x w d y. s exp( ( Now, from the equaton (6 we have E E y. y ( E x x. w.exp(.( y Or ( M x ( x ( d y. s w.exp( (8 (9 Now, from the equaton (6 we have E E y. y = y ( x x. w exp( E or, ( x exp( M ( d x y. s. w. (0 95

5 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, So that, we have from equatons (8, (9 & (0 the expressons for change n weght vector & bass functon parameters to accompsh the earnng n supervsed way. The adustment of the bass functon parameters wth supervsed earnng represents a non-near optmzaton probem, whch w typcay be computatonay ntensve and may be prove to fndng oca mnma of the error functon. Thus, for reasonabe we-ocazed RBF, an nput w generate a sgnfcant actvaton n a sma regon and the opportunty of gettng stuc at a oca mnmum s sma. Hence, the tranng of the networ for L pattern par.e. ( x, y w accompsh n teratve manner wth the modfcaton of weght vector and bass functon parameters correspondng to each presented pattern vector. The parameters of the networ at the m th step of teraton can express as; M ( x w( m w( m ( d y. s ( y.exp( ( M ( m ( m ( d y. s. w. that some of the man advantages of the rada bass functon networ, s fast two-stage tranng and nterpretabty of the hdden unt representaton. Hence, among the neura networ modes, RBF networ seems to be qut effectve for pattern recognton tas such as handwrtten character recognton. Snce t s extremey fexbe to accommodate varous and mnute varatons n data. Now, n the foowng subsecton we are presentng the smuaton desgned and mpementaton detas of reda bass functon wored as a cassfer for the handwrtten Engsh vowes and recognton of handwrtten Engsh curve scrpts of three etters. III. SIMULATION DESIGN AND IMPLEMENTATION DETAILS The experments descrbed n ths secton were desgned to evauate the performance of feed forward neura networ when evoved wth the bac propagaton agorthm for MLP & RBF networ wth decent gradent method. To accompsh ths tas two neura networs are consdered namey NN and NN. NN s the conventona MLP and NN s the MLP wth RBF mposed n ts earnng sequence. The probem doman has been dvded nto two parts. In the frst part the tas ( x. assocated to the neura networs n both experments was to accompsh the tranng of the handwrtten Engsh anguage x vowes n order to generate the approprate cassfcaton. For ( ths, frst we obtaned the scanned mage of fve dfferent types of sampes of handwrtten Engsh anguage vowes as ( shown n Fgure. After coectng these sampes, we parttoned an Engsh vowe mage n to four equa parts and cacuated the densty of the pxes, whch beong to the ( m ( m centra of gravtes of these parttoned mages of an Engsh vowe. Le ths, we w get 4 denstes from an mage of M x handwrtten Engsh anguage vowe, whch we use to provde ( d y. s. w. ( x. the nput to the feed forward neura networ. We use ths procedure of generatng nput for a feed forward neura ( networ wth each sampe of Engsh vowe scanned mages. where, & are the coeffcent of earnng rate. The dscussed gradent decent approach for mpementaton of RBFNNs system s ncrementa earnng agorthm n whch the parameters update for each exampe ( x, y. The RBFNNs traned by the gradentdecent method s capabe of provdng the equvaent or better performance compared to that of the mut ayer feed forward neura networ traned wth the bac propagaton. The gradent decent method s sow n convergence snce t cannot effcenty use the ocay tuned representaton of the hdden ayer unts. When the hdden unt receptve feds, controed Fg Scanned mages of fve dfferent sampes of handwrtten Engsh anguage vowes. by the wdth are narrow for a gven nput ony a few of In the second part the compete set of experments can be the tota number of hdden unts w be actvated and hence sub-dvded nto two segments. In the frst segment nput ony these unts need to be updated. Thus, there s no vector s of the dmensons 50x, and for the second segment guarantee that the RBFNN remans ocazed after the of experment, the nput vector s of x. For the frst supervsed earnng [55]. As a resut the computatona segment of experments, a three systems are smuated usng advantage of ocaty s not utzed. Indeed, n numerca a neura networ system that consst of 50 neurons n nput smuatons t s found that the subset of the bass functons ayer, 0 neurons n hdden ayer and 6 output neurons. Each may evove to have very broad responses. It has been reazed networ has 50 nput neurons that are equvaent to the nput 96

6 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, character s sze as we have reszed every character nto a bnary matrx of sze 5x0. Character s mage s acheved by appyng the segmentaton technque [5]. The dstngushng factors among FF-MLP (NN and RBF-MLP (NN are that n the case of FF-MLP the networ contans Log-Sgmod transfer functons whereas RBF-MLP contans RBF transfer functon as shown n Fg.. For the second set of experments a networ arrangements are havng nput neurons, 0 hdden neurons and 6 output neurons. The transfer functons used to propagate the weght update among the ayers are same as dscussed above. The nput neurons correspond to tangent vaues of each word s mage. The 6 output neurons correspond to 6 etters of Engsh aphabet. The number of hdden neurons s drecty proportona to the system resources. The bgger the number more the resources are requred. The number of neurons n hdden ayers s ept 0 for optma resuts. The 65 word sampes are gathered form 5 subects of dfferent ages ncudng mae and femae for the nput patterns. After the preprocessng modue 600 nput sampes were consdered for tranng. Each sampe was presented to the networ 6 tmes ( types of nput patterns for each sampe mutped by networ archtectures thus 600 experments have been conducted. archtectures have evauated for the cassfcaton of handwrtten Engsh vowes and n the second phase the performance has been evauated for the recognton of handwrtten curve scrpts of three etters. The parameters used for both experments n both the probem domans are descrbed n Tabe and. TABLE I PARAMETERS USED FOR BAC PROPAGATION ALGORITHM Parameter Vaue Bac propagaton earnng Rate 0. Momentum Term 0.9 Doug s Momentum Term Adapton Rate.0 Mnmum Error Exst n the Networ MAXE Inta weghts and based term vaues Randomy Generated Vaues Between 0 and TABLE II PARAMETERS USED FOR DECENT GRADIENT -RBF ALGORITHM. (a Parameter Vaue Bac propagaton earnng Rate 0. Momentum Term 0.9 Doug s Momentum Term Adapton Rate.0 Spread parameter.0 Mean of nputs( c Between maxmum & mnmum vaues Mnmum Error Exst n the Networ MAXE Inta weghts and based term vaues Randomy Generated Vaues Between 0 & (b Fg. Networ archtecture of MLP wth RBF n Hdden Layer for (a Frst set of experments where bnary nput n form of 50x matrx s used. And (b Second set of experments where tangent vaues based nput n the form of x matrx s used A. Experments As we have mentoned that two sets of experments were executed. In each of the sets same type of networ archtecture was used. The probem doman has been dvded n two phases. In the frst phase the performance of both the networ B. RBF Impementaton n the Neura Networ Archtecture for the frst phase The frst neura networ (NN structura desgn was based on feed forward mutayer generazed perceptron. Four nput unts have been used, wth one hdden ayers of sx numbers of neurons and fve numbers of neurons n output ayer. The second neura networ (NN structura desgn was aso based on a competey connected feed forward mutayer generazed perceptron. But four nput unts have been used, wth snge hdden ayer of sx neurons and fve neurons n 97

7 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, output ayer. The NN networ s empoyng the sgmod functon for generatng the output sgna from the processng eements of a the hdden ayers and output ayer. The NN s usng the same sgmod functon for the processng eements of output ayer, but the Gaussan form of rada bass functon s used for the hdden ayer eements. The hdden ayers were empoyed to nvestgate the effects wth bac propagaton and decent gradent-rbf woud have on the hyper pane. The MLP networ has a snge output ayer wth the foowng actvaton and output functons for the pattern vector ( x, y y 0 w s ( q (4 h h and, s f [ y ] f [ w s ( q ] (5 0 for (,,...,, M and (,,..., L, where functon f [ y ] can defne as, s (6 y e Now, smary, the output and actvaton vaue for the neurons of hdden ayers and nput ayer can be wrtten as, q h N 0 w x and, s ( q f [ q ] f [ w x ], h h h N 0 for h (,,..., (7 sh ( q h ( q e h (8 h h In the Bac propagaton earnng agorthm the change n weghts are beng done accordng to the cacuated error n the networ, after each, teraton of tranng. The change n weghts and error n the networ can be cacuated as, E w t w t w w t (9 w E Where N E w t w t L d s d s w (0 t ( s the squared dfference between the actua output vaue and the target output vaue of output ayer for pattern. Here, we have used the doug s momentum term [47] wth momentum descent term for cacuatng the change n weghts n eqn. (9 & (0. Doug's momentum descent s smar to standard momentum descent wth the excepton that the pre-momentum weght step vector s bounded so that ts ength cannot exceed (one. After the momentum s added, the ength of the resutng weght change vector can grow as hgh as / ( - momentum. Ths change aows stabe behavor wth much hgher nta earnng rates, resutng n ess need to adust the earnng rate as tranng progresses. Now, n the decent gradent earnng for the RBF networ the change n weghts and bass functon parameters can be computed as; w M x ( d y. s exp( w ( t ( ( t w ( t ( ( t ( d y. s. w x.exp( x.( ( t ( ( t ( and M x x ( d y. s. w.exp(.( ( t ( ( t ( t (4 98

8 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, Here, agan we are usng the Doug s Momentum Term wth momentum decent term for cacuatng the change n weghts and bass functon parameters. The reda bass functon networ has the snge output and hdden ayer wth the foowng output functons for the pattern vector ( x, y. y and w ( x s f [ y ] f [ w ( x ] (5 (6 for (,,..., M & (,,..., N Where functon f [ y ] can defne as; s y e x and ( x exp( (7 C. RBF Impementaton n the Neura Networ Archtecture for the Second phase In ths phase of the probem doman we consder both test and tranng patterns. Handwrtng word sampes from 5 peope of dfferent ages and genders mosty unversty and schoo students, were coected. Each wrter was ased to wrte 5 sampe characters n bocs made n the request form. Each of the 5 forms was scanned and word s mages were extracted. The nput patterns for frst set of experments are prepared based on method dscussed n [5]. A bref of ths method s:- Segment each word s mage usng vertca segmentaton technque. Reshape each character s mage n to 5x0 bnary matrx. Resze the mage nto 50x matrx. 4 Repeat and for a three characters and cub together n 50x matrx to form a sampe. 5 Repeat -4 for a words to create 600 sampes. The nput sampe for one word from above referred technque can be depcted as n Fg. 4. Word s mage s bnarzed and reszed to 80x560. Each mage s then dvded nto equa sub-mages of the sze 70x70. For better understandng the (, nd sub-dvson s shown n the foowng Fg. 5. Fg 5 Dvdng the character s mage nto equa parts. After ths dvson, the bnary sub-mage s vertcay (Coumn Based traced to ft a ner curve. Durng the vertca readng of ths sub-mage we fnd bac pxes, for these bac pxes row numbers are summed and dvded by tota number of bac pxes n a partcuar coumn. AvgRowNum = (/n* R (8 here s the coumn number beng traced, R s the row number n ths coumn that contans the bac pxe. Ths way we acheve the average row number aganst the coumn number beng traced. The process s expaned n the Fg. 6 shown beow. Fg 4 Input sampe for the frst set of experments. Fg. 6 The Coumn-Based tracng of a bnary sub-mage. Ths fgure s ust a pctora presentaton to show the process of vertca tracng. The output of ths vertca (.e. coumn based tracng s a set of ordered pars of (Co, AvgRowNum. Co represents the number of coumn number where at-east one bac pxe s found. AvgRowNum s, as aready expaned n prevous secton s the smpe average of a row numbers where the bac pxe s found. Ths data s usefu to draw a straght ne on the segmented mage of the handwrtten character. Ths 99

9 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, output set can be shown n foowng Tabe. Ths tabe shoud be read from eft to rght (row wse to get nowedge about data. TABLE III THE RESULT TABLE OF COLUMN BASED TRACING OF SEGMENTED CHARACTER WHERE COLUMN NO REPRESENTS COLUMN NUMBER WHERE BLAC PIXEL IS FOUND AND CORRESPONDING CALCULATED AVGROWNUMBER. Coumn No The above mentoned agorthm when apped to ths sub-mage we retreve both tangent vaue and correspondng ner curve. The graphca representaton of the computed data from Tabe III and ner curve s shown n foowng dagram from MATLAB. The tangent of the ner curve s computed from the straght ne as shown n Fg. 7. AvgRowNum Coumn No AvgRowNum Coumn No AvgRowNum Coumn No AvgRowNum Coumn No AvgRowNum Coumn No AvgRowNum After the vertca tracng of the mage a ner curve s ftted on ths data, and tangent of ths ner curve.e. ne s computed. Ths process s foowed for a segments. For each of the character mages, the foowng agorthm was apped to get tangent vaues:- Agorthm # : Pattern Preparaton Input: Character s Image. Resze the mage (I nto 80 X 560 pxes, usng the - Nearest Neghborhood agorthm.. Dvde the mage (Ireszed nto equa sub-mages usng standard IMCROP method of MATLAB. As shown n Fgure 5.. Covert the sub-mage (ISub nto bnary form (IBn 4. Trace IBn vertcay (Coumn - wse to get the bac pxe. 4. Add row number to SumRowCount and Increment RowCount by one f a bac pxe s found. And store correspondng Coumn Number n an Array ArrCoumnNum. 4. Cacuate the average RowNum by dvdng SumRowCount wth RowCount when row number s reached to max.e. 70 and RowCount s non-zero. 4. Store non zero vaues of RowNum n an array ArrAvgRowNum as shown n Tabe I. 5. Repeat step 4 t the a coumns are read. 6. Ft a ner poynoma on (ArrAvgRowNum, ArrCoumnNum. 7. For sampe test Coumn Numbers 0:5:70 get the vaues from poynoma. From these vaues cacuate Tangent vaue for ths poynoma. Usng the formua :- Tangent T = (Y-Y/(X-X 8. Repeat steps -7 for each of sub-mages and arrange the obtaned tangent vaues n a matrx of sze x. Output: tangent vaues correspondng to the character s mage Fg 7 Fttng a ne on segmented character mage to fnd the tangent vaue the red coored vaues are taen from Tabe and bue ne s derved from the ftted poynoma. The tangent vaues acheved from above mentoned agorthm for each word of the sampe coecton form. Each nput sampe comprses of tangent vaues for each word. Ths s best depcted n foowng matrx where Wd_Tan to Wd_Tan are tangent vaues for Frst Word, Wd_Tan to Wd_tan are tangent vaues for second word and so on for a 600 words. TABLE IV INPUT SAMPLE PREPARED FROM TANGENT VALUES OF INDIVIDUAL WORDS. The smuaton program, whch we have been deveoped n MATLAB 6.5, for testng these two networs for handwrtten Engsh anguage vowes cassfcaton probem and the recognton of handwrtten curve scrpts of three etters, generates nta weghts randomy through ts random generator but the same set of weghts have been used for both the networ archtectures. So the epochs for the agorthms w be dfferent every tme wth the same networ structure and the same tranng data set. IV. RESULTS AND DISCUSSION A. Resuts for Frst phase of probem doman (Engsh Vowes The resut presented n ths secton are demonstratng the arge sgnfcant dfference exst between the performance of 940

10 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, BPNN and DG-RBF for handwrtten Engsh anguage vowes cassfcaton probem. The smuated resuts for ths probem have been consdered for the 5 tras wth both agorthms up to maxmum mt of teratons. A the resuts of 5 tras contan fve dfferent types of handwrtten sampes for each Engsh vowes character. The tranng has been performed n such a way that repetton of same nput sampe for a character cannot be happen smutaneousy,.e. f we have traned our networ wth a nput sampe of a character then next tranng cannot be happen wth the other nput sampe of the same character. Ths nput sampe w appear for tranng after other sampe of other characters tranng. We have consdered the mean of performance wth ther best case of convergence for a the tras. The comparatve resuts are presented n Tabe V and the Fg. 7 s representng the comparson charts desgned on the bass of vaues avaabe n the tabe. TABLE V RESULTS FOR CLASSIFICATION OF HANDWRITTEN ENGLISH VOWELS USING BAC PROPAGATION FOR MLP AND DECENT GRADIENT WITH RBF NETWOR Characters Sampe Bac propagaton Epochs Sampe A E I O U Characters Sampe Sampe Sampe DG-RBF Epochs A E I O U It can observe from the resuts of tabe and graph that BPNN has converged conversng approxmatey for the 0 percent cases but the RBFN has converged for 75 percent cases. The tabe s aso showng some rea numbers. These entres represents the error ext n the networ after executng the smuaton program up to teratons.e. up to teratons the agorthm coud not converge for a sampe of a hand wrtten Engsh anguage vowes nto the feed forward neura networ. Sampe Sampe 4 Sampe 4 Sampe 5 Sampe 5 Fg 8 The Comparson Chart for Handwrtten Engsh Vowes Cassfcaton Epochs for two earnng agorthms. B Resuts for Second phase of probem doman (Three etter curve scrpts In ths phase of the probem two sets of experments were executed. In each of the sets same type of networ archtecture was used. In the frst experment set we have used conventona MLP and ths networ was traned and evauated wth two types of nput patterns (tangent vaues and bnary matrx. Ths process was executed three tmes for a two types of networs (MLP.e. NN, RBF-MLP.e. NN. In tota the number of experments conducted for tranng of a of the networs was 600 ncudng 00-experments for each type of networ. The testng of the performance of networs was done n 00 experments. Tranng and testng sampes of both nds (Bnary format 50X and tangent vaues X for a partcuar word; when presented to the Networs; yeded 6 sets of data. The performance of the partcuar networ has been evauated based on the comparson done for same sampes of data wth other networs. For testng of the networ, 5 test sampes were created by randomy seectng character mages from any of the 600 sampes. Thus the networ was traned wth 600 dfferent sets of nput patterns. The foowng tabes (tabe 6 and 7 contan epoch average of 0 teratons for 600 sampes, thus ony 60 readngs have been mentoned. Sampe depcts average epoch vaue for sampe to sampe 0; Sampe depcts average epoch vaue for sampe to sampe 0 and so on. Each sampe has been presented to two networs. Frst set of experments are conducted wth tranng and test patterns formed as bnary format of 50x. The number of teratons (epochs requred by each networ to earn the partcuar sampe was captured and an average of such 0 vaues s summarzed n Tabe VI. Ths tabe data s used to compare the earnng and convergence performance of each networ. Epoch data captured for ths set of experments are shown n Tabe VI. 94

11 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, TABLE VI EPOCHS OR NUMBER OF NETWOR ITERATIONS FOR THE TWO NETWOR SETTINGS RBF-MLP AND FEED FORWARD MLP USING BINARY INPUT SAMPLES OF 50X SIZES. Sampes Epochs wth MLP Epochs Wth RBF-MLP Sampes Epochs Wth MLP Epochs Wth RBF-MLP Sampe 7 5 Sampe Sampe 48 Sampe Sampe 9 8 Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe 84 9 Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Fg. 9 Comparson chart for the two networs usng bnary nput sampes of 50x szes. The presence of n above mentoned tabe shows that the maxmum number of epochs has been reached but the networ dd not converge to the desred output; due to the error exsts n the networ. The graphca representaton for both networs n terms of epochs has been dspayed n Fg. 9. Presences of maxmum epoch vaues n ths graph represent the poweressness of networ to earn the behavor n specfed mtaton of teratons. Ony 67 sampes were earnt by the MLP and RBF-MLP coud mae up to 49 sampes. Same set of networs was aso examned for ts performance aganst the other type of tranng and test sampes. The sampes used for ths experment executon phase was n the form of tangent vaues for each word. Thus 600 words mae X600 sampe szes. The number of teratons or epochs used to earn the behavor was captured for each word then smpe average has been cacuated. The resuts for the performance of these two networs for ths data have been presented n the Tabe VII. 94

12 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, TABLE VII Epochs or number of teratons for the two networs settngs wth tangent vaues for each nput sampe of x szes Sampes Epochs wth MLP Epochs Wth RBF-MLP Sampes Epochs Wth MLP Epochs Wth RBF-MLP Sampe 7 Sampe 9 56 Sampe 9 9 Sampe Sampe 5 5 Sampe Sampe 4 64 Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe 9 84 Sampe Sampe Sampe Sampe 55 9 Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe Sampe For ths set of experments aso, maxmum number of epochs assgned to the networs s We can see n Tabe 7 that error st exsts n the networ, because of the presence of n above mentoned tabe agan. But at the same tme, t s ceary evdent that n ths set of experments, a networs have performed better as they have converged n fewer epochs. The graphca representaton for both networs n terms of epochs has been dspayed n Fg. 0. The networs have acheved better performance n ths case of experments. It s evdent from the data captured n tabe 7 that a networs have earnt more sampes n ess teraton. As compared to tabe 6, ths tme FF-MLP has earnt 48 sampes; whereas, RBF-MLP earnt 54 sampes. Fg. 0 Comparson chart for the three networ settngs usng tangent vaues for each nput sampe of x szes V. CONCLUSIONS. We have consdered the two probems n ths research wor. The frst probem s for the cassfcaton of handwrtten Engsh vowes and the second probem s for the recognton of handwrtten curve scrpts of three etters wth the two 94

13 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, technques of feature extractons. The resuts descrbed n ths paper ndcate that, for the handwrtten Engsh anguage vowes cassfcaton probem, feed forward neura networ traned wth bac propagaton agorthm does not perform better n comparson to feed forward neura networ traned wth decent gradent wth RBF. We found that, n each and every case, the DG-RBF networ gves better resuts for the cassfcaton of Engsh vowes, n comparson to the bac propagaton for the MLP networ. It has been aso observed that the RBF networ has aso stuc n oca mnma of error for some of the cases. The reason for ths observaton s qute obvous, because there s no guarantee that RBFNN remans ocazed after the supervsed earnng and the adustment of the bass functon parameters wth the supervsed earnng represents a non-near optmzaton, whch may ead to the oca mnmum of the error functon. But the consdered RBF neura networ s we ocazed and t provdes that an nput s generatng a sgnfcant actvaton n a sma regon. So that, the opportunty s gettng stuc at oca mnma s sma. Thus the number of cases for DG-RBFNN to trap n oca mnmum s very ow. The drect appcaton of DG-RBF to the handwrtten character cassfcaton has been expored n ths research. The am s to ntroduce an aternatve approach to sove the handwrtten character cassfcaton probem. The resuts from the experments conducted are qute encouragng and refect the mportance of rada bass functon for the optma cassfcaton to the gven probem. The expermenta resuts for the second probem aso confrm that the DG-RBF networ resuts n hgh performance n terms of recognton rate and cassfcaton accuracy, at the same tme competey emnatng the substtuton error. The presented resut demonstrate that, wthn the smuaton framewor presented above, arge sgnfcant dfference exsts between the performance of Bacpropagaton feed-forward neura networ and BG-RBF for handwrtten Engsh words recognton probem. The resuts descrbed n ths paper ndcate that the handwrtten Engsh anguage words cassfcaton probem, feed-forward neura networ traned wth Bacpropagaton agorthm does not perform better n comparson of feed-forward neura networ traned wth DG- RBF. The performance of DGRBF s effcent and accurate n a the smuatons. The hgher speed of convergence n the DG-RBF tranng process suggests that ths archtecture may not be fascnated n the fase mnma of the error surface. It may aso mnmze the possbtes of mscassfcaton for any unnown testng nput pattern. Nevertheess, more wor need to be done especay on the tests for arge compex handwrtten characters. Some future wors shoud aso be expored. For nstances, current wor s showng the performance of DG-RBF over bac propagaton for MLP networ n cassfcaton process up to handwrtten Engsh anguage vowes and the recognton of handwrtten curve scrpts of three etters but we can proceed further to use ths dea for more compex handwrtten character recognton probems n whch the character are not of the same sze and may be not havng the same rotaton ange. The conugate decent for the weghts between hdden ayer and output ayer and for the parameters of bass functon can aso be cacuated to ncrease the performance of the networ for convergence as the extenson wor. REFERENCES [] M. Redmer, Rprop - Descrpton and Impementaton Detas Technca Report, Unversty of arsruhe: W-768 arsruhe (994. []. Fuushma and N. Wae, "Handwrtten aphanumerc character recognton by the neocogntron," IEEE Trans. on Neura Networs, ( (99. [] Mansh Manga and Manu Pratap Sngh, Anayss of Cassfcaton for the Mutdmensona Party-Bt-Checng Probem wth Hybrd Evoutonary Feed-forward Neura Networ. Neurocomputng, Esever Scence, ( (007. [4] D. Aha and R. Banert, Coud cassfcaton usng error-correctng output codes, Artfca Integence Appcatons: Natura Resources, Agrcuture, and Envronmenta Scence, ( -8 (997. [5] Y.L. Murphey, Y. Luo, Feature extracton for a mutpe pattern cassfcaton neura networ system, IEEE Internatona Conference on Pattern Recognton, (00. [6] L. Bruzzone, D. F. Preto and S. B. Serpco, A neura-statstca approach to muttempora and mutsource remote-sensng mage cassfcaton, IEEE Trans. Geosc. Remote Sensng, ( (999. [7] J.N. Hwang, S.Y. ung, M. Nranan, and J.C. Prncpe, The past, present, and future of neura networs for sgna processng'', IEEE Sgna Processng Magazne, 4( (997. [8] C. Lee and D. A. Landgrebe, Decson boundary feature extracton for neura networs, IEEE Trans. Neura Networs, 8( 75 8 (997. [9] C. Apte, et a., "Automated Learnng of Decson Rues for Text Categorzaton", ACM Transactons for Informaton Systems, - 5 (994. [0] Mansh Manga and Manu Pratap Sngh, Handwrtten Engsh Vowes Recognton usng Hybrd evoutonary Feed-Forward Neura Networ, Maaysan Journa of Computer Scence, 9( (006. [] Y. Even-Zohar and D. Roth, A sequenta mode for mut cass cassfcaton, In EMNLP-00, the SIGDAT. Conference on Emprca Methods n Natura Language Processng, 0-9 (00. [] S. N. Srhar, Recognton Of Handwrtten and Machne prnted text for posta address Interpretaton, Patterns Recognton Letters, 4 (99. [] T. Gergey,. rsof and Csaba, Anaogca preprocessng and segmentaton agorthm for offne hand wrtng recognton, Journa of Crcuts Systems and Computers, ( (00. [4] S. Procter, J. Ingworth, F. Mohtaran. Cursve Handwrtng Recognton usng Hdden Marov Modes and a excon drven eve budng agorthm, IEEE Proc Vson, Image and Sgna Processng, 5-67 (000. [5] P. Schneder. Computer asssted speng normazaton of 8th century Engsh. Language and Computers, 6 (00. [6] F. Rossant and I. Boc, Robust and adaptve OMR system ncudng Fuzzy modeng, fuson of musca rues, and possbe error detecton, EURASIP J. App. Sgna Process (007. [7] R. Pamondon and S. N. Srhar. On-Lne and Off-Lne Handwrtng Recognton: A Comprehensve Survey. IEEE Trans. Pattern Ana. Mach. Inte. (000. [8] M. Ofer, "Decson Regon Connectvty Anayss: A Method for Anayzng Hgh-Dmensona Cassfers", Machne Learnng, 48 (00. [9] V. Aessandro, P. Mchae, "Combnng Onne and Offne Handwrtng Recognton", ICDAR, Seventh Internatona Conference on Document Anayss and Recognton (ICDAR'0 844 (00. [0] R. Paacos and A. Gupta, A System for Processng Handwrtten Ban Checs Automatcay, MIT Soan, (00. [] N. Gors, V. Ansmov, E. Augustn, O. Baret, D. Prce, J. Smon, AA Chec Reader: A famy of ban chec recognton systems, ICDAR -99. Proceedngs of the Ffth Internatona Conference on Document Anayss and Recognton, 5 56 (999. [] S. Lee, Off-Lne Recognton of Totay Unconstraned Handwrtten Numeras Usng Mutayer Custer Neura Networ, IEEE Trans. Pattern Ana. Mach. Integence 8 ( (996. [] J. Ferrez, J. Ortega-Garca, D. Ramos, and J. Gonzaez-Rodrguez. HMM-based on-ne sgnature verfcaton: Feature extracton and 944

14 Naveen umar Sharma et a, / (IJCSIT Internatona Journa of Computer Scence and Informaton Technooges, Vo. (, 0, sgnature modeng, Pattern Recognton Letter, 8 (6 5-4 (007. [4] D. X, S. W. Lee, Extracton of reference nes and tems from form document mages wth compcated bacground, Pattern Recognton 8 ( (005. [5] M. H. Shaheed, "Feedforward neura networ based non-near dynamc modeng of a TRMS usng RPROP agorthm", Arcraft Engneerng and Aerospace Technoogy, 77 (- (005. [6] A. E. Hassan, J. Wu and R. C. Hot, Vsuazng Hstorca Data Usng Spectrographs In Proceedngs of the th IEEE nternatona Software Metrcs Symposum METRICS, IEEE Computer Socety, Washngton, (005. [7] C. Y. Suen, and R. Legaut, C. Nada, M. Cheret and L. Lam, Budng a New Generaton of Handwrtng Recognton Systems, Pattern Recognton Letters (99. [8] S. B. Cho, Neura-Networ Cassfers for Recognzng Totay Unconstraned Handwrtten Numeras, IEEE Trans. on Neura Networs, (997. [9] H. Yamada and Y. Naano, Cursve Handwrtten Word Recognton Usng Mutpe Segmentaton Determned by Contour Anayss, IEICE Transactons on Informaton and Systems, 79 (E (996. [0] F. mura, N. ayahara, Y. Myae and M. Shrdhar, Machne and Human Recognton of Segmented Characters from Handwrtten Words, 4th Internatona Conference on Document Anayss and Recognton (ICDAR 97, Um, Germany, (997. [] P. D. Gader, M. Mohamed and J. H. Chang, Handwrtten Word Recognton wth Character and Inter-Character Neura Networs, IEEE Transactons on Systems, Man, and Cybernetcs-Part B: Cybernetcs, (997. [] Martn M. Anthony, Peter Bartett, Learnng n Neura Networs: Theoretca Foundatons, Cambrdge Unversty Press, New Yor, NY, (999. [] M. Wrght, Desgnng neura networs commands s and savvy, EDN, 6(5, (99. [4] T. P. Vog, J.. Mangs, W. T. Zn, A.. Rger, D. L. Aon "Acceeratng the Convergence of the Bac Propagaton Method, Bo. Cybernetcs, (988 [5] R. A. Jacobs, Increased Rates of Convergence through Learnng Rate Adaptaton, Neura Networs, (988. [6] X.H. Yu, G.A. Chen, and S.X. Cheng, Dynamc earnng rate optmzaton of the bacpropagaton agorthm, IEEE Trans. Neura Networ, (995. [7] Stansaw Osows, Potr Boarcza, Mace Stodos, Fast Second Order Learnng Agorthm for Feedforward Mutayer Neura Networs and ts Appcatons, Neura Networs, 9( (996. [8] R. Battt, "Frst- and Second-Order Methods for Learnng: Between Steepest Descent and Newton's Method". Computng: archves for nformatcs and numerca computaton, 4( 4-66 (99. [9] S. Becer, & Y. Le Cun, Improvng the convergence of the bacpropagaton earnng wth second order methods, In D. S. Touretzy, G. E. Hnton, & T. J. Senows (Eds., Proceedngs of the Connectonst Modes Summer Schoo. San Mateo, CA: Morgan aufmann, 9-7(988. [40] M. F. Moer, A Scaed Conugate Gradent Agorthm for Fast Supervsed earnng, Neura Networs, (99. [4] M.T. Hagan, and M. Menha, "Tranng feed-forward networs wth the Marquardt agorthm", IEEE Transactons on Neura Networs, 5( (999. [4] P. Petra and A. Chd, "Emprca evauaton of feature subset seecton based on a rea-word data set", Engneerng Appcatons of Artfca Integence, 7( (004. [4] M. Berare, M. Thémans, "Deang wth snguartes n nonnear unconstraned optmzaton", European Journa of Operatona Research, 96( -4 (009. [44] L. Janhua and B. Laeh, An Effectve Custerng Agorthm for Mxedsze Pacement, Proc. of ISPD, Austn, (007. [45] N. Shenta, A. Bar-he and D. Wensha, "Computng Gaussan Mxture Modes wth. EM usng Sde-Informaton", n Proc. of Int. Conference on Machne Learnng, ICML-0, Washngton DC, (00. [46] C. M. Bshop. Neura Networs for Pattern Recognton. Oxford Unversty Press, (995. [47] P. Chrstenson, A. Maurer & G. E. Mner, handwrtten recognton by neura networ, csc.mrs.umn.edu/ UMMCScW/ pub/ CSc4555s04/ InsertTeamNameHere/ handwrtten. pdf (005. [48] M. J. D. Powe, Rada bass functons for mutvarabe nterpoaton: A revew, n Agorthms for Approxmaton of Functons and Data, J. C. Mason, M. G. Cox, Eds. Oxford, U..: Oxford Unv. Press, 4-67 (987. [49] Guobn Ou, Y Lu Murphey: Mut-cass pattern cassfcaton usng neura networs. Pattern Recognton, 40 ( 4-8 (007. [50] D. P. Bertseas, Nonnear Programmng, Bemont, MA: Athena Scentfc, ISBN (995. [5] D. P. Bertseas and J. N. Tstss, Neuro-Dynamc Programmng, Bemont, MA: Athena Scentfc, ISBN (996. [5] V. S. Dhaa and M. P. Sngh, Handwrtten character recognton usng modfed gradent descent technque of neura networs and representaton of conugate descent for tranng patterns, Internatona Journa of Engneerng, Iran, ( (009. [5] S. Saarnen, R. Bramey and G. Cybeno, "I-condtonng n neura networ tranng probems", Sam J. of Scentfc Computng, (99. [54] J. Moody and C.J. Daren. Fast earnng n networs of ocay-tuned processng unts, Neura Computaton, ( 8-94 (989. [55] T. Poggo, and F. Gros, Reguarzaton agorthms for earnng that are equvaent to mutayer networs, Scence, (990b. [56] M. T. Musav, W. Ahmed,. H. Chan,. B. Fars, D. M. Hummes, On the tranng of rada bass functon cassfers, Neura Networs, 5( (99. [57] D. Wettscherec, T. G. Detterch, Improvng the performance of rada bass functon networs by earnng center ocatons. In J. E. Moody, S. J. Hanson, and R. P. Lppmann, (Eds. Advances n Neura Informaton Processng Systems, San Mateo, CA: Morgan aufmann, 4-40(99. [58] W. P. Vogt, Dctonary of Statstcs and Methodoogy: A Nontechnca Gude for the Soca Scences, Thousand Oas: Sage (99. [59] S. Hayn, Neura Networs, Macman Coege Pubshng Company, Inc, New Yor (994. [60] D. S. Broomhead and D. Lowe, "Mutvarabe functona nterpoaton and adaptve networs", Compex Syst, -55(988. [6] T. Poggo, F. Gros, Networs for approxmaton and earnng, Proc IEEE, 78( (990. [6] F. Gros, T. Poggo, Networs and the best approxmaton property, Bo Cybern (990. [6] M.A. raaved and R.P.W. Dun, Generazaton capabtes of mnma erne-based networs, Proc. Int. Jont Conf. on Neura Networs (Seatte, WA, Juy 8-, IEEE, Pscataway, U.S.A., I-84 - I- 848(99. [64] S. Chen, C. F. N. Cowan, and P. M. Grant, Orthogona east squares earnng agorthm for rada bass functon networs, IEEE Transactons on Neura Networs, ISSN , ( 0-09(99. [65] C. M. Bshop, Novety detecton and neura networ vadaton, IEEE Proceedngs: Vson, Image and Sgna Processng. Speca ssue on appcatons of neura networs, 4 (4 7 (994b. 945