ON NEURAL NETWORK CLASSIFIERS WITH SUPERVISED TRAINING. Marius Kloetzer and Octavian Pastravanu

Transcription

1 ON NEURAL NETWORK CLASSIFIERS WITH SUPERVISED TRAINING Mriu Kloetzer nd Octvin Ptrvnu Deprtment of Automtic Control nd Indutril Informtic Technicl Univerity Gh. Achi of Ii Blvd. Mngeron 53A, Ii, , Romni Phone / Fx: +4(0) E-mil: kmriu@ delt.c.tuii.ro, optrv@delt.c.tuii.ro Atrct: A tudy on clifiction cpility of neurl network i preented, conidering two type of rchitecture with upervied trining, nmely Multilyer Perceptron (MLP) nd Rdil-Bi Function (RBF). To illutrte the clifier contruction, we hve choen prolem tht occur in rel-life experiment, when one need to ditinguih etween overlpping nd Guin ditriuted cle. An mply commented comprtive tudy i elorted etween MLP- nd RBF-type clifier, in order to revel dvntge nd didvntge encountered when the two type of neurl network rchitecture re ued. Key word: clifiction, deciion oundry, neurl network, upervied trining INTRODUCTION The pproximtion cpility of ome neurl network topologie mde them populr tool for nonliner ytem identifiction nd clifiction tk. A clifiction tk cn e regrded proce where ech preented input (pttern) i igned to one of predefined numer of cle (ctegorie). The purpoe of thi pper i to illutrte the uge of the (Multilyer Perceptron) MLP nd (Rdil-Bi Function) RBF in pttern clifiction nd to elorte comprtive nlyi of their efficiency, relying on ome relevnt ce tudie. The MLP nd RBF network re feed-forwrd rchitecture which re trined in upervied mnner nd they re oth topologie cple of univerl pproximtion (Cyenko 989; Prk et l. 99). The deign of neurl network clifier i mde in two ditinct tep, nmely the trining eion nd the uge clifier. The clifiction performed y feedforwrd neurl network cn e regrded feture extrction performed y the neuron plced in the hidden lyer followed y clifiction performed y the output neuron. The orgniztion of the mteril preented in the pper correpond to the following pln. The expoition trt with the ytemtic contruction of neurl clifier (econd ection). It continue with the ce tudie which conider clifiction tk the eprtion etween Guin ditriuted overlpping cle, for which the proility of correct clifiction uing the Byein clifier cn e etimted (Hykin 999). The ce tudie tke into ccount vriou trining prmeter nd topologie of the network, including redundncy on the network output lyer, in order to develop comprtive tudy etween MLP- nd RBF- type clifier (fourth ection). The qulity of the neurl clifier i interpreted from the point of view of the trining time nd from the point of view of the proility of correct clifiction. The orgniztion of the preented mteril cn e regrded contining the tep to e tken in order to deign neurl clifier uitle to the clifiction tk required y rel-life experiment. THEORETICAL PRELIMINARIES The neurl rchitecture ued in the two tep involved in the deign of clifier the trining eion nd the uge clifier re lightly different. The neurl rchitecture ued in the firt tep i tndrd two-lyer topology, either MLP or RBF, ut in the econd tep new lyer mut e dded (Dud et l. 973; Fukung 990; Richrd et l. 987). During the trining eion of neurl clifier et of input vector (which re relevnt for certin experiment) i preented to the network long with their correponding ctegorie, ech cl eing coded in inry mode (uully comintion of poitive vlue, nmely, nd negtive vlue, nmely -). In order to exploit the trined network clifier, new lyer with ipolr tep ctivtion function mut e dded t the output of the tndrd MLP nd RBF topologie. The numer of perceptron in the new lyer

2 i equl to the numer of output of the trined network. The rchitecture of the otined neurl clifier i preented in figure. the proility of correct clifiction (p clif ) nd p B reflect the qulity of neurl clifier. Due to the rndom nture of the input pttern, ll the numericl reult to e given repreent the men of erie of 5 experiment. Figure : Modulr rchitecture of neurl clifier The two lyer of the trined MLP or RBF network extrct feture from the preented input, nd the dded output lyer trnpoe the feture vector into code correponding to one of the predefined cle. In order to otin correct mpping, the weight mtrix of the new lyer will e equl to the unity mtrix, nd the perceptron will hve zero ie. ILLUSTRATIVE CASE STUDIES In order to tudy the qulity of the MLP- nd RBF- type neurl clifier, the following clifiction tk i conidered. The ojective i to ditinguih etween two overlpping equiprole cle. Ech cl contin two-dimenionl Guin ditriuted pttern, the firt 0 cl hving the men vector m = nd the tndrd 0 devition =, nd the econd cl hving the men 3 vector m = nd the tndrd devition =. 3 The optimum (Byein) deciion oundry for thi kind of prolem (Lippmnn 987; Hykin 999) i circle with the center locted t: m m x B = = () nd with the rdiu: r B = m m + 4 ln 3,4 () Figure preent the Byein deciion oundry nd et of 500 vector from ech cl, plotted y uing different ymol. The proility of correct clifiction of the Bye clifier w etimted y performing computer experiment which tke into ccount lrge numer of vector from ech cl (nmely million), the otined vlue eing p B 93.73%. The following two uection illutrte the development of MLP- nd RBF- type clifier, repectively, for which the proility of correct clifiction i etimted y conidering et of vector uneen during the trining phe. Both the time necery for trining the network (t trin ) nd the comprion etween Figure : The ditriution of the two cle nd the Byein deciion oundry Uge of n MLP-Type Clifier The exmple preented in thi uection illutrte the dependence of the qulity of the MLP clifier on the network rchitecture nd the trining prmeter. The MLP rchitecture to e ued in clifiction tk i two lyer feed-forwrd neurl network, with trnfer (ctivtion) function of the input nd output lyer repectively tf nd tf. There re four different MLP topologie recommended in literture, the difference etween them coniting in the trnfer function, follow: ) tf = tnigmoid function, tf = tnigmoid function; ) tf = tnigmoid function, tf = liner function; c) tf = tnigmoid function, tf = logigmoid function; d) tf = logigmoid function, tf = logigmoid function. Tle preent the reult otined y uing the four MLP topologie, ech network hving two input neuron nd one output neuron nd the me trining prmeter. otined MLP topology reult ) ) c) d) t trin () p clif (%) Tle : Reult otined uing four different MLP topologie

3 The et reult in thi ce were otined y uing the MLP clifier with tnigmoidl neuron in the firt lyer nd with liner node in the output lyer. For thi reon, ll the experiment to e preented in thi uection will ue thi rchitecture. The deign of neurl clifier which preent redundncy in the output lyer cn e ueful option in ome ce. For the propoed clifiction tk, thi would e the ce of n MLP network with more thn one neuron in the output lyer. The following reult were otined y uing network with two output liner neuron (nd two input tnigmoidl neuron): - Proility of correct clifiction, p clif = 9.6 %; - Proility to detect n error (ecurity), p detect = %; - Trining time, t trin = The uge of uch redundnt rchitecture led to more ecure clifier, the proility of correct clifiction eing cloe to the one of non-redundnt topology; the didvntge i the increed time necery for trining the network. A lrge et of experiment w performed in order to tudy the dependence of the qulity of n MLP clifier on the numer of input neuron nd on the numer of epoch ued in trining; the otined reult re preented in the next ection. In the ce of vrying the numer of input neuron it i intereting to oerve the poition of the deciion oundry of the MLP clifier veru the Byein one (figure 3). Figure 3: The oundry of the MLP clifier (dhed line) veru the Byein oundry (olid line):. MLP network with input neuron;. MLP network with 8 input neuron; c. MLP network with 0 input neuron. c Uge of n RBF-Type Clifier The tndrd topology of n RBF neurl network exhiit, on the firt lyer, collection of node with Guin-type trnfer function, the econd lyer coniting of liner neuron. The deign of n RBF neurl network cn e undertood curve-fitting prolem in high-dimenionl pce; the lerning proce i equivlent to finding urfce tht provide the et fit to the trining dt, ccording to the deired ccurcy; the pred of rdil i function determine the moothne of the pproximtion. The trining lgorithm dd neuron to the input lyer of the network until the pecified men qured error gol i met. Figure 4 revel the deciion oundry of n RBF network veru the Byein one, when uing the RBF rchitecture clifier for the previouly mentioned tk. Unlike the MLP ce, the oundry impoed y n RBF clifier h nerly the me hpe, regrdle the numer of the rdil neuron. Figure 4: The oundry of the RBF clifier (dhed line) veru the Byein oundry (olid line):. RBF network with input neuron;. RBF network with 5 input neuron A for the MLP rchitecture, the influence of trining prmeter of n RBF network on the qulity of clifiction i preented in the next ection. COMPARISON BETWEEN MLP- AND RBF- TYPE CLASSIFIERS MLP nd RBF network re oth univerl pproximtor, property which cn e ued, hown, in clifiction tk. For thi reon it i worth to develop comprtive tudy with regrd to their exploittion clifier. Therefore, RBF nd MLP clifier hve een contructed for the me clifiction tk, nmely tht one conidered in the previou ection. All the reult commented elow hve een otined y tch trining, conidering tht ech cl i known y 500 pttern. To enure the relevnce of the comprion, ll the trining condition of MLP network hve ued unique vlue for the error gol (nmely 0) nd unique vlue (nmely ) to initilize weight nd ie. Interet will firt focu on the min

4 feture of trining procedure, nd, fterwrd, emphi i plced on the qulity of the deigned clifier. All the imultion experiment were conducted under Neurl Network Toolox provided in Mtl oftwre (The MthWork Inc. 00) Comment on Network Trining The reult of network trining re nlyzed for vriou condition ued in the lerning proce. Thu, for MLP topology, the chieved men qured error i regrded the reult of trining (which depend on two key prmeter: the numer of trining epoch nd the numer of tnigmoidl neuron). For RBF rchitecture, the numer of rdil neuron i regrded the reult of trining (which depend on two key prmeter: the men qured error gol nd the pred of rdil function). Thi point of view in undertnding the role of the trining condition llow comprehenive interprettion ed on the three dimenionl plot given in figure 5 ( for MLP network, for RBF network). From figure 5., one cn oerve tht for the MLP clifier the reult of trining (otined men qured error) i influenced in the firt intnce y the numer of igmoidl neuron, for medium numer of epoch (greter thn 00). In order to compre the computtion requeted y the trining of MLP- nd RBF-type clifier, figure 6 diply the dependence of the trining time on the me trining prmeter previouly conidered. The numericl vlue of trining time mke ene only if ll experiment re conducted on the me computer nd in imilr condition. For MLP network, the trining time incree when the numer of igmoidl neuron nd/or the numer of trining epoch incree (figure 6.). The plot in figure 5. nd 6. hve nerly the me hpe ecue the time necery to trin n RBF network depend on the numer of rdil neuron to e dded in order to otin the deired error. Beide the proility of correct clifiction, the trining time reflect the qulity of the contructed clifier, which i dicued in the next uection. Figure 5: Grphicl interprettion of the key element chrcterizing the network trining:. chieved men qured error of MLP network depending on numer of epoch nd numer of igmoidl neuron;. numer of input neuron of RBF network depending on men qured error gol nd pred of rdil function. Figure 6: Dependence of the trining time on the trining prmeter:. numer of epoch nd numer of igmoidl neuron for MLP network;. men qured error gol nd pred of rdil function for RBF network.

5 Comment on Clifiction Qulity The proility of correct clifiction of the trined clifier i evluted in term of imultion reult otined for input vector which hve not een preented to the network during the trining eion. A grphicl interprettion of thee reult cn e given long the me line in the previou uection, y conidering the three-dimenionl plot depicted in figure 7. According to our current interet, the urfce plotted in thee figure reflect the dependence on the trining condition (invetigted in the previou uection) of the clifier min qulity, expreed (for oth RBF nd MLP network) the proility of correct clifiction reulting from imultion. The plot given in figure 7, together with thoe preented in figure 6, llow compring the qulity of the MLP- nd RBF-type clifier from the point of view of the proility of correct clifiction nd of the trining time, repectively. The direct viul exmintion of figure 7. how tht the qulity of MLP clifier decree y uing lrge numer of trining epoch, fct which ugget tht the ize of the trining dt et i too mll for the network to get correct generliztion. In ccordnce with figure 5. nd 7., it i worth noticing tht the mller chieved error in trining doe not gurntee tht the otined clifier i the et. Anywy, the qulity of the MLP clifier i not ignificntly ffected y the trining prmeter, the difference etween minimum nd mximum proility of correct clifiction eing mller thn %. By uing the RBF rchitecture, one cn otin etter proility of correct clifiction, ut n indequte choice of trining prmeter of the rdil i network cn led to d performnce of thi clifier. Moreover, y compring the plot in figure 5. nd 7., one cn oerve tht n RBF clifier with mny input neuron cn hve nerly the me qulity n RBF clifier with ignificntly mller numer of neuron. All the remrk referring to MLP- nd RBF- type clifier re ctully founded on precie numericl informtion, ued for contructing the grphicl plot. When uing lrge et of trining dt i ville, the MLP network generlize well, nd the reult of n ccurte trining i uully le to provide trutle informtion out the qulity of the otined clifier. The itution i preented in figure 8, uing 5000 vector from ech cl in order to trin the network. A expected, the trining time i pproximtely ten time igger thn in the previou ce. Unfortuntely, in rellife experiment lrge et of trining dt it might e difficult to otin or improper to ue ecue of the required computtion time. Figure 7: Grphicl interprettion of the neurl clifier qulity depending on the trining prmeter:. proility of correct clifiction of MLP network depending on numer of epoch nd numer of tnigmoidl neuron;. proility of correct clifiction of RBF network depending on men qured error gol nd pred of rdil function. Figure 8: Proility of correct clifiction of n MLP network (trined uing lrge et of dt) depending on numer of epoch nd numer of tnigmoidl neuron The remrk iuing from the preented comprion cn e compred with the concluion of tudy involving the uge of MLP- nd RBF- rchitecture in identifiction tk (Kloetzer et l. 00, 00).

6 CONCLUSIONS The purpoe of thi pper w to illutrte the uge of upervied trined neurl network in clifiction tk. The prolem choen in order to contruct neurl clifier reflect itution tht cn occur in rel-life experiment, nmely the need to ditinguih etween overlpping nd Guin ditriuted cle. The experiment were conducted in mnner tht mke poile the development of n mply commented comprtive tudy etween MLP- nd RBF- type clifier. The MLP network provide good qulity nd it performnce i not much influenced y the fine tuning of the trining prmeter, mking thi type of clifier the optimum tool for le experienced uer. It i poile to otin etter RBF clifier (from the point of view of complexity nd proility of correct clifiction), ut thi require either mny tet or wide experience. REFERENCES Cyenko, G., 989, Approximtion y uperpoition of igmoidl function, Mthemtic of Control, Signl nd Sytem, vol., pp Dud, R.O. nd Hrt, P.E., 973, Pttern Clifiction nd Scene Anlyi, New York: Wiley. Fukung, K., 990, Sttiticl Pttern Recognition, nd Edition, New York: Acdemic Pre. Hykin, S., 999, Neurl Network. A Comprehenive Foundtion, nd Edition, New Jerey: Prentice Hll. Kloetzer, M., Ardelen, D. nd Ptrvnu, O., 00, Developing Simulink tool for teching neurl-neted identifiction, Med 0: The 9 th Mediterrnen Conference on Control nd Automtion, pp. 63 (trct), pper on CD-ROM. Kloetzer, M., Ardelen, D. nd Ptrvnu, O., 00, Crere unei ilioteci Simulink pentru explotre retelelor neuronle în identificre, Revit Român de Informtic i Automtic, vol., no., pp Lippmnn, R.P., 987, An introduction to computing with neurl net, IEEE ASSP Mgzine, vol. 4, pp. 4. Prk, J. nd Snderg, I.W., 99, Univerl pproximtion uing rdil-i-function network, Neurl Computtion, vol. 3, pp Richrd, M.D. nd Lippmnn, R.P., 987, Neurl network clifier etimte Byein poteriori proilitie, Neurl Computtion, vol. 3, pp * * *, The MthWork Inc., 00, Neurl Network Toolox 4.0., MATLAB 6. (Relee.).