Extreme Learning Machine for Function Approximation Interval Problem of Input Weights and Biases

Transcription

1 Etreme earnng Machne or Functon Appromaton Interval Problem o Input Weght and Bae Grzegorz Dudek Department o Electrcal Engneerng Czetochowa Unvert o Technolog 4- Czetochowa, Al. Arm Krajowej 7, Poland dudek@el.pcz.czet.pl Abtract In th artcle the appromaton capablt o the etreme learnng machne tuded. Speccall the mpact o the range rom whch the nput weght and bae are randoml generated on the tted curve complet analzed. The gudance or how to generate the nput weght and bae to get good perormance n appromaton o the uncton o one varable provded. Keword etreme learnng machne; uncton appromaton; eedorward neural network I. INTRODUCTION Due to the ecellent appromaton properte eedorward neural network (FNN) are wdel ued n regreon and clacaton problem. Ther adaptve nature allow them or learnng rom oberved and generalze well n uneen eample. A unveral appromator FNN are able to model an gven uncton and nonlnear relatonhp. The tranng algorthm ued n FNN uuall emplo ome orm o gradent decent method. It known that gradent decent-baed learnng generall low and converge to local mnma. Recentl, a new learnng algorthm ha been propoed or tranng ngle-hdden-laer FNN named the etreme learnng machne (EM) []. The novelt n EM that the nput weght (lnkng the nput wth hdden laer) and hdden neuron bae need not to be adjuted. The are randoml ntated accordng to an contnuou amplng dtrbuton and ed wthout teratvel tunng. The onl parameter need to be learned are the output weght lnkng the hdden and output laer. For th reaon, EM can be mpl condered a a lnear tem n whch the output weght can be analtcall determned through mple generalzed nvere operaton o the hdden laer output matrce. The learnng peed o EM can be thouand o tme ater than tradtonal gradent decent-baed learnng. A theoretcal tude have hown, even wth randoml generated hdden node, EM wth wde tpe o actvaton uncton (AF) mantan the unveral appromaton capablt [] and t can cla an djont regon [3]. In the opnon o the author o [] EM, derent rom tradtonal learnng algorthm, tend to reach the mallet tranng error and the mallet norm o weght, whch mple good generalzaton perormance. In th work we look nde EM and analze t weght and bae. We tud how the range rom whch nput weght and bae are randoml generated aect the appromaton ablt o EM. The am o th work to provde gudance or how to generate the nput weght and bae to get good perormance n appromaton o the uncton o one varable. To acheve th, we perorm eperment ung the target uncton o varng complet. The ngle varable target uncton enable u to vualze reult and look at how the actvaton uncton o neuron compoe the ttng curve. Th gve u tp on how to generate weght. II. BASIC EXTREME EARNING MACHINE EM wa orgnall propoed b Huang et al. [] a a new learnng algorthm or the ngle-hdden-laer FNN. The man concept behnd the EM are that: () the weght and bae o the hdden node are generated randoml and are not adjuted, and () the output weght are determned analtcall. The output uncton o EM o the orm (one output cae): = = βh = h()β, () where a number o hdden neuron, h() = [h (), h (),..., h ()] the output vector o the hdden laer wth repect to the nput, and β = [β, β,..., β ] T the vector o the output weght (between the hdden laer and the output node). h () the output o the -th hdden node (or t AF), whle () t weghted output. The output uncton () the lnear combnaton o AF h (). Hdden neuron map the rom d-dmenonal nput pace to the -dmenonal eature pace H, and thu, h() a eature mappng. Th mappng nonlnear t ormed b an nonlnear pecewe contnuou AF. Derent AF ma be ued n derent /5/$3. 5 IEEE

2 hdden neuron. The mot popular AF are: gmod, Gauan, multquadrc, tep, trangular and ne uncton. The gmod AF o the orm: h ( ) = + a + b )), () where a = [a,, a,,..., a,d ] T the vector o the nput weght o the -th hdden neuron, b t ba, and a denote the nner product o a and. Charactertcall or EM the hdden node parameter, a and b, are randoml generated accordng to an contnuou probablt dtrbuton ntead o beng eplctl traned. Th proce ndependent o the tranng. The output weght β are olved b mnmzng the appromaton error: mn Hβ T, (3) where H the hdden laer output matr and T the tranng output matr. The optmal oluton to (3) : β* = Η + T, (4) where H + the Moore Penroe generalzed nvere o matr H. To mprove generalzaton perormance o EM t regularzed veron wa propoed [3]. Recent development n EM ncludng mprovng t tablt and compactne, learnng on onlne equental, mbalanced, no and mng, and ncremental EM are decrbed n [4]. III. APPROXIMATION PROPERTIES OF EM DEPENDING ON THE INTERVAS OF INPUT WEIGHT AND BIASES In th ecton we eamne EM n the uncton appromaton problem. To vualze reult we lmt to the realvalued uncton o a ngle varable. In the rt eperment we ue the SnC uncton, whch wa alo ued n [] a benchmark or EM perormance evaluaton: n( ) /, or, g = (5), or =, For brevt, we ue the ollowng acronm: : target uncton g(), : tted curve (), AF: actvaton uncton h (), II: nput nterval,.e. the nterval to whch nput are normalzed. All the mulaton are carred out n MATAB Rb envronment. For EM we ue Matlab uncton elm created b the author o EM algorthm (downloaded rom node.html). In th mplementaton the nput weght and bae are generated randoml rom the unorm dtrbuton: weght rom the range [, ] and bae orm the range [, ]. In our mulaton we ue gmod AF. In the rt eperment we repeat mulaton perormed n []. The tranng et contan 5 pont (, ), where are unorml randoml dtrbuted on the nterval (, ) and are dtorted b addng the unorm noe dtrbuted n [.,.]. The tetng et generated mlarl but wthout noe. Thu the tet pont repreent the. Accordng to [] the nput are normalzed nto the range [, ], whle the output are normalzed nto the range [, ]. The EM compoed o hdden neuron wth gmod AF. The n Fg. hown. Th curve unepectedl der rom that one hown n Fg. n []. In our cae the demontrate ubtantal devaton rom the, whle n [] t almot concde wth the. Addng neuron do not mprove the. Note that the devaton between and are not due to overttng to the tranng pont. In cae o uch a large number o tranng pont a n our eperment the overttng hould not have happened, becaue the tranng pont cover evenl ever part o the n ece. We oberve that the nput are not preproceed b normalzaton to the range [, ] the almot perectl appromate the. Thu the problem n the II,.e. the range o. The nput o EM are proceed b the AF o neuron. The output o our neuron or nput normalzed to [, ] and wthout normalzaton,.e. rom the nterval (, ), are hown n Fg.. It can be een rom th gure that EM operate n thee two cae on derent ragment o AF. Thee AF are unchanged n EM durng learnng. The AF ragment n II are the ba uncton ormng the ttng curve b lnear combnaton. The component o th combnaton,.e. AF multpled b the output weght β, are hown n Fg. 3. The um o thee curve gve. The lope o each ndvdual AF more or le the ame along the nterval [, ], wherea n the nterval (, ) the AF lope change gncantl. In the latter cae the larger varablt o the AF n the mddle o the II, wherea at the border o II lat part o man AF can be oberved. A meaure o the AF varablt n t lope n th pont,.e. t dervatve dh ()/d. et u dene the lope uncton o the et o AF a the average lope o ndvdual AF: dh h =, (6) d and epre t a a percentage o the mamum value o h n the II: h% h =. (7) ma h

3 The ame can be dened or the weghted AF: d dh = = β, (8) d d % =. (9) ma A we can ee rom (8) the lope o AF are moded n the output laer b the output weght β to get that ha the bet t to pont. The plot o % () how the varablt o the et o AF or weghted AF n II. We can oberve how the varablt change along II. The AF varablt or our eample o [, ] and (, ) nterval llutrated n Fg. 4. In th gure the varablt o the alo hown a the percentage lope uncton dened a ollow: dg( ) dg( ) g % = ma. () d In (, ) nterval the larget varablt o the AF n the mddle part o II, where there the larget varablt o. And the le varablt o AF at the border correpond to the le varablt o n thee regon. In the cae o [, ] nterval, ttng the curve to n the mddle part o II requre the adequate lope o AF, whch are adjuted b the output weght. But becaue ndvdual AF ha mlar lope n the whole II, ncreang the lope b output weght or a good ttng n the teep part o caue the ame ncreang n the lat regon o at the border. Th reult n too much varablt o EM at the border. Th problem would not are n the cae o o the ame varablt n II, e. g. the pure ne uncton. I the varablt o the change n II the lope o the eparate AF hould change a well. Moreover, the mot varable regon o AF hould correpond to the mot varable regon o. In the cae o mult-laer perceptron (MP) the varable regon o AF are matched to b adaptaton o bae b. But n EM bae are not adapted, o ther ntal value generated b random are ver mportant. The hould be matched to II. To llutrate th better let u conder appromaton o another o the orm: g d = n( e ), () or [, ]. The complet o th uncton ncreae along the nterval o [, ]. At the let border o th nterval lat, whle at the rght border t varablt the hghet. The tranng et nclude 5 pont (, ), where are unorml randoml dtrbuted on [, ] and are dtorted b addng the unorm noe dtrbuted n [.,.]. The tetng et created mlarl but wthout noe. The output are normalzed nto the range [, ]. The EM compoed o hdden neuron wth gmod AF. The nput weght and bae are generated randoml rom the deault nterval ([, ] or weght and [, ] or bae). In the rt eample the nput are normalzed to the range o [, ]. In th II the mot varablt o AF at the let border, wherea the mot varablt o at the rght border. Fg. 5 how, AF and % () n th cae. luctuate n the lat part o and undertted n the comple part. More neuron n the hdden laer (up to ) doe not mprove the reult. When the varablt regon o AF correpond to the varablt regon o TS the t much better. Th occur when II, to whch nput are normalzed, [, ]. Th cae hown n Fg. 6. Here lat part o AF correpond to lat part o, and comple part o modeled ung teeper part o AF. For comparon n Fg. 7 the reult or MP are hown. The number o hdden neuron and ther AF tpe were the Fg.. Reult o uncton (5) appromaton n II o [, ]. h () Fg.. The ragment o AF n the II: [,] and (, ). () h () () Fg. 3. The ragment o the weghted AF n the II: [,] and (, ). % () 5 g% h% % % () Fg. 4. The percentage lope uncton n II: [,] and (, ).

4 ame a or EM. The output neuron wa lnear. The MP wa learned ung the evenberg-marquardt algorthm (Matlab mplementaton wa ued). A can be een rom Fg. 7 the t better than or EM. The weghted AF varablt % (), whch adjuted b learnng nput and output weght and bae, correpond to the varablt. The reultng AF are evenl dtrbuted n the II. When we changed the II, th dtrbuton and nal were mlar. The bae o the hdden neuron adapt to the II. All AF preented n Fg. 7 have the common eature, the rapdl pa rom to. Th ndependent on II and even on the, the number o neuron hgh. The pattern o AF dtrbuton n II der rom thee one or EM (compare h () plot n Fg., 5, 6 and 7). et u cop th pattern nto EM. et u aume that: II contant: [, ], the nput weght are contant: even-numbered neuron have a = + and odd-numbered one have a =, AF (gmod) are evenl dtrbuted n II. To dtrbute AF n II we aume or a moment that all nput weght are + and that the value o the rt AF n = (let border) equal to q =.99: h () = =.99 + a + b )) q =. () From th equaton we get b 4.6. Then we aume that the value o the lat AF n = (rght border) equal to q =.: h ( ) = = =. + a + b )) q. (3) From (3) we get b 4.6 a = 4.6. The bae o the ucceve neuron are evenl dtrbuted n the nterval [b, b ]. Thu the -th neuron ba : b b b = b + ( ). (4) Now we put mnu beore bae o the odd-numbered neuron havng negatve nput weght. The reultng dtrbuton o AF n Fg. 8 hown. Th AF pattern ued n EM or uncton () appromaton. The varablt o AF hgh and unorm n the whole II ( h% () = cont = %). Th allow EM to appromate the comple uncton. But n the cae o our () havng changng complet, n the lat part the luctuaton o the are oberved. Th eceve varablt o EM at the let border wa not reduced b output weght a n the cae o MP (compare % () uncton n Fg. 8 and 7). But when the number o hdden neuron ncreae, the mprove. In Fg. 9 reult or neuron are preented. Baed on the eperence o the above decrbed mulaton, n the lat eperment we tr to elect the nterval or random nput weght and bae n EM. Frt let u aume that II contant: [, ]. So or each appromaton problem the nput are normalzed nto th range. To enure hgh varablt o AF n II let u aume that the nput weght a are generated randoml wthn the range [, ] rom the un- RMSE =.75 RMSE = h () 5 () h () - -5 () g% h% % () 5 g% h% % () 5 % % Fg. 5. Reult o uncton () appromaton n II o [,] Fg. 6. Reult o uncton () appromaton n II o [, ].

5 orm dtrbuton. (arger abolute value o weght mean greater AF lope; or a = we get the completel lat AF.) Bae are generated randoml rom unorm dtrbuton n uch a wa to enure varablt o each AF n II. Thereore, we aume that the mnmum value o the gmod AF hould be not greater than q [, ], and t mamum value hould be no le than q: mn( h ) q and ma( h ) q, =,,...,. (5) For gmod wth potve weght a, the mnmum n the let border o II, and mamum n the rght border, o: h ( ) = q, (6) + a + b )) h = ( ) q + a + b )). (7) From the rt nequalt we get the upper lmt o the range rom whch the bae are randoml generated. It the ame or all neuron. The lower lmt obtaned rom the econd nequalt dependent on the neuron nput weght. The range o the ba or the -th neuron havng the potve weght a : h () % () g% h% % RMSE =.436 () 4 - q ln, ln, q b a a. (8) q q For gmod wth negatve weght a the mnmum n the rght border, and mamum n the let border o II. Smlar conderaton a above lead to the range o the ba value or the -th neuron havng the negatve weght: q q b ln, ln, < a a. (9) q q The lower value o the threhold q reduce the ntroducton o the lat ragment o AF nto II. The cloer q-value to, the more lat part o AF ntroduced nto II. Th at the epene o maller number o the teep ragment. Th llutrated n Fg., where the lope uncton (6) or derent q hown. Note that or maller q the AF lope at the border o II are much lower than n t central part. The dtrbuton o twent AF generated randoml accordng to the above rule or q =.9 n Fg. hown. Note hgher varablt o AF n II than outde th nterval. Reult o uncton () appromaton or randoml generated nput weght rom the range [, ], and bae rom the range (8) and (9) n Fg. are hown. Thee reult, and RMSE, are ver mlar to the cae preented n Fg. 9, where nput weght and bae were ntated determntcall. The anwer to the queton whch method o the nput weght and bae better requre urther tud. h () % () - RMSE = () g% h% % Fg. 7. Reult o uncton () appromaton b MP Fg. 8. Reult o uncton () appromaton b EM wth evenl dtrbuted neuron.

6 h () % () - RMSE = () Fg. 9. Reult o uncton () appromaton b EM wth evenl dtrbuted neuron. h.8.6 q = IV. CONCUSIONS The perormance o EM entve to the localzaton o the AF varablt regon wth repect to the nput nterval. Th becaue the AF are ed and cannot be moded b adaptaton o nput weght and bae a n the cae o MP. In EM when the varablt o the AF doe not correpond to the complet o the target uncton advere eect can be een uch a luctuaton o the tted curve n the lat part o the target uncton or underttng n t comple part. When we have no normaton about the localzaton o the lat and comple part o the target uncton or th uncton o the ame complet n the whole range, we hould provde the AF whch varablt cover the nput nterval,.e. the mean varablt o AF mlar n the whole nterval. In uch a cae the AF (workng a the ba uncton) are able to contruct b lnear combnaton (th perormed b the output neuron) the output uncton whch appromate the target uncton wth acceptable error. In th work we recommend the range rom whch the nput weght and bae hould be randoml generated or one - g% h% % Fg.. The lop uncton (6) or derent value o q. q =,9 h () - q =, Input nterval - Fg.. The dtrbuton o AF or q =.9. h () % () - RMSE = () Fg.. Reult o uncton () appromaton b EM wth neuron randoml dtrbuted accordng to the propoed rule. varable uncton appromaton. The propoed range or nput weght enure derent lope o AF and the range or bae (determned ndvduall or each neuron) enure approprate ht o AF, o that the varablt o AF n the nput nterval wa the hghet. REFERENCES [] G.-B. Huang, Q.-Y. Zhu and C.-K. Sew, Etreme earnng Machne: Theor and Applcaton, Neurocomputng, vol. 7, pp , 6. [] G.-B. Huang,. Chen and C.-K. Sew, Unveral Appromaton Ung Incremental Contructve Feedorward Network wth Random Hdden Node, IEEE Tranacton on Neural Network, vol. 7, no. 4, pp , 6. [3] G.-B. Huang, H. Zhou, X. Dng, and R. Zhang, Etreme earnng Machne or Regreon and Multcla Clacaton, IEEE Tranacton on Stem, Man, and Cbernetc - Part B: Cbernetc, vol. 4, no., pp ,. [4] G. Huang, G.-B. Huang, S. Song, and K. You, Trend n Etreme earnng Machne: A Revew, Neural Network, vol. 6, no., pp. 3-48, g% h% %