Enhanced Artificial Neural Networks Using Complex Numbers

Transcription

1 Enhanced Artfcal Neural Networks Usng Complex Numers Howard E. Mchel and A. A. S. Awwal Computer Scence Department Unversty of Dayton Dayton, OH Computer Scence & Engneerng Wrght State Unversty Dayton, OH Astract The model of a smple perceptron usng phase-encoded nput and complex-valued weghts s proposed. The aggregaton functon, actvaton functon, and learnng rule for the proposed neuron are derved and appled to two and three nput Boolean logc functons. An mprovement of 35% over the theoretcal maxmum of 04 lnearly separale prolems (of three varales solvale y conventonal perceptrons s acheved wthout addtonal logc, neuron stages, or hgher order terms such as those requred n polynomal logc gates. Such a network s very attractve for optcal mplementaton snce optcal computatons are naturally complex. Introducton The processng power of an artfcal neuron s dependent on the nformaton representaton used n the neuron. Tradtonally, artfcal neural networks (ANNs used to process real-valued physcal data have reled on real-valued weghts. The nterconnecton weghts whch represent the learned ehavor of the ANN are derved from the recognton that at a smplfed level, a ologcal neuron s frng rate represents the nformaton n the network. However, some of the lmtatons of exstng ANNs may e traced to the lmtatons n the representaton of nformaton. The ojectve of ths work s to develop a new neuron model and a new learnng paradgm that can encode nformaton such that large-scale prolems can e more easly solved on dgtal computers. It s hypotheszed that representng real world dgtzed scalar data as phase and operatng on ths data n the complex-doman, mght mprove the performance of ANNs. The dea of usng complex numer n ANNs, however, s not new. Varous researchers have developed complexvalued ANNs and appled them to complex-valued data, such as complex sgnals and Fourer transform [-7]. Others have explored optcs n the course of fndng a sutale canddate for mplementaton of neural networks, whch naturally perform calculatons n the complex doman [8-0]. Complex numers have also een exploted for Hopfeld type assocatve memory for assocatve retreval wth partal nput [, ] and for rotaton nvarant retreval usng Fourer transform of edge data [3]. Stll others have developed complex-valued artfcal neural networks to solve Boolean logc functons of n varales y selectng an output state from a complex plane dvded nto m regons, wth m > n [4]. The work proposed here extends complex numers for general ANN archtectures and proposes a new learnng paradgm. The representaton of the new neuron s shown to e at least as computatonally powerful as, and n many cases, more powerful than exstng ANNs. Mathematcal representaton of the proposed neuron The proposed complex-valued artfcal neuron s smlar n composton to a tradtonal artfcal neuron except all weghts, w, wll e represented y complex numers. Externally, nput data and output data wll e real. Therefore, nput mappng and output mappngs are requred, along wth complex-valued nternal neuron functons. These nternal functons wll e called aggregaton (f and actvaton (f. Each of these operatons s developed n the followng sectons. The nput mappng defnes how the real-world data wll e represented n the ANN calculatons. In the complex-

2 valued artfcal neuron, ths mappng wll e from a realworld value typcally defned as a real numer, logc value, or real-world data nto a complex numer. It may e noted that even n the tradtonal ANNs, real-valued data from the real world must e mapped nto a specfed range. Therefore, the nput-mappng s not an addtonal stage only requred n the complex-valued artfcal neuron. m m β p e ψ = λ p ( 0 f data = FALSE : ψ = π ( f data = TRUE cv_percept ron_nput 0 e f data = FALSE : p = π (3 e f data= TRUE To express the nput mappng for the complex valued artfcal neuron, assume that the set of nput varales P s composed of n-tuples p, where each of the components p s expressed as equaton. One possle nput mappng for Boolean data s shown n equaton. Equaton 3 s the fully developed verson of equaton for λ p =. Dscrete logc levels are thus coded as perodc pulse trans wth unty magntude and dfferent phases. Ths s a mappng from R n C n. The complex-valued aggregaton functon s desgned after the form of a tradtonal neuron s aggregaton functon as shown n equaton 4. Here, p C n s column vector of the nput components p, and w C n s a row vector of weghts terms w. The aggregaton functon s thus a mappng C n C. Unlke tradtonal neurons, ths aggregaton functon s not lnear, and the resultant output s dependent on the relatonshps among the varous weghts and nputs, as well as ther ndvdual values. These relatonshps wll e descred n detal elow. q = wp (4 The aggregaton functon feeds drectly nto the actvaton functon; therefore, the range of the aggregaton functon s the doman of the actvaton functon, whch we wll call the ntermedate space. The range of the actvaton functon s the output space. Note that the output space s not the real-world value, ut the representaton of the soluton wthn the artfcal neuron; however, the neuron must eventually respond wth a real valued answer. The complex-valued neuron wll use a perceptron-lke actvaton functon, that s, a hard lmtng functon. Because the magntude of complex numer s easy to compute, and easy to measure optcally and electroncally, and ecause t captures the effects of angle dfferences and ndvdual component magntudes, t was chosen as the doman varale for the actvaton functon. That s, the range of the complex valued actvaton functon wll e the magntude of the values n the ntermedate space, q. The actvaton functon s shown n equaton 5,where a and T are real numers, and q s complex. 0 f q a = f q < T T As opposed to the lnear threshold used n conventonal neuron, ths s equvalent to a crcular threshold. Thus the value les ether nsde or outsde of the decson crcle. The actvaton functon mappng s thus of the form C R. In a tradtonal neuron, the output mappng from an nternal representaton to the physcal representaton s requred. Ths s a mappng of the form R R that typcally s concerned wth scalng and/or numercal accuracy. Because the complex-valued artfcal neuron s actvaton functon s of the form C R, the output mappng n the complex-valued neuron s of the form R R and s dentcal to the tradtonal neuron s output mappng. The aggregaton functon of a two-nput complex-valued neuron s shown n equaton 6. The varales ψ, and λ p were defned earler y equaton. The varales λ w and θ correspond to the magntude and angle of the weght term respectvely. For smplfcaton purposes n the present dscusson, we wll assume λ w s, and the learned weghts are represented n θ. q = λ + λ w w λ λ p p[ cos( θ + sn( θ ] [ cos( θ + sn( θ ] ( θ θ Snce only the magntude of the resultant vector (and not the phase wll effect the outcome of the actvaton functon, and the fact that all λ s =, equaton 6 can e replaced y a smplfed formula. Equaton 7 expresses the magntude squared of the ntermedate result n terms of the magntudes and phases of the nputs and weghts for the smple -tuple neuron. (5 (6 r = q = + cos ψ (7

3 The effectve change n output n response to a weght change depends on the relatonshp of that weght to the other weghts and all nputs. A weght term s not smply assocated wth only ts correspondng nput. Ths ssue wll e consdered further as a new learnng rule s developed for complex-valued artfcal neurons. w = θ = r δθ Assume that, the weght change s w = w new - w old. As dscussed aove, the relevant part of the weght term s ts angle, θ, therefore, w = θ = θ new - θ old. The requred change n the resultant s expressed as r = r new - r old. Equaton 8 relates the change n the weght, w, to the change n the resultant vector, r. Equaton 9 s thus selected as the tranng rule for the complex-valued neuron. However, equaton 9 assumes that the desred change n the resultant r, that s, r, s known. In actualty, the desred change n the output, the error d a, (desred actual s known. To arrve at the resultant r, ths error must e rought ack across the actvaton functon, defned y equaton 5. If the actvaton functon was contnuous, the partal dervatves n equatons 8 and 9 could e extended ack to the output. However, t s not; t s dscontnuous at the threshold pont. Therefore, t s not mathematcally correct to take ths dervatve. As an approxmaton, t wll e assumed that a correcton of r n the drecton toward the threshold of the actvaton functon, on ether sde of the threshold, wll satsfy the tranng goal n a local manner. Specfcally, f d a s postve, r should e postve, and vce versa f r s negatve. Therefore, y replacng r n equaton 9 wth a proporton of the output error d a, a fnal tranng rule for the -nput complex-valued artfcal neuron s shown n equaton 0. The proportonalty constant, η, s also known as the learnng rate. [ θ θ ] = [ θ θ ] new new old old [ θ θ ] = [ θ θ ] + η( a new new old old The complex-valued -nput-plus-as neuron Bas n a tradtonal neuron can e vewed n two ways. The frst s that the as shfts the threshold pont for the actvaton functon. In ths context, an equvalent as term n the complex-valued neuron s a shft n the decson threshold of the actvaton functon. Ths s equvalent to a (8 + r (9 δθ δθ d (0 δθ δθ shrnkng or expandng of the decson crcle. The second vew of the as term n a tradtonal neuron s that t adds an nput-ndependent value to the summaton performed y the aggregaton functon. Ths vew of as can e accommodated n the new complex valued neuron y addng an nput-ndependent complex numer to the complex summaton performed y the aggregaton functon. The ncorporaton of complex numer n the complexvalued neuron s equvalent to a vector-lke shft of the resultant vector n the ntermedate space efore thresholdng. Note that n a tradtonal neuron, the threshold s the addtve nverse of the as, thus provdng one addtonal degree of freedom. In the complexvalued neuron, the as and threshold provde three addtonal degrees of freedom one for the threshold, and one for each of the magntude and angle of the as. The addton of a as term to an artfcal neuron can e expressed y ncorporatng a as element nto the nput and weght vectors to create extended vectors resultng n an ncrease of ther dmensonalty y one. Therefore, the -nput-plus-as complex-valued neuron uses a 3-tuple nput set nstead of a -tuple nput set. Ths added term s a constant, ndependent of the nput. Ths addtonal term should not e confused wth the transformaton from a -nput threshold logc gate (TLG to a 3-nput polynomal logc gate (PLG, n whch the addtonal term s a functon of the other two nputs. The -nput-plus-as artfcal neuron s stll a sngle level operaton. Changng the weght vector to an extended weght vector nvolves addng an addtonal weght term to e appled to the as term. The extended p s defned y equaton, wth ts components, p, defned y equaton. The component can e ether real or complex. For smplcty, and wthout loss of generalty, t wll e assumed that =. The extended w s defned y equaton. ( p p T p = ( θ θ θ ( λ e λ e λ e w w By applyng equatons and to the aggregaton functon defned y equaton 4, an expresson for the resultant q s otaned smlar to equaton 6. Ths s shown n equaton 3. By makng smlar smplfyng assumptons aout the magntudes of the nput and weght terms, λ p and λ w respectvely, an expresson for the magntude squared of the resultant, r, smlar to equaton 7 s otaned. It s shown n equaton 4. Note that no assumptons were made aout the magntude of the as w = (

4 term, λ, and t s ncluded n expresson 4. Ths ssue wll e covered further elow. q = λ r = w + λ + λ q w p[ cos( θ + sn( θ ] λ p[ cos( θ + sn( θ ] [ cos( θ + sn( θ ] λ = + λ + + λ cos + λ cos Followng the development, a learnng rule smlar to equaton 0 s developed. Those equatons are shown formally as equatons 5 through 8. δθ δθ δθ Up to ths pont, the development of the -nput-plus-as complex-valued artfcal neuron has followed the development of the -nput verson. All nput and aggregaton equatons developed for the more complex neuron have smlar counterparts n the smpler neuron. The parameters λ and T not addressed wll now e dscussed The actual output a s related to the threshold T through the hard-lmtng functon depcted n equaton 5. That s, f the magntude of the ntermedate-space resultant q s less than the threshold T, the actual output wll e set to 0; otherwse, the actual output wll e set to. Two error condtons can exst. Frst, the desred output d s and the actual output s 0, and second, the desred output s 0 and the actual output s. These errors can e corrected as follows. If the desred output s, and the actual output s zero, the threshold should e reduced. Conversely, f the cos( θ θ ψ ( θ θ ( θ θ ( θ θ + λ ( θ θ = sn sn (3 (4 λ (5 ( θ θ ψ λ sn( θ θ = sn (6 ( θ θ ψ λ sn( θ θ = sn (7 [ θ θ θ ] = [ θ θ θ ] + η( d a new new new old old old δ r δθ δθ δr δθ (8 desred output s 0 and the actual output s, the threshold should e ncreased. If the error s defned as the dfference etween the magntude of the desred output and the actual output, t can e seen that y sutractng the error (whch can e, 0 or from the threshold, the threshold moves n the correct drecton. By parallelng the perceptron learnng rule, a new learnng rule for the complex valued neuron was developed. The learnng rule for the threshold T s shown n equaton 9. In equaton 9, η s a learnng constant smlar to equaton 8 aove, ut the values need not e equvalent. T new old ( d a T old = T η (9 Changng λ, the as term magntude, changes whch ntermedate-value terms q wll exceed the threshold magntude T, and therefore, ther correspondng output values a. There s a very complex relatonshp etween these terms however. It can e seen that the effect of changes n the magntude of the as term on the output s related to amount of change, and the angle of the as term as a component of the total angle of the ntermedate term, q. Equaton 4 expresses the relatonshp etween the ntermedate resultant squared, r, and the magntude of the as term λ. The partal dervatve of r, wth respect to λ mathematcally captures the effect of changng λ has on r. The goal of ths learnng rule s to change λ n such a manner as to effect a desred change n r. That s, gven a desred change n r, what should e the change n λ? Equaton 0 expresses the relatonshp. Agan, η s a learnng constant, not necessarly equvalent to the other learnng constants used n equatons 8 and 9. ( d a η λ = λ + (0 new old δr δλ Computer Smulaton result Applyng the learnng rules developed aove, the -nputplus-as complex-valued artfcal neuron s capale of learnng all 6 possle functons of two Boolean varales, x and x. Tradtonal perceptrons are capale of learnng only 4 of those functons. The learned weght-values for all 6 functons are shown n tale. In tale, the Y column represents all 6 possle functons of Boolean varales. Each Y entry represents a functon y specfyng whch of the four mnterms are ncluded n the output. That s, Y = y y y 3 y 4, wth y = x x, y = x x, y 3 = x x and y 4 = x x. A assocated wth a partcular mnterm s nterpreted to mean that mnterm s ncluded n the output functon, a 0 means the mnterm s not ncluded. For example, Y = 000 ncludes only mnterm y 4 and s thus

5 the AND functon, whle Y = 00 ncludes mnterms y and y 3, and s thus the XOR functon. The learned weghts n tale represent solutons when all λ = and T =.. The nput values are encoded as specfed n equaton 3. The correctness of the learned values can e verfed y applyng equaton 3 to the vectors. For example, mnterm y of the AND functon s calculated q = e e + e e + e = and mnterm y 4 s calculated q = e π π e + e e + e e =.443 Applyng equaton 5, t can e seen that q <. and q 4 >., therefore, the actual output s 0 for mnterm y (as t would e for mnterms y and y 3 f they were shown and s for mnterm y 4. The -nput-plus-as complex-valued artfcal neuron has een extended to 3-nput plus as neuron. The 3-nputplus-as complex-valued artfcal neuron was traned to learn all three-varale Boolean functons. Based on these smulatons, the neuron was ale to effectvely compute solutons to 45 of the 56 possle functons. Snce only 04 of these are lnearly separale, the complex-valued neuron s at least 35% more powerful that a conventonal perceptron. Y = q q q y y y 3 y Tale. Learned weghts, n radans, for -nput-plusas complex-valued perceptron Cost Issues Those modes of mplementaton that are nherently more powerful.e. optcal computng, software mplemented on parallel computers, or software mplemented on computers wth co-processors wll eneft more than mplementatons on standard seral computers. Expected enefts nclude reduced network sze, reduced delay when operatng n the recall phase, and qucker learnng. These enefts wll arse ecause the complex-valued representaton wll e computatonally more powerful than the exstng representatons. For example, a sngle complex-valued neuron constructed usng the new representaton can solve prolems that are not lnearly separale. Conventonal neurons requre at least two layers to solve ths prolem; therefore, ANNs can e constructed wth fewer artfcal neurons. Although each ndvdual neuron wll e more complex, the overall ANN wll requre less hardware or use fewer mathematcal operatons to solve exstng prolems, therefore, speed of operaton wll e ncreased and cost wll e lowered. These expected enefts are mplementaton dependent. The cost of complex-valued neuron s less n all cases than the tradtonal neuron when mplemented optcally. Therefore, all the enefts the complex-valued artfcal neuron can e otaned wthout addtonal cost. Addtonally, the complex-valued neuron should e equally superor n those mplementatons that provde hardware support for complex arthmetc, for example computers wth neural-network co-processors ased on dgtal sgnal processng chps. On those mplementatons dependent on standard seral computers, the complexvalued neuron wll e more cost effectve only n those applcatons where ts ncreased power can offset the requrement for addtonal neurons. Concluson The complex-valued neuron was shown to demonstrate hgher computatonal capalty for a large class of prolems nvolvng Boolean functons. The complexvalued neuron s ale to solve all 6 functons of -nput Boolean logc, and 45 of the 56 functons of the 3-nput Boolean logc. References:. Ntta, T., "An extenson of the ack-propagaton algorthm to complex numers," Neural Networks, 0 (8, 39-45, Benvenuto, N., and Pazza, F., "On the complex ackpropagaton algorthm," IEEE Transactons on Sgnal Processng, 40 (4, , Leung, H., and Haykn, S., "The complex ackpropagaton algorthm," IEEE Transactons on Sgnal Processng, 39 (9, 0-04, Georgou, G. M., and Koutsougeras, C., "Complex doman ackpropagaton," IEEE Transactons on Crcuts and Systems

6 II: Analog and Dgtal Sgnal Processng, 39 (5, , Smth, M. R., and Hu, Y., "A data extrapolaton algorthm usng a complex doman neural network," IEEE Transactons on Crcuts and Systems II: Analog and Dgtal Sgnal Processng, 44 (, 43-47, Arena, P, Fortuna, G., Muscato, G., and Xla, M. G., "Multlayer Perceptrons to approxmate quaternon valued functons," Neural Networks, 0 (, , Hrose, A., "Dynamcs of fully complex-valued neural networks," Electroncs Letters, 8 (6, , Casasent, D., and Natarajan, S., "A classfer neural network wth complex-valued weghts and square-law nonlneartes," Neural Networks, 8 (6, , Weer, D. M. and Casasent, D. P., "The extended pecewse quadratc neural network," Neural Networks,, , Hrose, A., "Applcatons of complex-valued neural networks to coherent optcal computng usng phase-senstve detecton scheme," Informaton Scences,, 03-7, Khan, J. I., "Characterstcs of multdmensonal holographc assocatve memory n retreval wth dynamc localzale attenton," IEEE Transactons on Neural Networks, 9 (3, , Khan, J. I, and Yun, D. Y., "A parallel, dstruted and assocatve approach for pattern matchng wth holographc dynamcs," Journal of Vsual Languages and Computng, 8 (, Awwal, A. A. S. and Power G., "Oject Trackng y an Opto-electronc Inner Product Complex Neural Network," Optcal Engneerng, 3, , Azenerg, N. N., and Azenerg, I. N., "Unversal nary and mult-valued paradgm: Concepton, learnng, applcatons," Lecture Notes n Computer Scence, 40, , 997.