BACKPROPAGATION GENERALIZED DELTA RULE FOR THE SELECTIVE ATTENTION SIGMA IF ARTIFICIAL NEURAL NETWORK

Transcription

1 Int. J. App. Math. Comput. Sci., 12, Vo. 22, No. 2, DOI: /v BACPROPAGATION GENERALIZED DELTA RULE FOR THE SELECTIVE ATTENTION SIGMA IF ARTIFICIAL NEURAL NETWOR MACIEJ HU Institute of Informatics Wrocław University of Technoogy, Wyb. Wyspiańskiego 27, Wrocław, Poand e-mai: In this paper the Sigma-if artificia neura network mode is considered, which is a generaization of an MLP network with sigmoida neurons. It was found to be a potentiay universa too for automatic creation of distributed cassification and seective attention systems. To overcome the high noninearity of the aggregation function of Sigma-if neurons, the training process of the Sigma-if network combines an error backpropagation agorithm with the sef-consistency paradigm widey used in physics. But for the same reason, the cassica backpropagation deta rue for the MLP network cannot be used. The genera equation for the backpropagation generaized deta rue for the Sigma-if neura network is derived and a seection of experimenta resuts that confirm its usefuness are presented. eywords: artificia neura networks, seective attention, sef consistency, error backpropagation, deta rue. 1. Introduction In nature, seective attention is a mechanism which provides iving organisms with the possibiity to sift incoming data to extract information which is most important at a given moment and which shoud be processed in detai (Broadbent, 1982; Treisman, 19. When imited processing capabiities do not aow rapid anaysis of the whoe scene of visua and other senses, seective attention can be viewed as a strategy of dynamica input space seection for gaining predefined goas by the system (e.g., an organism interacting with a very compicated environment (Noton and Stark, 1971; Tsotsos et a., 01; Vanruen and och, 03. Accordingy, seective attention systems are found to be very interesting from a theoretica point of view, and aso as toos for many practica appications, such as anaysis of arge data sets, rea time route panning for autonomic robots in dynamica environments, and dispersed sensor networks contro (Desimone and Duncan, 1995; Oshausen et a., 1993; Houghton and Tipper, 1996; Hager and Toyama, 1999; Stark et a., 00; örding and önig, 01; Gupta, 08; Indiveri, 08; Ferguene and Toumi, 09; Pedro and Dahunsi, 11. As most of the seective attention systems observed in nature use neurona contro mechanisms, many researchers try to reaize seective attention soutions by using artificia neura networks. Unfortunatey, networks that use higher-order neuron modes, such as Sigma-Pi (Fedman and Baard, 1982; Rumehart et a., 1986; Me, 19; Oshausen et a., 1993, Power Unit (Durbin and Rumehart, 19 or Custeron (Me, 1992, reaize ony a very imited set of attentiona mechanisms (Nevie and Edridge, 02; Weber and Wermter, 07. Thus it can be very interesting that seective attention functionaity, which seems to effectivey mimic ow-eve attentiona processes observed in humans, was found in a recenty deveoped simpe generaization of the we-known MLP network caed Sigma-if (Huk, 04; 06; 09. However, the Sigma-if neura network mode to be trainabe with use of the backpropagation agorithm (typica for MLP needs a new, generaized form of the deta rue that wi take care of the noncontinuous character of the aggregation functions of the Sigma-if neurons. 2. Preiminaries The Sigma-if neura network is a type of synchronous, feedforward mutiayer Artificia Neura Network (ANN and possesses seective attention abiities (Niebur et a., 02; Huk, 04; 06. Such a neura network does not use separate centraized attention guidance modues. Its abiity to reaize ow-eve seective attention functiona-

2 450 M. Huk ity emerges as an effect of synergy between its hidden, Sigma-if neurons. Each Sigma-if neuron is a specia direct generaization of a sigmoida neuron which impements basic seective attention functionaity via input connections grouping and stepwise conditiona input signa accumuation. This is due to the new neuron s aggregation function (Duch and Jankowski, 1999; Huk, 04. Formay speaking, N dendrites of the Sigma-if neuron are divided into distinct groups, by compementing each i-th input connection with an additiona integer parameter θ i {0, 1,..., 1}, determining membership in one of the groups. This aows us to divide the process of signa accumuation into steps, where is a function of the neuron s grouping vector θ T =[θ 1,θ 2,...,θ N ]: (θ = max N (θ i. (1 During each step k (from 0 to 1, the neuron accumuates data beonging to one seected group, such that θ i = k. (2 Within each k-th group, partia activation Δϕ(k is determined as a weighted sum of input signas and the appropriate ronecker deta: Δϕ k (w, x, θ = N w i x i δ(k, θ i, (3 where w i and x i are coefficients of the neuron s weight vector w and an input vector x. This process is repeated unti the activation derived from respective groups exceeds a preseected aggregation threshod ϕ. It can be described by the foowing recursive formua (vectors w, x and θ are omitted for carity: ϕ k = { Δϕk H(ϕ ϕ k 1 +ϕ k 1 if k 0, 0 if k<0, (4 whereh is Heaviside s function. This sum is then treated as the neurona activation vaue. The input from remaining (heretofore unconsidered groups is negected. Thus, the form of the aggregation function ϕ Sigma-if is ϕ Sigma-if (w, x, θ =ϕ (w, x, θ. (5 In the fina stages of determining the output vaue Y of the neuron, the function (5 serves as a parameter of the noninear threshod (e.g., sigmoida function F : Y (w, x, θ =F (ϕ Sigma-if (w, x, θ. (6 It is worth noting that the described mode assumes that the state graph used during signa aggregation is aways a simpe directed path of nontermina nodes corresponding to the accumuation procedure of neura activation. In a genera case, the Sigma-if neuron, besides the vector of weights w, incudes one rea parameter for the aggregation threshod ϕ, and an additiona vector θ coecting connections with ony one nomina coefficient for each neurona input connection. In comparison with MLP neura network training, searching for a gobay optima set of the Sigma-if network parameters woud be very computationay chaenging. This is due to the noncontinuous character of the Sigma-if neuron grouping vector. Whie there is no quick and effective method of goba searching for network weights and grouping vectors, one can assume that, at each Sigma-if neuron, coefficients of the grouping vector θ are in fact direct functions of the weight vector. In the work of Huk (04 the proposed soution is to sort inputs of a given Sigma-if neuron by their weights, and assign N/ connections with the highest weights to the most significant group θ 1,next N/ connections to group θ 2, and so on. In the above soution, the mutua reationship between connection weights and grouping vectors aows an improvement of the backpropagation agorithm by the appication of the sef-consistency idea widey used in physics (Noh et a., 1991; Fonseca et a., 1998; Raczkowski et a., 01. To reaize that, Sigma-if network training begins with random vaues of connection weights and with a connections assigned to the singe group. This assures that at the beginnig of the training process the network behaves as a mutiayer ANN with sigmoida neurons, and a of the connections between neurons are treated as equay important. Then Sigma-if network connection weights are changed by an error backpropagation agorithm for ω training epochs without changes in grouping vectors. After ω training epochs, actua grouping vectors are computed for a Sigma-if neurons and then connection weights are changed again by the error backpropagation agorithm for the next ω training epochs. This process is repeated unti the resuting network meets the stop condition of the backpropagation method. Such aternate changes of two mutuay dependant sets of parameters of the Sigma-if mode can ead to optimization of both weights and grouping vectors even if changes of ony one of these sets (e.g., weights are directy guided by a known optimization agorithm. Thus the ony eement needed to impement such a process for a Sigma-if neura network is to know the generaized deta rue for this mode. 3. Backpropagation deta rue for the mutiayer feedforward neura network It is convenient to show the derivation of a generaized deta rue for Sigma-if neura network in comparison with a backpropagation generaized deta rue for the MLP network. Thus, regardess of common knowedge about the

3 Backpropagation generaized deta rue for the seective attention Sigma-if artificia neura network 451 backpropagation agorithm, first we need to reca eements of this method (Rumehart et a., 1986; orbicz et a., This wi simpify further parts of the derivation and wi serve as a definition of a common set of symbos. Let us consider the genera case of a mutiayer feedforward neura network with a fu network of connections between neurons in adacent ayers, and a nondecreasing and differentiabe activation function in individua neurons. To estabish the symbos, we assume that every μ-th earning pattern is a pair containing the input vector x zµ and the corresponding output vector y zµ. Simutaneousy, consecutive ayers of the network are numbered with index m and vaues from 1 to M, where ayer m consist of n m neurons. Consequenty, the weight of the connection between the -th neuron in the m-th ayer and the i-th neuron of the previous ayer is written as wi m (in the case of doube ower indices, the eft subscript is the number of the neuron in the ayer of the number indicated in the superscript, whie the right subscript is the index of the neuron input. Simiary, the vaues of the aggregation function ϕ and the activation function F (ϕ for the -th neuron of the m-th ayer are denoted, respectivey, by ϕ mµ and u mµ, whie for the i-th neuron of the input ayer, which by definition reaizes an identity transfer function, u 1µ i is equa to x zµ i. Using the above notation and assuming that a neurons of network hidden ayers are sigmoida (with the aggregation function being a inear combination of input vaues and weights connections, we get the output vaues of neurons in the form ( nm = F (ϕ mµ =F u mµ w m,iu (m 1µ i. (7 Operation of the backpropagation agorithm comes down to a cycic repetition of four main phases. Using the designations made above, we can write that in each t-th cyce of the training process the phases for each μ-th training vector are as foows: 1. Provide the μ-th training vector for the network inputs and determine the vaue u mµ of the output of each -th neuron, in a ayers of the network from inputs to outputs (for m =1,2,...,M. 2. Cacuate the vaue of the error δ Mµ for each of n M output neurons and the sum ξ µ of their squares. 3. Propagate the output error backward from outputs to inputs with cacuation of errors δ mµ for a neurons in hidden ayers (for m = M,M 1,...,2. 4. Modify connection weights, starting from the output ayer and ending in the input ayer, according to the generaized deta rue for sigmoida neurons and with the formua w m(t+1 i = w m(t i +Δw m i. (8 After presenting a the training vectors the stopping condition of the agorithm is checked and, if it is not met, a the above steps are repeated in the next training cyce. Leaving aside the question of the maximum aowabe number of agorithm cyces, a typica backpropagation stopping condition is to determine whether the neura network output error for a vectors is ower than a given threshod. The network output error for a given μ-th training vector is a sum of squares of output neuron error vaues, given by the formua ξ µ = 1 2 n M (y zµ =1 u Mµ 2. (9 We can thus define an error created in the -th neuron of the m-th ayer as δ mµ = ξ µ ϕ mµ which can be converted to the form δ mµ = ξ µ ϕ mµ = ξ µ, (10 F (ϕ mµ. (11 For the output ayer we can directy write ξ µ u Mµ = (y zµ u Mµ. (12 In the case of hidden ayers, an anaogous partia derivative is, however, a bit more troubesome to cacuate, due to the compexity of the dependence of ξ µ on u mµ. To perform necessary transformations, one shoud use the dependence of the neuron aggregation function ϕ (m+1µ in ayer m +1on the vaue of u mµ. But by taking into account a contributions of the corresponding changes in aggregation functions to the change in the network error, and by using the chain rue of differentiation of composite functions, we obtain n ξ µ m+1 = =1 ξ µ ϕ (m+1µ ϕ (m+1µ. (13 Recaing now (10 and performing the differentiation of (7 with respect to u mµ, we can finay write n ξ µ m+1 = δ (m+1µ w (m+1µ. (14 =1 On the basis of Eqns. (11, (12 and (14, we can assign each neuron of a mutiayer network a suitabe output

4 452 M. Huk error vaue. For the output ayer, the error of the neuron output is given by δ Mµ = F (ϕ Mµ (y zµ u Mµ, (15 and in the case of hidden neurons their output error has the form n m+1 δ mµ = F (ϕ mµ δ (m+1µ w (m+1µ. (16 =1 However, to specify the reevant rue of changing connection weight wi m in the direction of the error gradient in the space of weights, which woud provide improved network operation in the next step of the training agorithm, we have to determine the vaue of the expression Δwi m = η ξ µ wi m = η ξ µ ϕ mµ ϕ mµ wi m. (17 Equation (7 shows that the second partia derivative occurring on the right-hand side of (17 is equa to u (m 1µ i. Moreover, its first partia derivative on the basis of (11 can be written as ξ µ ϕ mµ = ξ µ ϕ mµ = δ mµ. (18 Thus, we finay get a generaized form of the deta rue: Δwi m = ηδmµ u (m 1µ i, (19 whie for the output neurons it is expressed as Δw M i = ηu(m 1µ i F (ϕ Mµ (y zµ u Mµ ( and for hidden neurons as n m+1 Δwi m = ηu (m 1µ i F (ϕ mµ δ (m+1µ w (m+1µ. =1 (21 As the effect of the use of the above set of expressions, in each cyce of the backpropagation agorithm the neura network parameters are changed in the direction of the argest possibe decrease in the error function. As a resut, repeated presentation of a training vectors (at each cyce, if possibe, in different order eads to oca minimization of the error function, whie the size of the optimization steps is steered by the parameter η, often caed the earning factor. 4. Generaized deta rue for the Sigma-if neura network For a mutiayer Sigma-if neura network, the first two phases of the backpropagation agorithm computation of the network output vaues and determination of neurons output errors amost do not change in comparison with a mutiayer ANN with sigmoida neurons. The method of cacuating the error components δ Mµ for the output ayer and the μ-th training vector remains unchanged as a resut of the independence of the derivative of (10 of the form of aggregation functions of network output neurons. As a resut, the function (9, determining the mean square error over a outputs of the neura network, remains unmodified. In turn, the main difference in Sigma-if network training is the need to memorize for each -th Sigma-if neuron in the m-th ayer the number k mµ of groups of input connections activated during its output computation for the μ-th training vector. Thus, ooking one more time at the definition (5 we can formay write that in an interesting case of ow-eve seective attention, when not a input connections are used to compute neuron output vaue, k mµ <: ϕ k mµ (w, x, θ ϕ. (22 The vaues k are aso essentia for proper execution of the error backpropagation procedure, as they keep information about which input connections of the given neuron infuenced its output for the given training vector. These vaues aso aow rewriting the definition of the Sigma-if aggregation function (5 in the non-recursive form: ϕ Sigma-if (w, x, θ = Δϕ k (w, x, θ = k k=1 k k=1 N w i x i δ(k, θ i, (23 which is usefu in practica impementations and, which is more important, it wi be needed during further forma transformations. Due to the use of the aggregation function ϕ Sigma-if during the error backpropagation phase, the method of determining the errors in the output neurons undergoes a forma change. It can be shown for the aggregation function given by the expression (23, by repacing the number of neuron inputs N and neuron input vaues x i with the number n m of neurons in the previous ayer m and their output vaues u mµ, respectivey, and by cacuating again the derivative (13 (in the case of doube ower indices, the eft subscript is the number of a neuron in the ayer of the number indicated in the superscript, whie the right subscript is the number of the neuron input; for simpicity, the Sigma-if subscript of the aggregation function is further

5 Backpropagation generaized deta rue for the seective attention Sigma-if artificia neura network 453 omitted: ϕ (m+1µ = k (m+1µ k=1 n m ( w m+1,i u mµ i δ(k, θ m+1,i. (24 Hence, after expanding the sum over k and performing the differentiation of the right-hand side, the above equation takes the form ( nm + + w m+1,i u mµ i δ(1,θ m+1,i +... n m w m+1,i u mµ i δ(k (m+1µ,θ m+1,i = w m+1, δ(1,θ m+1, +w m+1, δ(2,θ m+1, w m+1, δ(k (m+1µ,θ m+1,. (25 Then, by factoring out the common weight terms, we can write ϕ (m+1µ k (m+1µ =w m+1, k=1 δ(k, θ m+1,. (26 However, the sum of ronecker detas appearing on the right-hand side of (26 may take ony two vaues: one when the -th input of the -th neuron beongs to one of the groups active during signa aggregation for the vector μ, and zero otherwise. In the first case, the component of the θ m+1, grouping vector assigned to the -th input connection is ess than or equa to the number of active groups k (m+1µ, and in the second one it is greater than this vaue. This aows us to concude that ϕ (m+1µ = w m+1, H(k (m+1µ θ m+1,. (27 Finay, by appying the derivative cacuated in this way to (13, one can determine the formua for the output error of -th neuron in the m-th hidden ayer of the Sigma-if neura network (based on (11: δ mµ = F (ϕ mµ n m+1 ( =1 δ (m+1µ w m+1, H(k (m+1µ θ m+1,, (28 where the parameter enumerates consecutive neurons in ayer m +1. The above expression differs from the corresponding formua (16 for the mutiayer feedforward network with sigmoida neurons ony by the appearance of the Heaviside function. Due to this change, when not a inputs of the Sigma-if neuron are invoved in determining its output vaue, during the backpropagation phase the error is propagated ony by the connections that were used. However, this is fuy consistent with the idea of the backpropagation agorithm. Neuron connections inactive during the aggregation of the input signas, despite non-zero weights and avaiabiity of signas, do not make any contribution to the activation of a neuron, and consequenty, they do not infuence the Sigma-if neurons output error vaues. Thus the weights of inactive connections shoud not be changed. To determine the genera rue of weight modification in the network of Sigma-if neurons, one shoud cacuate the expression (17 with the use of Eqn. (28. Therefore, the foowing derivative requires consideration: ϕ mµ w,i m = k mµ w,i m n m k=1 w m,iu (m 1µ i δ(k, θ m,i. (29 However, it is easy to note the simiarity between the above expression and the formua (24. By anaogy, without unnecessary transformations, we get ϕ mµ w m,i =u (m 1µ i H(k mµ θ,i m. (30 As a resut, the generaized deta rue specifying the change in the weight vaue of the i-th input of the -th neuron in the m-th Sigma-if network ayer takes the form Δw m,i =ηδmµ u (m 1µ i H(k mµ θ,i m, (31 where u (m 1µ i is the output vaue of the i-th neuron in m-1 ayer for training vector μ, andη is a earning factor. Finay, after taking into account the reevant formuas for errors of different eements of the Sigma-if network, the generaized deta rue for the output ayer of its neurons is given by Δw M,i = ηu (M 1µ i H(k Mµ θ,if M (ϕ Mµ (y zµ y µ, (32 whie its counterpart for the hidden ayers of Sigma-if neurons is Δw m,i = ηu (m 1µ i n m+1 ( =1 H(k mµ δ (m+1µ w m+1, θ m,if (ϕ mµ H(k (m+1µ θ m+1, (33. The Heaviside function appearing in the expression (31 can be viewed as a mechanism that counteracts unnecessary modifications of the network structure in those

6 454 M. Huk parts which are not used for determining the output vaues of individua neurons for a given training vector. Thus, both in the hidden and the output ayer, the weights of connections that were inactive during the process of input signas accumuation are not modified. 5. Resuts of experiments The generaized deta rue for the Sigma-if neuron and its conditiona aggregation function presented above was additionay examined by verification of the whoe Sigma-if network properties using exampe cassification tasks of seected benchmark probems of the UCI Machine Learning Repository. During tests, simuated Sigma-if neura networks were compared with MLP networks with the same architectures (one hidden ayer, the number of neurons in ayers dependent on the soved probem see figures beow. Their generaization abiities were additionay anaysed against the best resuts of other machine earning cassification methods (see, e.g., Huk, 06. As the sigmoida neuron is a specia case of a Sigma-if neuron, mutiayer networks with sigmoida neurons were simuated by Sigma-if networks with the number of inputs groups of a Sigma-if neurons set to one. In a cases, standard input signa coding was used, output coding was bipoar and answers of the neura network were computed in the winner-takes-a manner. Aong with cassification accuracies u for training and γ for test data, properties such as the neura network data processing time τ as we as hidden connections and network input activity (designated by hca and nia, respectivey were considered. Hidden connections activity hca and network inputs activity nia were used to represent the percentage ratio of the number of hidden and input connections used during data processing, compared with a of the network s hidden and input connections, respectivey. These parameters aowed checking if hidden Sigma-if neurons use their seective attention abiity in practice. For the competeness of the anaysis, for each given probem and trained network, the percentage of a inputs used to cassify a test vectors niu was cacuated. This was important in order to determine if seective attention functionaity is aso reaized at the eve of the whoe Sigma-if network. A measured vaues were cacuated as averaged outcomes of ten independent 10-fod cross vaidations. To precisey check how seective attention abiities of the Sigma-if network infuence the properties of the resuting modes, beside generaization γ of the networks that were fina resuts of each training, cassification performance was measured aso for each network mode generated in each step of backpropagation during vaidation steps. This aowed finding out how seective attention changes maxima cassification accuracy of test data (γ m reachabe by the networks generated during one average training. For networks with the greatest γ m, aso cassification accuracy for training data (u m was measured. Again, to reduce the infuence of initia network weights seection on the resuts, a cassification accuracies were averaged for a steps of ten independent 10-fod cross vaidations. It must be stressed that cassification performance of networks measured during backpropagation was not used to contro the training process. The reason to coect additiona data was checking for possibe unpredicted infuence of using Sigma-if neurons on the course of the training process. During experiments, aso the average data processing time τ of the input vector for a trained networks was measured to check reative data processing costs of MLP and Sigma-if networks. A time measurements were conducted on a singe dedicated computer with a 2.6 GHz processor. Regardess of the very precise time measurement procedure used, actua timings on other hardware setups may vary consideraby. But the presented resuts can sti be used to show the order of possibe gains for time-critica appications if the Sigma-if network is used. Other parameters, such as the aggregation threshod ϕ and the grouping vector actuaization interva ω, were set to 0.6 and 25, respectivey, foowing preiminary tests. During those tests, aso the number of hidden neurons for each probem was preseected to the vaue for which a mutiayer feedforward neura network with sigmoida neurons achieved the owest average generaization error during ten independent 10-fod cross vaidations. The backpropagation stop condition, identica in a experiments, was using two constant threshods to check if the training agorithm reached given the cassification accuracy of training data or the maxima number of 00 training epochs. The obtained resuts indicate that increasing the num- [%] m s um u Fig. 1. Time of Sigma-if network output signa generation τ,the cassification accuracies of training and test data for the fina (u and γ and for the best networks obtained (u m and γ m for the Sonar probem vs. the number of hidden neuron input connections groups (network architecture: inputs, 30 hidden neurons, 2 outputs [ s]

7 Backpropagation generaized deta rue for the seective attention Sigma-if artificia neura network 455 [%] m s um u Fig. 2. Time of Sigma-if network output signa generation τ, the cassification accuracies of training and test data for the fina (u and γ and for the best networks obtained (u m and γ m for the HeartC probem vs. the number of hidden neuron input connections groups (networks architecture: 28 inputs, 10 hidden neurons, 2 outputs [ s] 12.5 [%] m s um u Fig. 4. Sigma-if network activity of hidden connections hca, the cassification accuracy of test data for the fina (γ and the best networks obtained (γ m for the Mushroom probem vs. the number of hidden neuron input connections groups (network architecture: 125 inputs, 2 hidden neurons, 2 outputs [ s] ber of Sigma-if neuron input groups to more than one resuts in an increase in generaization γ and cassification accuracy of test data γ m of the best networks obtained during trainings. At the same time one can observe a simutaneous decrease in the overa data processing time τ. The drawback here is a decrease in the cassification accuracy of training data u aso visibe in the case of the best generated networks (parameter u m. Typica exampes of such dependencies, for sma and medium size benchmark probems such as Wine, Votes, Crx or Wisconsin Breast Cancer, can be observed for the Sonar and HeartC probems, which are presented in Figs. 1 and 2 (as the number of input connections groups is discrete, vaues in the presented figures are connected with ines ony to ease the anaysis of the resuts. For arger probems, e.g., Adut and Mushroom (Figs. 3 and 4, the increase in γ and γ m for given parameters is at most sma and can be observed [%] m s um u Fig. 3. Time of Sigma-if network output signa generation τ,the cassification accuracies of training and test data for the fina (u and γ and for the best networks obtained (u m and γ m for the Adut probem vs. the number of hidden neuron input connections groups (networks architecture: 105 inputs, 4 hidden neurons, 2 outputs [ s] ony for numbers of input groups ess than five. The observed decrease in the cassification accuracy of training data is most probaby caused by the fact that it is harder to earn when the neuron s input space is changed every ω epoch. It shoud aso be remembered that, especiay for neura networks with a arger number of inputs, a ow vaue of the aggregation threshod ϕ can have significant infuence on the network performance both on training and test data, indirecty setting a strong imit on the number of network inputs being processed for greater vaues of. In the case of the Mushroom data set for =11,the increase of ϕ from 0.6 to 1.8 resuted in an increase of the average vaues of u, γ and γ m to 99.3±0.9%, 99.4±0.7% and 99.8 ± 0.7%, respectivey, whie average vaues of the activity of hidden connections hca, the activity of network inputs nia and the number of inputs niu used were sti as ow as 11 ± 4%, ± 7% and 52 ± 9% (cf. Figs. 4 and 9. Thus by tuning parameters of the Sigma-if network to the probem size one can achieve very good resuts aso for big data sets. However, and more importanty, for a benchmark probems considered, the obtained increase in the cassification accuracy of test data (γ and γ m is a resut of reecting redundant or noisy signas from processed data and the consequence of the reduction of probem compexity by decreasing its dimensionaity. Another source of such properties of the Sigma-if network is spitting the initia probem into a set of subprobems due to muti-step, conditiona generation of neurons outputs. In turn, a decrease in the network s outputs generation time τ is caused by reduction of the network s hidden connections activity hca (see Fig. 5 for Sonar and Fig. 6 for Votes probems. It is aso worth noting that the visibe increase in the HeartC data processing time τ, for greater than seven inputs groups, is the effect of a inear increase in the time cost connected with the existence of additiona instruc-

8 456 M. Huk tions for processing the grouping vector θ. This factor can be easiy seen for a benchmark probems considered for the numbers of groups greater than the given number of network inputs. Without it, the data processing time woud semi ogarithmicay decrease with rising. This refects the character of the changes of Sigma-if network hidden hca and input connection activities nia as a function of, which can be observed for the Votes probem in Figs. 6 and 8. Therefore the obtained strong reduction of hidden connections activities confirms earier concusion that the data processing time reduction is connected with Sigma-if neurons seective attention abiities, which can be observed aso for arge probems in Figs. 7 and 9 (Adut and Mushroom probem, respectivey. A this is cear evidence that Sigma-if neurons use seective attention, and that this can reduce the generaization error eve as we as data processing costs. The conducted experiments discose aso that for a Sigma-if mode generated with the use of the presented training method, seective attention can be observed on the eve of the whoe Sigma-if network. The anaysis of resuts indicates (Figs. 7 9 that, when a significant decrease in network input activity nia occurs, one can expect a simutaneous reduction in the number (niu ofsigma-if network inputs used to cassify data, without a notabe decrease in cassification accuracy in comparison to the anaogous mutiayer feedforward network with sigmoida neurons. The above resuts form strong evidence that the presented generaized deta rue for the Sigma-if neura network can be effectivey used to generate vauabe cassification modes with seective attention functionaity. But it is aso interesting how such a method infuences the ength of the training process. As can be seen in Fig. 10 for chosen benchmark cassification probems such as Sonar, Crx, Wine, HeartC and Breast Cancer Wisconsin, the training, [%] m m max hca avg hca min hca Fig. 6. Sigma-if network activity of hidden connections hca, the cassification accuracy of test data for the fina (γ and the best networks obtained (γ m for the Votes probem vs. the number of hidden neuron input connections groups (network architecture: 48 inputs, 2 hidden neurons, 2 outputs. of the Sigma-if neura network (for greater than 1 and ess than 9 takes 25% fewer training epochs than the training of the MLP network ( = 1. In connection with observed 25 % reduction of the computation time for the Sigma-if network outputs, this can acceerate the training process even more than twice. 6. Summary and future work In this work the generaized deta rue for the Sigma-if neura network was formay presented. Its detaied derivation was shown on the basis of an anaogous derivation for the MLP network. For competeness, the backpropagation agorithm combined with the sef-consistency idea was discussed, as the training method which can use the derived equation to train Sigma-if neura networks. In the second part of this artice, resuts of experiments that demonstrate the usabiity of the derived hca [%] u, [%] m m m u m max hca avg hca min hca Fig. 5. Sigma-if network activity of hidden connections hca, the cassification accuracy of training u m and test γ m data for the best networks obtained for the Sonar probem vs. the number of hidden neuron input connections groups (network architecture: inputs, 30 hidden neurons, 2 outputs. 0 hca [%] u, [%] m m m niu max nia avg nia min nia Fig. 7. Sigma-if network inputs activity nia, the number of network inputs used niu, the cassification accuracy of training u m and test γ m data for the best networks obtained for the Adut probem vs. the number of hidden neuron input connections groups (network architecture: 105 inputs, 4 hidden neurons, 2 outputs nia, niu [%]

9 Backpropagation generaized deta rue for the seective attention Sigma-if artificia neura network 457 u m, [%] m 70 m u m niu max nia avg nia min nia nia, niu [%] Number of training epochs Sonar Crx Wine HeartC BrcW Fig. 8. Sigma-if network inputs activity nia, the number of network inputs used niu, the cassification accuracy of training u m and test γ m data for the best networks obtained for the Votes probem vs. the number of hidden neuron input connections groups (network architecture: 48 inputs, 2 hidden neurons, 2 outputs. equation were shown. For seected cassification benchmark probems of the UCI Machine Learning Repository, trained Sigma-if networks were abe to achieve better cassification resuts than the best MLP networks. But what is more important, the obtained Sigma-if neura networks possessed aso the seective attention abiity. It was shown how it increases neura network cassification properties and how it reduces the time of data processing by the network. The resuting effect of training epochs number reduction was aso discussed. Whie the Sigma-if network has no speciaized or separate attention guiding unit, a observed attentiona activities can emerge ony as an effect of synergy between individua neurons. Thus the Sigma-if mode accompanied with the presented training method can be a very promising soution for appications such as remote sensing in dispersed sensor networks as we as automatic u, [%] m m m u m niu max nia avg nia min nia Fig. 9. Sigma-if network inputs activity nia, the number of network inputs used niu, the cassification accuracy of training u m and test γ m data for the best networks obtained for the Mushroom probem vs. the number of hidden neuron input connections groups (network architecture: 125 inputs, 2 hidden neurons, 2 outputs nia, niu [%] Fig. 10. Number of training epochs of the backpropagation agorithm for the Sigma-if network for seected UCI Machine Learning Repository probems vs. the number of hidden neuron input connections groups. robot navigation and contro. This is because the seective attention feature introduces new possibiities in the area of anayzing the network decision process via its inputs activity interpretation. This can point at features of given data sets that are most important for cassification, and hep to identify features that are irreevant, redundant or contaminated by noise. A this makes the Sigma-if neura network a very usefu too in the data acquisition and anaysis domain. Due to very interesting theoretica and practica properties, the Sigma-if mode and the presented training method shoud be further tested on benchmark and reaife data. Aso, the whoe idea of synchronized conditiona signas aggregation shoud be further expored, as aggregation functions other than the one considered in this work can be proposed, and for many of them derivation of the generaized deta rue can be chaenging. Preiminary experiments show that there exist at east a few of such aggregation functions which aow achieving even better resuts than presented in this work. Another issue worth exporing is examination if the Sigma-if network coud be successfuy trained with use of fast converging methods such as Broyden Fetcher Godfarb Shanno and Levenberg Marquardt. Those methods use oca approximates of Hessian matrix of the neura network error function, which can fai as the Sigma-if network in each training step operates potentiay in a different subspace of initia set of parameters. A this makes a wide and promising direction of research on neurona modes of ow-eve seective attention, and is presenty a subect of continuous investigation. References Broadbent, D. (1982. Task combination and seective intake of information, Acta Psychoogica 50(3:

10 458 M. Huk Desimone, R. and Duncan, J. (1995. Neura mechanisms of seective visua-attention, Annua Review of Neuroscience 18(1: Duch, W. and Jankowski, N. (1999. Survey of neura transfer functions, Neura Computing Surveys 2(1: Durbin, R. and Rumehart, D. (19. Product units: A computationay powerfu and bioogicay pausibe extension to backpropagation networks, Neura Computation 1(1: Fedman, J. and Baard, D. (1982. Connectionist modes and their properties, Cognitive Science 6(3: Ferguene, F. and Toumi, F.F. (09. Dynamic externa force feedback oop contro of a robot manipuator using a neura compensator Appication to the traectory foowing in an unknown environment, Internationa Journa of Appied Mathematics and Computer Science 19(1: , DOI: /v Fonseca, L., Jimenez, J., Leburton, J. and Martin, R. (1998. Sef-consistent cacuation of the eectronic structure and eectron-eectron interaction in sef-assembed InAs-GaAs quantum dot structures, Physica Review B 57(7: Gupta, M. (08. Correative type higher-order neura units with appications, IEEE Internationa Conference on Automation and Logistics, ICAL 08, Qingdao, China, pp Hager, G. and Toyama,. (1999. Incrementa focus of attention for robust visua tracking, Internationa Journa of Computer Vision 35(1: Houghton, G. and Tipper, S. (1996. Inhibitory mechanisms of neura and cognitive contro: Appications to seective attention and sequentia action, Brain and Cognition 30(1: 43. Huk, M. (04. The sigma-if neura network as a method of dynamic seection of decision subspaces for medica reasoning systems, Journa of Medica Informatics & Technoogies 7(1: Huk, M. (06. Sigma-if neura network as a use of seective attention technique in cassification and knowedge discovery probems soving, Annaes UMCS Informatica AI 5(2: Huk, M. (09. Learning distributed seective attention strategies with the Sigma-if neura network, in M. Akbar and D. Hussain (Eds., Advances in Computer Science and IT, In-Tech, Vukovar, pp Indiveri, G. (08. Neuromorphic VLSI modes of seective attention: From singe chip vision sensors to muti-chip systems, Sensors 8(9: orbicz, J., Obuchowicz, A. and Uciński, D. (1994. Unidirectiona networks, in L. Boc (Ed., Artificia Neura Networks: Foundations and Appications, Akademicka Oficyna Wydawnicza PLJ, Warsaw, pp örding,. and önig, P. (01. Neurons with two sites of synaptic integration earn invariant representations, Neura Computation 13(12: Me, B. (19. The sigma-pi coumn: A mode of associative earning in cerebra cortex, Technica report, CNSMemo 6, Computation and Neura Systems Program, Caifornia Institute of Technoogy, Pasadena, CA. Me, B. (1992. The custeron: Toward a simpe abstraction for a compex neuron, in J. Moody, S. Hanson and R. Lippmann (Eds., Advances in Neura Information Processing Systems, Vo. 4, Morgan aufmann, San Mateo, CA, pp Nevie, R. and Edridge, S. (02. Transformations of sigmapi nets: Obtaining refected functions by refecting weight matrices, Neura Networks 15(3: Niebur, E., Hsiao, S. and Johnson,. (02. Synchrony: A neurona mechanism for attentiona seection?, Current Opinion in Neurobioogy 12(2: Noh, T., Song, P. and Sievers, A. (1991. Sef-consistency conditions for the effective-medium approximation in composite materias, Physica Review B 44(11: Noton, D. and Stark, L. (1971. Scanpaths in saccadic eye movements whie viewing and recognizing patterns, Vision Research 11(9: Oshausen, B., Anderson, C. and Van Essen, D. (1993. A neurobioogica mode of visua attention and invariant pattern recognition based on dynamic routing of information, The Journa of Neuroscience 13(11: Pedro, J. O. and Dahunsi, O.A. (11. Neura network based feedback inearization contro of a servo-hydrauic vehice suspension system, Internationa Journa of Appied Mathematics and Computer Science 21(1: , DOI: /v Raczkowski, D., Canning, A. and Wang, L. (01. Thomasfermi charge mixing for obtaining sef-consistency in density functiona cacuations, Physica Review B 64(12: Rumehart, D., Hinton, G. and McCeand, J. (1986. A genera framework for parae distributed processing, in D. Rumehart and J. McCeand (Eds., Parae Distributed Processing: Exporations in the Microstructure of Cognition: Foundations, Vo. 1, The MIT Press, Cambridge, MA, pp Stark, L., Privitera, C. and Azzariti, M. (00. Locating regions-of-interest for the mars rover expedition, Internationa Journa of Remote Sensing 21(17: Treisman, A. (19. Contextua cues in seective istening, Quartery Journa of Experimenta Psychoogy 12(4: Tsotsos, J., Cuhane, S. and Cutzu, F. (01. From foundationa principes to a hierarchica seection circuit for attention, in J. Braun, C. och and J. Davis (Eds., Visua Attention and Cortica Circuits, MIT Press, Cambridge, MA, pp Vanruen, R. and och, C. (03. Visua seective behavior can be triggered by a feed-forward process, Journa of Cognitive Neuroscience 15(2:

11 Backpropagation generaized deta rue for the seective attention Sigma-if artificia neura network 459 Weber, C. and Wermter, S. (07. A sef-organizing map of sigma-pi units, Neurocomputing 70(13 15: Macie Huk works at the Institute of Informatics of the Wrocław University of Technoogy, Poand. He received the M.Sc. degree in 01 and the Ph.D. in 07, both in computer science. His current research interests within the scope of artificia inteigence are the theory and appications of artificia neura networks in seective attention systems, efficient crossover operators for genetic agorithms and mutipe cassifier systems. He aso works on distributed sensor networks and contextua data anaysis. He is the coordinator of the Seective attention in data anaysis research group within the Poish Custer on nowedge and Innovation Community for Information and Communication Technoogies. Currenty he aso works as a software architect for the Gigaset software deveopment center. Received: 22 November 10 Revised: 14 June 11 Re-revised: 19 October 11