Minimizing Distribution Cost of Distributed Neural Networks in Wireless Sensor Networks

Transcription

1 1 Minimizing Distribution Cost of Distributed Neura Networks in Wireess Sensor Networks Peng Guan and Xiaoin Li Scaabe Software Systems Laboratory, Department of Computer Science Okahoma State University, Stiwater, OK {pguan, Abstract This paper presents a nove study on how to distribute neura networks in a wireess sensor networks (WSNs) such that the energy consumption is minimized whie improving the accuracy and training efficiency Artificia neura network (ANN) earning has been shown robust to noisy and uncertain sensory data for function approximation and pattern cassification appications With the advances of miniature hardware technoogies for powerfu sensor nodes, embedded neura networks wi emerge as important decision-making brains for WSNs and vast surveiance appications to enabe adaptive data quaity and sef-managing capabiities To distribute neura networks in WSNs in an energy-efficient manner, we propose parae transmission and adaptive neura seection agorithms(ansa) in mutiayer backpropagation(mlbp) earning process of neura networks, which is a popuar supervised earning technique used for training feedforward artificia neura networks We further anayze the energy consumption components in the onine training process and evauate the reduced energy consumption using our proposed agorithms I INTRODUCTION Wireess sensor networks (WSNs) have ushered in enormous opportunities bridging the gap between information systems and the physica word [1], [2] Meanwhie, WSN introduces new chaenges due to imited resources incuding energy Artificia neura network (ANN) earning has been shown robust to noisy and uncertain sensory data for function approximation and pattern cassification appications [3], [4], [5] With the advances of miniature hardware technoogies for powerfu sensor nodes, embedded neura networks wi emerge as important decision-making brains for WSNs and vast surveiance appications to enabe adaptive data quaity and sef-managing capabiities [6], [7] Motivated by this potentia significance, we investigate how to distribute neura networks in WSN such that the energy consumption is minimized whie improving the accuracy and training efficiency Characterized by imited resources and mutihop wireess inks, WSN has imited computation and communication capabiities and each node is typicay suppied with battery power However, training a neura network entais significant amount of computation and communication, which resuts in significant energy cost Energy consumption is infuenced by two main factors: computing and communication The radio communication is considered the most expensive one in terms of energy consumption To embed neura networks into a sensor network, it requires to substantiay reduce communication and energy costs We consider such a neura network, where each ayer is partitioned into a number of disjoint bocks and each bock is mapped onto a sensor node Assuming ANN is distributed eveny among sensor nodes, the computation and communication costs are anayzed according to each sensor node and their interactions To reduce communication and energy costs, we propose a parae transmission scheme which takes advantage of the pipeined communication to reduce communication overheads, and ANSA to further remove redundant neurons so as to reduce communication and energy costs and aso improve the training efficiency compared to traditiona methods without pruning The rest of the paper is organized as foows Section II briefy introduces the MLBP training agorithm and presents reated work Section III presents the improved transmission scheme and the adaptive neuron seection agorithm urther, Section IV presents numerica anaysis to demonstrate the improvement in energy consumption Section V concudes the paper I 1 I 2 I n 1 ig 1 II BACKGROUND AND RELATED WORK W 1 W n O 1 O L uy connected feedforward mutiayer neura network We consider feedforward neura networks where neurons are arranged in each ayer with the outer most ayers caed input ayer and output ayer [8], [9] The resut is a non-inear squashing function, which scaed between 0 and 1, and the output activity vaue is passed to the neurons in the next ayer No cyces are aowed inside, meaning each neuron propagates signas to ony ogica higher ayers ig 1 shows an exampe of mutiayer backpropagation neura networks n O 1 O 2 O n L

2 2 Based on gradient descent in error, MLBP can be viewed as a generaized east mean squared agorithm towards minimizing the sum of the squares of the errors between the actua and target outputs E = n L (EL i )2, where the non-inear error is given by, Ei L = Oi L (1 Oi L ) (T i Oi L ) (1) T i and Oi L are the target vaue and the activity output respectivey for the i-th neuron n L is the number of neurons in the output ayer The gradient descent equation is given by, Gd i = Gd i + α E i (2) where Gd i and E i are the gradient descent and the noninear error of i-th neuron on -th ayer, respectivey; and α is the earning rate So we use the BP agorithm by the foowing steps During each forward phase, we cacuate the output for each ayer using Oi = (Gd i + Wij O 1 j ) (3) n j=1 where Wij is the weight from the i-th input to the j-th hidden unit on -th ayer incuding output ayer, and ( ) is the sigmoid function (x) = 1 1+e x The backward phase incudes the cacuation of the error signa for each hidden ayers and update of the weight group, as prescribed by the foowing two equations n(+1) Ei = α Oi (1 Oi) (E +1 k W +1 ki ) (4) k=1 W ij = W ij + αe i O 1 j (5) for = L 1,, 1 To cacuate the errors effectivey, we introduce the foowing product expression, which wi be expained in the next section P P i = (E k W ki) (6) for k = 1, 2,, n We propose an improved parae transmission pattern and adaptive neuron seection agorithms to reduce energy costs and increase the convergence speed of traditiona MLBP agorithms for distributing neura networks in WSN Most works in sensor networks have been focused on designing efficient routing, ocaization, target tracking, and supporting systems [1], [10] Recenty, a handfu researchers started to appy neura networks in sensor networks [11], [12], [13] Kuakov et a [13] presented a combined approach using waveet method for data processing and ART neura network for cassification in wireess sensor networks However, they appy neura network agorithms as in other appications without expicity considering mapping ANN into WSN Recent papers [11], [12] studied the anaogy of neura networks and sensor networks considering mapping ANN into WSN to improve data quaity robustness Our investigation goes one step further to study dynamic reconfigurabe ANN to both minimize energy costs and improve data quaity and training efficiency III MODIIED MLBP ALGORITHMS A Anaysis of Communication and Computation of MLBP A neura network is eveny distributed among sensor nodes; a partition of the weight set and the corresponding neurons are mapped to each sensor node Energy cost in this setup consists of four components: communication and computation costs in both forward and backward phases C f (p) = C f comm(p) + C f comp(p) (7) C b (p) = C b comm(p) + C b comp(p) (8) where C f (p) and C b (p) denote the energy cost of a sensor in forward and backward phases, respectivey; C comm and C comp denote the energy cost due to communication and computation, respectivey We assume that C comm incudes the initia energy cost for initiaizing communication C i and energy cost of transferring unit data C u = C radio + C receive, where C radio, C receive represent the energy cost of transmitting and receiving In addition, energy cost due to oca data processing is aso counted In the forward phase, as the cacuation of an neuron activity vaue O in -th ayer needs a neuron activity vaues O 1 in the adjacent ower ayer, each sensor has to receive other sensor s data as iustrated in ig 2 The set of weight vaues in each sensor resides in rows of the matrix shown in Eq (13) It therefore needs n 1 transmissions of data to cacuate one of the outputs on the next ayer In terms of computation, we introduce C vo and C sig denoting the convoution and sigmoid computation respectivey Hence, the costs of MLBP forwarding process are given by, C f comm = (n 1) (C u + C i ) (9) C f comp = C vo + C sig (10) In the backward phase, simiary, there are two parts in energy costs The first part is for the top ayer L s error cacuation using Eq (1) and weight gradient descent update using Eqs (2) (5) The second part is for the other ayers = 1,, L 1 error cacuation by Eq (4) and weight Gradient descent updated by Eqs (2) (5) The difference between the first and second parts is caused by using Eq (1) or Eq (4) C b comm = 2 (n 1) (C u + C i ) (11) C b comp = 2 C vo + n C ex (12) where C ex represents the extra computation of summation and mutipication Note that athough communication and computation are both critica in wireess sensor networks, in genera communication costs are hundred to thousand times more expensive than computation costs O i = W ij O i (13)

3 3 node1 node2 node3 node(n-2) node(n-1) noden where C u is defined as the cost of unit transmission through a given distance shorter than that of C u Since the change of transmission pattern between sensors as iustrated in ig 3, the energy cost is ess than before node1 node2 node3 node(n-2) node(n-1) noden data sent from one node to another one computation for activity vaue time ig 2 Transmission process of MLBP [ Ei ] = [ E k ] W k (14) Therefore, the new transmission scheme wi change the route for acquiring adjacent output activity vaues or this pattern, we assume that each neuron cacuate its output activity vaue by obtaining adjacent ayer s data through sequentia shifting In order to reaize such a shifting, each sensor transfers its own adjacent ower ayer data to a neighbor sensor of one side whie receiving the other neighbor sensor s data from the other side Therefore it is ike that i-th sensor receives O 1 i 1 from (i-1)-st sensor and transmits O 1 i to (i+1)-st sensor, and then repeats to transmit the data it received ast oop before cacuating the activity vaue Assume that there exists n sensors for each ayer, then the n-th sensor transmits the data to the 1-st sensor as its next neighbor, and aso 1-st sensor assumes the n-th sensor as its previous neighbor to receive data from Eq (13) shows matrix expressions of convoution parts of Eq (3) In this pattern, this communication is done in pipeined fashion, during which cacuation of convoutions of Eq (3) using received data, O 1 j s, j=1,2,j, is done concurrenty and energy costs of such a scheme are expressed in Eq (15) (16) (17) (18) In forward phase, the Ccomm f and Ccomp f are given by, C f comm = (n 1)C u + C i (15) C f comp = C vo + C sig (16) or backward phase, the energy cost of the first part incudes the cacuation of Eq (2), which can be cacuated independenty in each sensor without any communication Whereas, Eq (4) and Eq (5) needs the other sensors data ike the forward phase, so it aso needs n 1 transmission of data to cacuate one of the outputs on the next adjacent ayer It is crucia to cacuate Eq (4), because this equation needs spread its weights to a of the sensors The way we use is ike the reverse pattern of data distribution in forward phase The other equations can be cacuated in the same way of sequentia shift as it in the forward phase C b comm = 2 (n 1) C u + C i (17) C b comp = 2 C vo + C ex (18) ig 3 data sent from one node to another one computation for activity vaue Transmission process using adaptive neuron seection In the second part, Eqs (2) (4) (5) are cacuated repeatedy L-1 times from =L-1 to 1 As described before, weights are distributed on each neuron, which provide the possibiity to cacuate O j without a communication of weight W ij Eq (14) show matrix expressions of convoution parts of Eq (4) The set of weight vaues in each sensor resides in rows of the matrix shown in Eq (14), though the set of weight vaues which each sensor needs when cacuating Eq (4) resides in coumns of the weight matrix in Eq (14) In order to cacuate Eq (4), we empoy the product shifting Such technique in a n sensors system is shown in Eq (6), where P P represents a set of partia product in Eqs (4) And then Eq (4) can be cacuated by using summation over Eq (6) and additiona cacuation, which is a coumn i s partia product of the rows which are hod in each sensor In order to summate a product of the same coumn i, each sensor transfers the sets of product to the corresponding sensor The technique is an summation of product for cacuating Eq (4), which is consist of two steps The first step is the cacuation of Eq (6) for a coumns in each sensor In the second step, the product shifting and auxiiary cacuation of Eqs (4) in each sensor s oca memory are repeated in the number of sensor minus 1 times After n 1 repetition, each sensor obtains the accumuation of a PP s which has the same coumn number, and abe to cacuate Eqs (4) Such an agorithm of energy cost in the backward phase is presented in Eq (19) or each ayer s energy cost, using Eq (7) and Eq (8), we obtain the cost of -th ayer C() as foows, C() = (C f (p i ) + C b (p i )) n Therefore, for a ayers, we have, C = = L n (C f (p i ) + C b (p i )) =1 (19) L n 3(n 1)C u + 2C i + 3C vo + C sig + C ex =1

4 4 B Adaptive Neuron Seection for Mutiayer Neura Networks In the previous section, we appied the new transmission pattern to reaize the training procedure, resuting in ess energy costs in WSNs In this section, we further empoy ANSA for the weight group to prune redundant neurons during each training epoch Reca that the proposed energy efficiency scheme of MLBP earning procedure aims to estabish an optima neura network through neuron seection in each epoch The detais of the neuron seection scheme are presented in this section The mode of earning procedure is ring shaped communication among sensors, as iustrated in ig 3 The eigenvaue change of covariance matrix appears the most significant variation occurred in the weight group or a weight group, we have severa weight sets for each cyce in a specific earning epoch, which provides the determinant indicating how much energy has been consumed [14], [15] In this paper, we use the characteristic of determinant for each neuron to seect perceptrons after each training epoch [16] to optimize the neura network or simpicity we define the neura network as a fuy connected feedforward mutiayer network containing L ayers and I neurons for each ayer at the very beginning of training, and the weight vector Wi is defined by Wi = [Wi1, W i2,, W ii ]T The covariance matrix of Wi is cacuated by cov(w i ) = E[(Wi E[W i ])(W i E[W i ])T ], where Wi represents the weights vector of i-th neuron on -th ayer, i = 1,, I and = 1,, L During the whoe earning procedure of severa epoches, we try to obtain a group of weights, which satisfies the east mean squared error for the given probem In this perspective, the earning process tries to adjust weights of a ANN to find the optima group of weights Lots of researches on weights distribution of ANN reveaed that the resuting set of weights can be quite different, though outputs of the ANN are quite simiar [17] According to such concusion, it eads us to assume that there exist a few soutions for one given probem, meaning the desired weights set are not unique Then we can save the energy of training ANN distributed in WSN by pruning redundant neurons In each epoch of training, we assume there is no difference among weights on each ayer for optimizing the whoe ANN Therefore, we prune neurons possessing minimum determinant using the foowing equation: N(, i) = arg min W i W ( cov(w i ) ) (20) Since the whoe earning procedure consists of a number of epoches, the variations of average weight change are aso dynamic during the earning period To reduce energy costs, for each neuron N i, we propose to prune its neuron N i based on the foowing rue: TABLE I ADAPTIVE NEURON SELECTION ALGORITHM (ANSA) 1 acquiring I i from n different paces for each sensor,transmitting I i to (i + 1)st neuron,the first sensor is reviewed as the next one of the ast one unti a sensors get a input for oca cacuation of output activity; 2 cacuate the output activity of 1 th ayer and transmit each output activity vaue Oi to (i + 1)st neuron, the first sensor is reviewed as the next one of the ast one; 3 repeat step 2 unti a sensors get a output activity vaue from adjacent ower ayer; 4 repeat step 3 unti finishing the feedforward process for a L ayers; 5 cacuate a error signas of L ayer and transmit partia products of them through sequentia shifting and cacuate the error signas of the adjacent ower ayer unti reach 1 st ayer 6 repeat forward and backward processes unti reach the one before the given cyces of each training epoch 7 do the forward training as before, and then cacuate the determinant for each neuron on L th ayer whie doing backward process on L th ayer, transmitting the minimum θ n L to the next ayer unti reach the 1 st ayer 8 if the east mean squared vaue satisfies the requirement then stop training, if not, prune the avaiabe minimum neurons and do next epoch { 0, cov(wi ) > θ Ψ(, i) P (N(, i)) = 1, cov(wi ) < θ Ψ(, i) (21) L n =1 Ψ(, i) = cov(w i ) L n (22) =1 where 1 θ is in the range of 005 to 015 based on the scae of ANN and the accuracy criterion The P ( ) function determines the neurons of network in each epoch period, and then hats the negative neurons by testing whether the eigenvector change of an certain neuron is more negative than a given criterion or not in order to ameiorate the energy consumption for MLBP training process of the whoe network During each epoch of training ANN wi mitigate its communication workoad through optimizing neuron group Starting from a fuy connected feedforwad neura network with n neurons on each of the L ayers, the operations of ANSA are described in Tabe I Therefore, the entire energy cost of each training cyce is given by C = L =1 θ n 3( θ n 1)C u + 2C i + 3C vo + C sig + C ex where 1 θ = L θ, The energy cost of each epoch can be generaized as C epoch = γ L e=1 =1 θ n (23) (C f (p i ) + C b (p i )) (24) where γ is the iteration of training cyces for each epoch

5 5 IV NUMERICAL ANALYSIS O ENERGY COST ANSA scheme ig 5 shows the decrease of inactive nodes during each epoch training procedure In ig 5, we choose γ = 50 to evauate the energy cost for whoe network rom the figure, we observe that that, as the training epoch proceeds, the seected neurons are graduay reduced This resut is due to the convergence characteristic of ANN training agorithm In the fina steps of the training process, the network wi enter a steady stage unti reach the given east mean squared vaue ig 4 Energy cost of sensor network with L =1 n neurons by using adaptive neuron seection parae agorithm, where γ = 50 V CONCLUSION We have presented a modified MLBP agorithm for distributing artificia neura network (ANN) in WSNs To enabe inteigent decision making and sef-adaptation for data quaity assurance in an energy-efficient way, ANN is emerging to pay important roes in WSNs As the resuts show, using the determinants of neurons to prune the most negative neurons in the sensor network wi reduce energy cost and proong the ifetime of the whoe WSNs Our ongoing work is to deveop an efficient scheme suitabe for imited memory in sensor nodes and an embedded reconfigurabe sef-adaptive ANN in a rea-word sensor network for target appications such as fire detection and precision agricuture ig 5 Improvement percentage each epoch The modified MLBP earning scheme is appied to WSNs to reduce energy costs and acceerate the convergence speed or each pattern cassification probem, assume that the weight and bias are initiaized by a uniform distribution, and the training procedure of ANN is performed unti the difference between the cassification error estimated by east mean squared vaue is smaer than 1% 5% based on various criteria of appications To evauate the scheme, we have estabished the fuy connected neura network mode, choosing γ = 50 and θ [085, 095] to evauate different energy costs in different scenarios ig 4 shows the tota energy cost with different θ vaues As expected, we observe that energy cost increases as the scae of ANN grows or ANN with over 80 neurons, the new scheme resuts in 1% 2% energy saving Through comparing traditiona MLBP with the new scheme, we found the decrease of energy cost depends more on the scae of network and determinant distribution of the whoe network Since the communication energy is infuenced significanty by the scae of sensor network, the communication times between sensors coud be reduced by the REERENCES [1] D Cuer, DEstrin, and M Srivastava, Overview of sensor networks, IEEE Computer, vo 37, no 8, pp 41 49, 2004 [2] D Estrin, D Cuer, K Pister, and G Sukhatme, Connecting the physica word with pervasive networks, Pervasive Computing, IEEE, vo 1, no 1, pp 59 69, Jan Mar 2002 [3] C Bishop, Neura Networks for Pattern Recognition Oxford, 1995 [4] R O Duda, P E Hart, and D G Stork, Pattern Cassification (2nd Edition) Wiey-Interscience, 2000 [5] T M Mitche, Machine Learning McGraw-Hi, 1997 [6] M Hoenderski, J Lukkien, and T C Khong, Trade-offs in the distribution of neura networks in a wireess sensor network, in Inteigent Sensors, Sensor Networks and Information Processing Conference 2005 [7] I P Stoyanov, An improved backpropagation neura network earning, in ICPR 1996, Pattern Recognition, 1996 [8] S Abid, naiech, and M Najim, A fast feedforward training agorithm using a modified form of the standard bankpropagation agorithm, in IEEE Transactions on neura networks, vo 12, 2001 [9] K Hara and K Nakayama, Seection of minimum training data for generaization and on-ine training by mutiayer neura networks, in IEEE ICNN (Neura Networks) 1996 [10] K Whitehouse, C Karof, A Woo, Jiang, and D Cuer, The effects of ranging noise on mutihop ocaization: an empirica study, in Information Processing in Sensor Networks, 2005 IPSN 2005 ourth Internationa Symposium on, 2005, pp [11] Odewurte and P Mahonen, Neura wireess sensor networks, in Systems and Networks Communication, ICSNC 2006 [12] J Patra, E Ang, N Chaudhari, and A Das, Neura-network-based smart sensor framework operating in a harsh environment, in EURASIP Journa on Appied Signa Processing, 2005, pp [13] A Kuakov, D Davcev, and G Trajkovski, Appication of waveet neura-networks in wireess sensor networks, in irst ACIS Internationa Workshop on Sef-Assembing Wireess Networks, 2005 [14] J Go and C Lee, Anayzing weight distribution of neura networks, in IJCNN 1999, vo 2, Ju 1999, pp [15] J Go, B Baek, and C Lee, Anayzing weight distribution of feedforward neura networks and efficient weight initiaization, in IJCNN99 [16] I ieser and E Beido, Do backpropagation trained neura networks have norma weight distributions? in ICANN 1993 [17] P Pames, T Hayasaka, and S Usui, Evoution and adaptation of neura networks, in IJCNN 2003