Training Spiking Neuronal Networks With Applications in Engineering Tasks

Transcription

1 Training Spiking Neuronal Networks Wit Applications in Engineering Tasks Pill Rowcliffe and Jianfeng Feng P. Rowcliffe is wit te Department of Informatics at te Scool of Science and Tecnology (SciTec, University of Sussex, Falmer, Brigton, East Sussex, BN 9QH, U.K. ( J. Feng is wit Te Department of Matematics, Hunan Normal University, 48, Cangsa, P.R. Cina, and Te Centre for Scientific Computing and Computer Science, University of Warwick, Coventry, CV4 7AL, U.K. (

2 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER Training Spiking Neuronal Networks Wit Applications in Engineering Tasks Abstract Spiking neuronal models employing means, variances and correlations for computation are introduced. We present two approaces in te design of spiking neuronal networks, bot of wic are applied to engineering tasks. In exploring te input-output relationsip of integrate-and-fire neurons wit Poisson inputs, we are able to define matematically robust learning rules wic can be applied to multi-layer and time series networks. We sow troug experimental applications tat it is possible to train spike-rate networks on function approximation problems and on te dynamic task of robot arm control. Index Terms Integrate-and-fire, Mean ISI, Variance, Kernel, Robot Arm. I. INTRODUCTION IN recent years, tere as been significant growt in te field of biological computation. In tis time, a closer unification between neuroscience and artificially intelligent computational models, as been observed. As a result, computational neural models exist, owing more to teir biological counterparts tan previous classical artificially intelligent models. Voltage tresold models suc as integrate-and-fire (IF model [7], [], [9] [3], and te more biopysical Hodgkin-Huxley (HH model [6], [], all incorporate more of te dynamics of actual biological neurons tan te traditional classical approac to neural modelling; suc as te perceptron [5]. In trying to understand te computational properties of te brain it is necessary to understand te biopysical mecanisms involved in te process. Defining tese mecanisms wit computational models allows us to furter explore some of te complex and adaptive processes wic may be employed by biological neural systems. As suc, we ave seen te field of computational neuroscience grow considerably in recent years. A result of tis is te emergence of a variety of engineering applications, and learning rules, wic now employ biologically plausible computational models. Indeed, we ave seen many successful applications witin te fields of uman arm movement [9], computer vision [4] and speec recognition [3], [3], to name just a few. In considering te application of biologically plausible neural models we need to consider wic type of model to use, and ow best to address te issue of training. Many of te engineering applications wic ave applied biopysical models, ave used te IF model as te main computational unit. In using tis model, it is often te temporal sensitivity of te neuron wic is exploited witin computation, i.e. te time interval between successive spikes. In fact Bote et al. [3], used tis principal to develop an error regression learning rule to train a network of IF neurons. It is wort noting at tis point, tat te rule developed by Bote et al. in [3] is one of te few learning rules, applied to spiking networks, wic is not based on a Hebbian [5] approac to synaptic weigt modification. Indeed many of te learning rules, developed for use on spike-time dependent models [3], rely on Hebbian correlation as te principal means for synaptic weigt modification. However, as Bote et al. ave sown, wit teir backpropagation learning rule, Hebbian learning need not be te only approac to training IF neurons for use witin engineering. In [6] a single biologically plausible spike-rate model used a matematically derived backpropagation learning rule, to solve a non-linear tractable task. Like Bote et al. in [3], te learning rule was based on error regression. However, te neuron model used in [6] represented te IF model in terms of its firing rate. Defining te model in tese terms, provided a relationsip between te synaptic input of a neuron, and te firing rate output of te model. Te spike-rate model as presented in [6], provided bot first and second order statistical representation of te synaptic input. As suc, computational information is sown to be present in bot te mean and variance of synaptic input. Wen plotting te spiking rate output of te model against its synaptic input, [6] presented a series of firing rate output profiles, kernel-like in nature. One of te main observation about tis model is tat its output firing rate appears to owe more to radial basis function (RBF model tan its classical predecessor: te perceptron. As a unit of computation toug, te single spike-rate model as advantages over some classical models wit te inclusion of bot te mean firing rate and te variance of te firing rate. Wit many classical models, like te perceptron, if tere is an equal balance of excitatory and inibitory inputs, te mean effect on te model is zero. Wit te spike-rate model, tis is not te case. Te spike-rate model includes bot first and second order statistics: te mean and variance. Indeed, it was sown in [9], [6], tat even wen te mean input vanises, te model and te learning rule still function due to te synaptic variance. As suc, te spike-rate neuron is a computing model of bot te mean and variance. Te output surface planes for tis model ave been sown to be controllable by synaptic modification troug te use of te derived learning rule [6]. Te spike-rate model terefore, as potential engineering applications wic we will introduce as part of tis paper. Te advantages of using second order statistics over te classical approac, wic mainly use first order statistics, ave been known in literature for many years. In [] for example, Feng and Tuckwell introduced an optimal control task based upon te control of second order statistics, te variance, wic presented some interesting properties. Introducing second order statistics in computations is toug, a minimal requirement if we intend to implement stocastic computations; an example being te Bayesian approac. Hence te framework we present ere opens up te

3 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER possibility of carrying out a random computation in neuronal networks. Te computational power of spiking neuronal networks ave alreadly been explored in te liquid state macine [3] and te eco state macine [8]. In teir work, te computational performance was acieved due to te ig-dimensional projection of te low dimensional input space (a kernel property. However, in te approac we present ere, toug we employ te kernel property as a natural result of te spiking neuronal network, we do not use te inefficient computational projection of te input space to a ig dimensional space. In tis paper terefore, we consider te design and structure of a network of spike-rate neurons and wat is involved in training tese networks for use in engineering tasks. We propose two applications of tese models, for use on specific tasks, as a basis for investigating teir computational properties. Our approac as been to derive learning algoritms based on a multi-layered network of spike-rate neurons. We ave expanded on te learning rule introduced in [6], were we identified te input-output relationsip of te spike-rate model and applied an error minimisation tecnique to train te model. Te network designs introduced ere are specific to te tasks of function approximation and robot control, and ave similar structures to RBF networks. We terefore present one of te first applications of a network of spike-rate neurons and sow tat it is possible to train tese networks wit a matematically derived learning rule. Toug te networks take a longer period of time to train tan teir classical counterparts, tey do offer a significant advantage over classical AI models, in tat tey include bot te mean µ, and te variance σ, of te input signal. Tis paper is set out as follows. Section II will define te model of te single spike-rate neuron; te basis of wic we will use in te building of our neuronal network, wic we will define in Section III. A network learning rule is introduced in Section IV togeter wit te results from te function approximation tasks wic we present in Section V. Finally in Sections VI and VII we detail te approac we took in applying te spike-rate model to te task of robot arm control, defining te recurrent structure of a time series network, togeter wit te modified backpropagation troug time version of our network learning rule. II. THE MODEL DESCRIPTION First we present te definition of te spike-rate model. We begin by considering te IF model [6], [3], [3], defined many times in literature and presented ere as follows. Suppose a cell receives excitatory postsynaptic potentials (EPSPs at n of its synapses, and inibitory postsynaptic potentials (IPSPs at m of its inibitory synapses. Wen te membrane potential V (t is between its resting state V rest and its tresold V tre, it satisfies te following equation dv (t τ(v (t V rest dt + dīsyn(t ( were τ is te decay rate of te membrane, and Īsyn(t is te synaptic input Ī syn (t m wije E j (t wiji I j (t ( Here E j (t and I j (t are renewal processes for t ; and wij E >, wi ij > are te magnitudes of te EPSP and IPSP respectively. Te total current input into te neuron is summed over all n excitatory, and m inibitory, synapses. Wen V (t crosses te membrane tresold V tre from below, a spike is generated and te membrane resets to its resting potential V rest. However in [3], Tuckwell sowed tat jump processes, suc as E j (t and I j (t in equation (, can be approximated using diffusion approximations, suc tat E j (t λ E j t + λ E α/ j B E j (t I j (t λ I jt + λ I j α/ B I j (t were Bj E and Bj I are Brownian motions; λ E j, λi j are te synaptic input renewal process rates wit λ E j (λe,,λ E n, λ I j (λi,,λ I m, and α > te parameter, discussed in [], wic produces a Poisson process input wen α. We ave seen in [6], tat equation ( can be approximated as dv(t τ(v(t V rest dt + dī syn (t (3 were and µ i σ i + ρ + ρ ī syn (t µ i t + σ i B(t (4 m λ E j wij E λ I jwij I n,m m (λ E j α (wij E + (λ I j α (wij I j k n,m j k (λ j E α (λk E α w E ij w E ik (λ j I α (λk I α w I ij w I ik ere ρ is te correlation coefficient between te it and jt input, a detailed discussion of wic is given in [8]. It is wort noting ere, tat te model s description of te synaptic input, as given in equation (4, presents it in terms of input mean and variance. For simplicity of notation we next consider excitatory and inibitory inputs to be independent, wit w ij wij E wi ij, m n and λ I j rλe j rλ. Here r if te unit only receives purely excitatory inputs and r wen te unit receives equal excitatory and inibitory inputs. Equation (5 can now be rewritten as µ i λ jw ij( r σi ( λ α j wij + ρ j k λ j α λk α wijw ik ( + r α (5 (6

4 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 3 Equations (3, (4 and (6 represent te model in terms of its membrane potential. As stated above, te synaptic input used in tis model, given in equation (4, presents a synaptic input in terms of te mean input µ i and te variance σi, about tis mean. Te importance of tis can be seen in te case wen r, i.e. wen a neuron as an equal balance of excitatory and inibitory synaptic inputs. In tis case, te µ i term in (6, i.e. te mean input, disappears. However, te variance term σi, remains and as a result te neuron still receives synaptic activity. Next, we represent tis model in terms of its firing rate. Wit te model represented in tis way, we are able to see a direct input-output relationsip, previously sown to exist in [7], in wic computational information is encoded witin te firing rate of te model. To acieve tis, first let us define f i (λ as te firing rate output of te IF unit i, subject to synaptic input rates λ (λ,,λ n. We can write te firing rate in terms of te interspike interval as f i (λ T ref + < T i (r > were T ref is te refractory period, < T i (r > is te mean interspike interval of output unit i; note, i typically covers te output space say of i,,n. Te definition of te mean interspike interval as previously been given in [6] as < T i (r > τ V tre τ µ i σ i (7 g(xdx (8 V rest τ µ i were τ is te decay rate of te IF model, and g(x, known as Dawson s Integral, is defined as g(x exp(x x exp( u du (9 Now, te model, as presented in equations (3 and (4, gives te neuron in terms of its mean firing rate and its standard deviation. As stated earlier, in classical artificial neural networks, for a model wit equally balanced excitatory and inibitory synaptic inputs, te firing rate activity is silent. In te neural model presented ere, it can clearly be seen tat tis is not te case. In Figure we see an example of te case wen r in (6, were te mean term µ i disappears, but te variance σ i term remains. Te figure plots of te firing rate output of a model wit synapses wit varying inputs λ (,, and λ (,,. Equation (7 presents te IF neuron in terms of its firing rate output and its mean interspike interval. It is tis aspect of te model wic as motivated muc of te remainder of tis paper. By examining te variance term of a neuronal model, rater tan trying to smoot out any noise element, we will examine its computational properties in its application to a series of engineering tasks. As suc, we will focus our experiments on te case of r. In doing so we will effectively ave a firing rate model wit synaptic input Output Input Input Fig.. A grapical plot of te firing rate output plane of a single neuron wit two synaptic inputs, were α and r. Input values were varied between - and. terms µ i σ i ( λ α j wij + ρ j k λ j α λk α wij w ik ( We will owever define te network and learning rules, in terms of a general approac, as suc a learning rule, and network design, sould old for te case wen r,...,. III. SPIKE-RATE NETWORKS Te kernel-like structure of te spike-rate model, similar to tat presented in figure, is an interesting property of te neuron model. Tese kernel-like output profiles were previously explored in [6], were similar outputs were observed for differing synaptic weigt configurations, i.e. differing values of r. Tese neural models appear to ave more in common wit RBF models tan teir classical predecessors, suc as te perceptron. Te spike-rate model s output for r, is very similar to tat of a multiquadratic function used in some RBF models. In comparing tis spike-rate model s output wit tat of a multiquadratic RBF, two important similarities ave been observed. Firstly, multiquadratic RBF networks use basis functions similar to tose introduced in [4], and are generally of te form y(x (x + c ( were x is te input, c > and x R. Te output of te spike-rate model defined in equation (7, is equivalent to tis function, wen < c < in (. Secondly, in [4] it was proven tat for a distinct set of N points in R m tere exists an N N interpolation matrix Φ, if te ji-t element φ ji φ( x j x i is non-singular. For a multiquadratic function to be non-singular, {x i } N i must be distinct. Tis is true for bot te multiquadratic RBF model and te spike-rate model over all real inputs. Tese two common features serve as a basis for te development of a network model based on te principles of RBF

5 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 4 network design. In te numerical models presented ere, spikerate neurons replace te basis functions presented in traditional RBF network arcitectures, and te network s output comprises a linear summation of tese spike-rate neuron s outputs. A. Kernel Neural Networks We now introduce our first design of a network of spike-rate neurons based on an RBF-style arcitecture. Te output profile of te spike-rate model, presented in figure, sows a kernel centred at zero wen input λ. We see tat a similar output is obtained in a multiquadratic RBF if c, in equation (, is set to zero. So we find tat by including centers for eac spike-rate neuron, similar to te approac taken in te positioning of basis units in RBF networks, we ensure te spike-rate network is positioned across its input space. In order to acieve tis terefore, equation (6 is modified as follows µ i (λ j λ i w ij( r ( n σi (λ j λ i α wij +ρ j k (λ j λ i α (λ k λ i α w ijw ik ( + r α ( were λ i is te centre for neuron i over te dimensions of te input space j,,n specific to neuron i, and T ref >. It is now possible to place individual neurons across te neuron s input space by a suitable coice of λ i. For n input nodes on te input layer, and M spike-rate neurons on layer i, te output from tis RBF-style network is defined as M y (λ w i f i (λ (3 i were y is te output from output node connected to neuron i by te weigt connection w i, as sown diagrammatically in figure. We observe tat te output y (λ is a special case of te spike-rate model presented in equation (7 if r is set to, i.e. wen tere are purely excitatory weigt connections for te neurons in te output layer of te network. Equation (6 terefore, can be rewritten in te form M µ f i(λw i σ i M f i(λ α wi + ρ i M i k f i(λ α f k (λ α w i w k (4 In te model defined in equations (3 and (4, te special case spike-rate neuron differs from te spike-rate model, discussed in section II, because it takes as inputs te firing rates from neurons in te previous layer. In a biological framework, neurons emit and receive spikes. Tis is not te case ere toug. We ave instead cosen to model an RBFstyle arcitecture as our initial step in te examination of te computational performance of a network of spike-rate neurons. It sould be noted owever, tat in [8] a teoretical framework as been presented for a network of IF neurons in terms of te first and second order statistics of te ISI. In te case tere, Fig.. Scematic plot of an RBF-style spike-rate network. Eac of te i,, M spike-rate neurons receives j,, n inputs λ, wit weigts w ij. Te output of te RBF-style network y is a summation of te product of te firing rate of eac of te spike-rate neurons f i (λ and te output layer s weigt connection w i. Feng et al. ave sown tat it is possible to build a neural framework wit a diffusion approximation for te renewal inputs, and tus an approximation to describe te beaviour of a network of IF neurons. IV. NETWORK TRAINING To support te learning rules we will present in tis paper, we ave included in appendix I, te derivation of a learning rule for a single spike-rate neuron, first introduced in [6]. Tis sows ow te synaptic weigts are updated in order to reduce te output error, during training. It is included ere so as to provide a broader picture in te training of te spike-rate neuron, at bot a single neuron level and a network level. In te previous section, we introduced a spike-rate network design similar to tat of an RBF network, i.e. wit an input layer, a idden layer of non-linear basis functions (or spikerate neurons in tis case and a linear output layer. We terefore coose to apply a similar learning algoritm to tose used in training RBF networks. Tis algoritm involves training te network in separate stages. Te first stage consists of centering eac of te spike-rate neurons across te input space. A k-means algoritm is used to identify te number of subsets witin te training data. Once k-means is complete, te output from tis stage will identify te optimum number of spike-rate neurons to use in te network, and were teir centres sould be positioned so as to cover te range of te input space. Te second stage in te training consists of a learning rule wic is used to update te weigts in te network. Te network is trained on te same data set used in te k- means section of te algoritm. Training is complete wen te network s output falls witin a specified error tolerance.

6 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 5 A. Stage One In tis first stage, te algoritm partitions te input training data sets into sub-groups. A set of training data is first identified, wic traditionally covers te input-output space. For tis type of network te input data is a set of synaptic inputs λ as defined in section II. Te remainder of te algoritm is defined as follows. Let a b be te set of input data points, were b,,b. For tese data points, employ te k-means algoritm [5] to partition tis data into a set of k vectors µ j were j,,k. Partition te input data {a b }, into K initial sets wic cover tis input space. Calculate te mean point of eac of te k sets µ j (t B b S j a b were µ j (t is te mean of te data points in set S j at iteration t. 3 For eac data point in {a b } calculate its distance from eac set s centre µ j (t, and reassign eac point to te set wit te closest mean. 4 Calculate te new mean for eac of te updated sets µ j (t + B b S j a b 5 Repeat until canges in te groupings stabilise. Te resulting centres of eac of te sets, are ten used as te centres for eac of te spike-rate neurons as presented in (. Here µ j λ j, wit te number of spike-rate neurons M, in te network design, equal to te number of sets partitioned wit te k-means algoritm, i.e. M k. Once te centres of te neurons ave been identified te unsupervised section of te training algoritm is concluded. Tis stage of te algoritm as now provided te optimum number neurons for te network, i.e. M k in (3, and te position of teir centres across te input space. Te network as defined in (3 can now be trained on te input data using stage two of te algoritm. B. Stage Two Tis second stage of te algoritm consists of a supervised learning rule. An approac similar to backpropagation [7], [34] is used to adjust te weigts between te output layer and te idden layer. Te Output Layer of an RBF-Style Network: Te error function E, also known as te sum of squares error, is defined as E P (d y (λ (5 were y (λ is te output of te network at neuron, as defined above in (3 and d is te desired target response of output neuron. Te error is calculated over te total number of output neurons P, in te output layer. Te output error ere depends upon te weigts w i and so any correction of tese weigts is proportional to te partial derivative of tis error. Tis gradient cange is obtained using te definition for y (λ in equation (3, suc tat E w i E y (λ (d y (λf i (λ y (λ w i (6 To compliment te learning rule presented above, at appendix II we ave included a proof concept for a version of tis learning rule wic can be applied to RBF-type networks wic use spike-rate neurons in te output layer. Toug it is not biologically realistic for a spike-rate neuron to ave firing rates as synaptic inputs, te proof completes te learning rule in terms of te backpropagation of te network error. V. APPLYING THE LEARNING RULE We now present a series of experimental results to illustrate te spike-rate network s ability to perform function approximation. Te network was tested on a variety of functions and we include two examples ere for consideration. Te two functions te network was trained to approximate are y(x ( + x x e x (7 y(x sin(x (8 In all experimental trials a single layered network of spike-rate neurons was used. Eac spike-rate neuron ad an equal balance of excitatory and inibitory synaptic inputs, i.e. r. Tese neurons were connected to an output node by a linear summation of te weigted connection between tem and te output node. An optimal learning rate of. was used, and a sample of input points taken from te curves in equations (7 and (8 were used to train te network. In figure 3(a a grapical representation is presented, sowing te network s output trougout te training process as it was trained to approximate function (7. Initially te network was set up wit equal weigt values of.5. Te training algoritm described in section IV was used to train te network, wic converged te output to witin an error of. by te 994t iteration. Figure 4(a sows te network s output wen trained to approximate function (8. Initially te network was set up wit equal weigt values of.5. Te algoritm trained te network to witin an error convergence of. in 976 iterations. Once training was complete, eac network was tested on te remaining input data, i.e. data wic ad not initially been sampled for use in te network training. Figures 3(b and 4(b sow ow te networks generalised tis data, for functions (7 and (8 respectively. In bot figures te dotted lines represent te original target output, and te tick black lines represent te network s approximation. Bot networks produced reasonable approximations using te data. VI. ROBOT CONTROL Following on from tis, a dynamic application for te spikerate network is investigated; namely a network designed to

7 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER Output Output Input Input (a (a Output Output Input Input (b (b Fig. 3. Figure 3(a sows te network s output wen trained to fit te curve defined by function (7. Te cange in te network s output (te tin grey line is sown at iterations intervals during te training. Te target in bot figures 3(a and 3(b, is represented by te dotted line. Figure 3(b sows te target output (te dotted line, and te network s output (te tick black line after training. Here te network is tested on data points witin te range [, ], wic were not part of te data set used in training te network. Fig. 4. Figure 4(a sows te network s output wen trained to fit te curve defined by function (8. Te cange in te network s output (te tin grey line is sown at iterations intervals during te training. Te target in bot figures 4(a and 4(b, is represented by te dotted line. Figure 4(b sows te target output (te dotted line, and te network s output (te tick black line after training. Here te network is tested on data points witin te range [, ], wic were not part of te data set used in training te network. act as a control mecanism for a simulated robot arm, in te task of goal locating. In order to accomplis tis, an artificial environment was designed, enabling arm data to be collected for use in te initial training of te network. Te artificial environment permits accurate and detailed observations to be made of te robot arm s performance and movement, bot during and after training. A. Te Two Joint Robot Arm Te robot controller and its environment are designed for te task of goal location. Te controller s input data is te set of Cartesian coordinates representing te target object s location witin a dimensional plane at figure 5. Te network s objective is to move te robot s arm towards tese target coordinates. Te arm itself consists of two sections of lengt l and l, and two joints represented by te angles θ and θ. Te first joint fixates te arm at a point in te plane, wit θ describing te circular movement of l about tis point. Te second joint is between te two sections l and l, wit θ describing te angular position of l about tis joint. Te network s outputs are te two joint angles θ and θ of te robot arm. Figure 5 sows a diagrammatic representation of te arm. A cange in tese angles directs te movement of te robot arm ( x + y l l θ arccos l l ( ( y l sinθ θ arctan arctan (9 x l + l cosθ

8 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 7 Fig. 5. A dimensional representation of a uman arm as seen from above. It sows te position of te joint angles θ and θ wic te network outputs in order to control te movement of te robot arm. Te lengt of te two arm segments are l and l, and te target position of te arm is represented by te Cartesian coordinate pair (x, y. Fig. 6. A grapical representation of time-series network, sowing te network unfolded over 3 time steps. Eac network as input nodes (te black squares, a single layer of spike-rate neurons (te wite circles and output nodes (te black circles. Here te network is duplicated over just 3 time steps, wit eac unit in eac network connected to its corresponding unit in te subsequent network by te weigt connection w (t ii, as represent in equation (. were x and y are te trajectory coordinates te arm traces during te movement it makes towards its target. It is possible to rewrite equation (9 in terms of tese Cartesian coordinates points: x l cosθ + l cos(θ + θ y l sinθ + l sin(θ + θ ( were l and l are te lengt of te robot arm segments. B. Design of te Neural Network In designing te network for use on tis task, we began by identifying te set of inputs-outputs wic would be employed by a robot controller. For te inputs tese would be te location coordinates of te object in te environment. For te outputs tese would be te robot arm s angular movement over time. How tis information is generated is discussed in section VI- D. We cose terefore, to expand on our RBF-style network, presented in section III-A, but wit te inclusion of recurrent connections. Tis approac, togeter wit te application of a Backpropagation Troug Time (BPTT learning rule [35], is one wic is perfectly suited to dynamic tasks, and one wic is commonly applied, to similar problems, witin te field of AI. By unfolding te network in time, we are able to treat te entire network as one large feedforward network. Using tis type of network, eac time step in te movement of te robot arm, i.e. t,..., T, is represented by a duplicate network. Eac network, at eac of te time steps, receives external inputs from te robot environment, as well as recurrent connections from neurons in previous time steps, as represented in figure 6. A neuron s output at time step t say, is used as part of te input for its equivalent neuron at te next time step t +. So for an RBF-style network, as described in section III-A, synaptic inputs into a spike-rate neuron i at time step t are comprised of two main elements: te normal synaptic input as introduced in (4 and a recurrent input, suc tat ī syn in (4 is rewritten as ĩ (t syn + ( λ j wij t ( rt λ α j (w(t ij + ρ B(t + f (t i (λ w (t ii j k λ j α λ k α w (t ij w(t ik ( + rα ( were ĩ syn is te synaptic input for a spike-rate neuron in a BPTT network; λ is te input into neuron i wic is external to te network; f (t i (λ i is te output from neuron i at te time step t (note for t, f ( i (λ i ; w (t ij is te magnitude of te postsynaptic potential for te network at time t; w (t ii is te time delay weigt connection between spikerate neuron i at time t and spike-rate neuron i at time t. In tis example, tere is no time delayed recurrent connection between te external inputs in te previous time step and te neurons in te current time step, since inputs external to te network remain constant for all t i.e. λ (t λ (t for all t,...,t and so we refer to tese synaptic inputs as λ. Also for all t, λ i,(t λ i,(t, i.e. te centre of eac spikerate neuron is constant for all t. To incorporate te time delayed inputs µ and σ are rewritten ( n ( µ (t i λ j λ i w (t ij ( r ( σ (t i +f (t i (λ w (t ii ( n (λ j λ i α (w (t +ρ j k ij (λ j λ i α (λ k λ i α w (t ij w(t ik ( + r α (

9 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 8 For an output node of an RBF-style network, as introduced in (3, te output y (t at time step t will terefore be wit and were y (t (λ M z (t i f (t i (λ < T (t i (r > τ i w (t i z(t i (3 f (t i (λ (4 T ref + < T (t i (r > Vtreτ µ(t i σ i (5 V rest τ µ (t i σ i g(xdx (6 wit being one of P units on te output layer and y (. For eac time step t,,...,t, eac network s weigt matrix can be represented as w,w,...,w T. So we see from (, tat not only does eac network receive inputs from external sources and oter neurons in te network, but also from neurons in previous time step. Te output node, is te weigted sum of te output it receives from all te neurons in its current network at time t, plus its recurrent connections wit previous output neurons over te previous t iterations. In definitions of time-delayed neural networks [33] te weigt connections below w (t i, in our definition given above, are set to zero. As tis network is a feed-forward time series network te omission of w (t i and below is appropriate since only information from te previous time step is directly passed forwards in te network. C. Dynamic Training Te training of tis type of recurrent network will require te development of a BPTT-style algoritm. To acieve tis te error function, defined in equation (5, is now defined for a single layered network. Using te network s output as sown in (3, te total error over te time periods t,...,t is given as E[,T] T P t ( d (t y(t (λ (7 were d (t is te desired response of te network to te input pattern λ at time t. To minimise tis error wit respect to te synaptic weigts a modified version of te BPTT algoritm presented in [36], is applied: Propagate information troug te network for te time interval t,...,t n ( < T n T, noting at eac stage te network s inputs, desired response and synaptic weigts. Perform a backwards pass to calculate te local gradient cange on te output layer: E[,T] (8 T (t (r δ (t for eac time step t, beginning wit t T and working backwards. 3 Adjust te weigts accordingly using: F w (t i w (t i w(t i ηf w(t i (9 were F is te ordered derivative [35] of te error function (7 wit respect to te weigts in te network. Tis is te feedback of any error in te network s output to tose weigts w (t i wic are responsible for tat output error. Eac of te networks for t,...,t will ave teir weigts adjusted using equation (9, were: T (t (r T (t (r w (t i E[,T] T (t (r T t δ (t w (t i T (t (r w (t i (3 is as defined in equation (4 and inputs are as stated in equation (. D. Generating Robot Data In order to generate a set of robot arm data for training and testing purposes, we tried to approximate some of te dynamics observed in uman arm movement. In [], [], [] and [], it was sown tat arm trajectories, between an initial starting point and a target goal, form an approximate straigt line. Wen measuring te velocities of te arm movement along tese trajectories tey appear bell-like in sape. Te movement begins wit an initial acceleration as te arm moves from its starting point towards te goal, and ten begins to decelerate as it approaces tis goal. To acieve an approximation of tis type of movement, to allow for te generation of training data, a series of (x i,y i coordinates are generated wic map out te arm s trajectory as it moves from its starting position towards its target goal. Given te initial starting point coordinates (, and te final target co ordinates (x, y tis straigt line is split into n segments x seg x n were x seg is a segment of te line. Using tis, a series of coordinate pairs is calculated using x i i x seg y i x i y x As eac layer of te network represents an iterative time step of equal lengt, velocity is terefore represented by te variation of te outputs of θ and θ from one time step to te next. Selecting data points from te trajectory data set, will generate te required velocity if te appropriate spacing is cosen between corresponding data points in order for tem to produce te desired bell-saped velocity.

10 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 9 (a (b 6 x 5 4 Error Iterations (c (d (e (f Fig. 7. Figure (a sows te target position, and trajectory, of te robot arm in locating te object goal, over 5 equal time intervals. Te position of te robot arm, at eac of te 5 time steps, is indicate by te tick black line wic represents te segments l and l as indicated in figure 5. Figure (b sows te trajectory te robot arm traces out after 39 training iterations. Figure (c is te total error output of te network during te training process. In figure (d te error over for eac network, at eac time step, is sown over te 39 training iterations. Figures (e and (f sow te cange in σ for eac of te two output units during te 39 training iterations.

11 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER (a (b (c (d (e (f Fig. 8. Figure (a sows te trajectory traced out by te robot arm in locating te goal at te Cartesian coordinates (,, and figure (b sows te trajectory traced out by te robot arm in locating te goal at te Cartesian coordinates (,. Te position of te robot arm, at eac of te 5 time steps, is indicated by te tick black line wic represents te segments l and l as indicated in figure 5. Te controller was initially trained to locate goal and ten goal. Figure (c sows te trajectory pats wen te controller was alternatively trained on goals and. Te error output during tis training process is sown in figure (d.figures (e and (f sow te cange in σ for eac of te two output units over 4 training iterations.

12 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER VII. ROBOT TRIALS Grapical results are now presented sowing te output wen a feed-forward time series network was trained in robot arm control for use in goal location. In tese experimental trials, networks of varying sizes were trained to locate a variety of points witin te robot arm environment. In eac case te BPTT algoritm presented in section VI-C was used to train te network. In most experiments, te learning rate η remained fixed at.. Single Goal Locating: Te results presented in figure 7 are from one of te experiments performed to test te network s performance wen trained to locate a single goal in its environment. In tese experiments te network consisted of input nodes wic were fully connected to spike-rate neurons, wic were in turn connected to output units. Te recurrent connections were only between neurons on similar layers. Te time series of te network was for 5 equally space time steps. Figure 6 sows a similar network arcitecture, unfolded over 3 time steps. Figure 7(a sows te desired output of te network. Eac of te 5 time steps are sown, indicating te position and trajectory of te robot arm from a starting point at (, to its goal at (,. Te tick black lines represent te position of te robot s angled joint arm at eac of te 5 time steps. Te circles represent te robots end point along te trajectory pat. Te spacing of tese points are included to elp represent te velocity cange along tis trajectory. Te BPTT training algoritm was used to train te network on tis trajectory, and figure 7(b sows te network s performance in locating te goal, after 39 training iterations. At tis point te algoritm ad trained te network to produce an output witin an error tolerance of.. On examining te output in figure 7(b a small error is observed in te line of te trajectory as it approaces te goal. Te error plots during te training are sown in figures 7(c and 7(d. Figure 7(c sows te cange in te output error averaged over all te time steps (i.e. eac one of te time series feed-forward networks. Early on in te training cycle, tere was a large output error wen compared to te network s target output. However, tis rapidly reduced during te training process. Figure 7(d presents a more detailed plot of tis error, sowing te output error for eac of te 5 feed-forward networks, at eac point in te 39 stages of te training process. It can clearly be seen tat te most substantial errors occur at tose networks wic represent te latter alf of te time series; wit te network at time step 5 contributing te largest error. Network errors are passed forward in te series of feed-forward networks, accumulating almost exponentially in networks furter along te time sequence. In figures 7(e and 7(f te cange in te variance, σ, is presented during te training process. An important aspect of tis model, over classical models is tat information is contained witin te variance and is affected in te training process. In te experimental models presented ere, r effectively making µ i, tus it is te information contained witin tis variance wic is used in te control of te robot arm. Multiple Goals: Te results presented in figure 8 are from te experiment performed to test te network s performance wen trained to locate multiple goals in its environment. In tese experiments te networks consisted of input nodes wic were fully connected to a series of 5 spike-rate neurons, wic were in turn connected to output units. Te recurrent connections were only between neurons on similar layers. Te time series of te network was tested for 5 equally spaced time steps. Due to te computational intensity of te network, we restricted te number of time steps to 5 so tat we would be able to test te dynamics of te network witin a reasonable time frame. In te first series of experiments te network was trained to locate one goal and ten trained to locate a second goal. Te results in figures 8(a and 8(b sow tat te network forgot ow to locate te first goal once it ad learned ow to locate te second goal. In bot instances te trajectory outputs were identical to tose produced by networks wic ad only been trained on a single goal. In te second series of experiments te network was trained alternatively on te two sets of trajectory data. Te training process was allowed to continue until convergence ad occurred witin a predetermined error tolerance of.5. Figure 8(c sows output trajectory pats for te robot arm after training te network for 6 iterations. Te input goals were te pairs of coordinate points (, and (,. Bot trajectory pats sow a degree of error wen compared to te expected straigt line pats of te desired response. Tere is a small error still present towards te end of te trajectory pat in bot trials, owever te spacing of te arm s position for eac of te time step intervals is more accurate. We saw evidence of tis in te function approximation experiments in section V. In tose experiments te basic sape of te target curve was quickly modelled by te network, but te exact positioning of te output curve required longer training before it exactly matced te target data. In te robot arm experiments, te sape of te curve would represent te angular cange of te robot arm s joints over time. Tis general sape appears to be modelled at te earlier stages of te network s training. Te majority of te training appears to be matcing te detail of te movement rater tan just te trend in te data, i.e. te pat of te trajectory. Figure 8(d is a grapical plot of te overall network error occurring during te process te network was trained to locate te two goals. Te training cycle took 6 iterations. However a large acceptable error of.5 was used to ensure convergence witin a reasonable time period. Figures 8(e and 8(f sow te cange in te variance, σ, during te training process. As in te single goal location task, tese biologically plausible models use te information, contained witin te variance, as part of teir computation. VIII. CONCLUSION We ave expanded on te single neuron model we defined and tested in [6], and developed learning algoritms for use wit spike-rate neuronal networks, in particular possible applications for use in engineering tasks. Te definition of te

13 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER spike-rate model sowed a relationsip between te synaptic stimulus and te neuron s spike-rate activity. We investigated te similarity between te single spikerate neuron model s output and te basis function of an RBF model, wic we supported in our design of an RBFstyle network of spike-rate neurons. Te element of synaptic variance σ, included in te model, proved to be an important component in te computational abilities of te spike-rate neuron. Wen we examined te case of te mean input µ (r we saw tat σ alone ad computational properties sufficient enoug for te model to be applied to a series of non-linear tasks. Indeed it was tis computational component of synaptic input, wic we used in te two sets of experiments included ere. In presenting a generalised learning algoritm similar in approac to te error minimisation used in backpropagation, we ave sown ow it is possible to train a spike-rate network model, and apply it to engineering tasks. Toug, te network structures were less biologically plausible tan te original design of te spike-rate model, te networks proved to be good approximators capable of learning a variety of matematical functions. Tese networks quite quickly learned te trends in sets of training data, toug te dynamics of te model meant tat exact data matcing took a large number of training iterations. We were also able to sow tat tis single-layered network model can be extended into a recurrent time-series structure, and tat it is possible to solve a dynamic control task wit biologically plausible neuron models. By expanding on our general definition of a spike-rate training algoritm introduced in [6], we adopted a similar BPTT style training algoritm applicable for use on time series spiking networks. Toug computationally expensive, te algoritm and network model were able to solve te robot arm control task and locate te target in te experiment. Te unfolded feedforward network structure presented in section VI-B, lacked te real biological plausibility tat one migt ave expected wen dealing te spike-rate neuron model. In tese networks, te recurrent connections used te firing rate outputs from one layer, as part of te synaptic input in te subsequent layer. Toug te firing rate as an important role to play in understanding te input-output relationsip of a neuron, its relevance as part of te synaptic input is meaningless. By constructing te networks in tis way, we were owever, able to sow a possible approac to a dynamic problem using spike-rate neuron. Bot tese experiments present us wit te opportunity to expand on our initial designs of a spike-rate network. By retaining more of te features of te IF neuron, i.e. diffusion approximations for te synaptic input renewal processes, we would want to ave similar approximations for te outputs from te neural model, retaining µ and σ in te output spike train. We would ten ave an opportunity of building networks of IF neurons, were input-outputs from eac layer in te network retain more of te biologically observed spiking features. We could ten develop te spike-rate learning rules presented ere, for use on more biologically plausible networks of spiking neurons. Tus far owever, te model as primarily been tested on spike-rate neurons wit r. Tis was done in order to test te computational effectiveness of te variance term. We propose an initial extension to te work by including te case wen r.5 and. Tis will ave te effect of including te mean and te variance in te computation. Comparing ow tese models perform in engineering tasks, wit results from similar classical AI models, will elp us identify any possible computational advantage te spike-rate model as over its classical counterpart. Could modelling te mean signal, and te noise add to te model s computational power? Furtermore, te idea of employing te variance in computation is not new. It as been extensively discussed in te literature on stocastic resonance. In a typical scenario of stocastic resonance, te output is maximised wen te variance is, usually, very small in value. In our set up ere, owever, te variance and te mean are coupled, wic implies tat we can not arbitrarily reduce te noise, wile keeping te mean (te first order statistics uncanged. Altoug we do use te important properties of second order statistics, our approac is very different from tat of stocastic resonance []. APPENDIX I THE LEARNING RULE FOR THE SINGLE NEURON MODEL We include ere, te learning rule for te single neuron model. Te model consists of a single layer of i,,n spike-rate neurons, eac wit j,,m synaptic inputs λ j. Te learning algoritm, sown ere, seeks to minimise te error between te spike-rate network s output f i (λ j, and te desired target output d i, i.e. E N (f i (λ d i (3 i To minimise suc an error, as in biological systems, we apply an approac similar to backpropagation w ij η ij E were η is te learning rate and ij E E w ij (3 wit w ij being te synaptic weigt connections between units i and j. Here we are seeking to minimise E. So taking equation (3 and differentiating we get E w ij (f i (λ d i f i(λ w ij (33 Using f as defined in equation (7 f i (λ w ij < T i (r > (T ref + < T i (r > (34 w ij

14 JOURNAL OF L A TEX CLASS FILES, VOL., NO., NOVEMBER 3 Now, using te mean ISI < T i (r >, as defined in (8, we get < T i (r > [ ( Vtre τ µ i g w ij τ w ij σ ( ] i Vrest τ µ i g σ i [ g( u ( i u i,j σ i u i σ i,j τ σ i σi g( v ( i v i,j σ i v i σ ] i,j (35 σ i wit u i V tre τ µ i, v i V rest τ µ i, u i,j u i/ w ij, σ i,j σ i/ w ij. Taken togeter, we ave obtained a learning rule, equations (34 and (35, for a single layered spike-rate neuronal network. Te weigt update rule in (35 uses bot te mean and variance of synaptic input. Te important feature is tat any cange in te firing rate is dependent of canges in tese two components of synaptic input. APPENDIX II THE OUTPUT LAYER OF A MULTI-LAYERED SPIKE-RATE NETWORK As a proof concept we present te case for te second stage of te learning rule, discussed in section IV-B., wen y (λ f (λ, i.e. te case wen output units are not linear weigted sums of idden spike rate neurons as defined in equation (3 but are temselves spike-rate neurons. In tis situation, equation (6 now becomes were: E w i E f (λ f (λ T (r σ i T (r w i δ T (r w i (36 δ E f (λ f (λ T (r (37 Equation (37 is te local error gradient for output neuron. Te amount by wic w i must cange is given by te delta rule were a proportion, η, of te rate cange in error E is taken. Tis is defined as te rate of cange of te error wit respect to te synaptic weigt connection w i w i η E w i (38 For te output layer te delta rule is defined as w i ηδ T (r w i (39 So taking < T (r >, were < T (r > is defined in (8 < T (r > [ ( Vtre τ µ g w i τ w i σ ( ] Vrest τ µ g σ [ g( u ( u,i σ u σ,i τ σ g( v σ σ ( v,i σ v σ,i ] σ (4 wit u V tre τ µ, v V rest τ µ, u,i u / w i, σ,i σ / w i, and were µ,i λ i ( r σ,i (λα i w α α i + λ i λik i k i w i k ( + r α (4 σ Here λ i is te input to neuron from anoter neuron i, but were neuron i k i. Te Hidden Layer: For tose spike-rate neurons on te idden layer te local error gradient will be E f i (λ δ i f i (λ T i (r E f i (λ (T ref + T i (r (4 and so using te cain rule, te partial derivative of te error E becomes E E f (λ f i (λ f (λ λ i P (d f (λ f (λ (43 λ i were λ i is te output from spike-rate neuron i. In tis definition of a multi-layered feed-forward spike-rate network, te output from te i,,m spike-rate neurons on te idden layer is represented by te term λ i, and tis is te synaptic input for te,,p spike-rate neurons on te output layer. Differentiating te rate output for unit wit respect to te input it receives from i, as sown in equation (43 f (λ λ i T (r (T ref + T (r λ i τ(t ref + T (r [ g( u (ũ,i σ u σ,i σ σ g( v (ṽ,i σ v σ ],i σ σ (44 wit u V tre τ µ, v V rest τ µ, ṽ,i,ũ,i µ / λ i, σ,i σ / λ i. Denoting tese derivatives wit respect to λ i µ,i w i ( r σ,i α[λα i wi α + λ α i w i i k i λi wik ]( + r α 4σ Using te definition of δ given in equation (37, δ i, for idden neuron i, is given as δ i f i(λ T i (r f i(λ T i (r P P (d f (λ i T (r (T ref + T (r λ i δ T (r λ i (45 and tus te weigt correction for te idden neuron w ij ηδ i T i (r w ij (46