Learning Deterministic Spiking Neuron Feedback Controllers

Transcription

1 Learning Deterministic Spiking Neuron Feedback Controers Tae Seung Kang and Arunava Banerjee Computer & Information Science & ngineering University of Forida Gainesvie, FL 6 mai: {tsk, arunava}@cise.uf.edu Abstract We consider the probem of feedback contro when the controer is constructed soey of deterministic spiking neurons. Athough spiking neurons and networks have been the subject of severa previous studies, anaysis has primariy been restricted to a firing rate mode. In contrast, we construct a deterministic spiking neuron controer whose contro output is one or mutipe sparse spike trains. We mode the probem formay as a hybrid dynamica system comprised of a cosed oop between a pant and a spiking neuron network controer. The construction differs from cassica controers owing to the fact that the contro feedback to the pant is generated by convoving the spike trains with a fixed kerne, resuting in a highy constrained and stereotyped contro signa. We derive a nove synaptic weight update rue via which the spiking neuron controer earns to hod process variabes at desired set points. We demonstrate the efficacy of the rue by appying it to the cassica probem of the cart-poe (inverted penduum). xperiments demonstrate that the proposed controer has a arger region of stabiity as compared to the traditiona PID controer and its trajectories differ from those of the PID controer. I. INTRODUCTION Whie there is considerabe debate in the scientific community regarding the cognitive capacity of various anima species, there is genera agreement that animas are exquisite contro systems. Whether it be the fight of a dragonfy or the waking of a biped (such as a human), engineered systems pae in comparison to the versatiity and robustness dispayed by their anima counterparts. ven more intriguing is the fact that in many instances the particuar ski, ocomotion for instance, is earned. Our goa in this paper is to address this question of earning to contro in the context of bioogicay motivated constraints specificay, the fact that the constituent neurons of anima brains communicate with one another using action potentias (aso known as spikes). In the vast majority of bioogica systems, the contro signa received by the musces are in the form of spike trains generated by motor neurons. The controer itsef is a network of spiking neurons that resides upstream from the motor neurons. The controer receives inputs, which in the case of a feedback controer are process variabes that are to be maintained at fixed or dynamicay varying set points. The process variabe input into the controer is in turn computed esewhere and incorporates the combined output of one or more sensory systems. To bring the probem of earning a spiking neuron network controer into sharp reief, we abstract away a aspects of the system that are of secondary concern and repace them with simpe, fixed, and predefined aternatives. In particuar, we mode the entire process beginning at the spike train output of the controer and cuminating at the contro signa generated (such as the force exerted by the musce) using fixed convoution kernes. The impact of the contro signa on the organism in its environment, we mode using a fixed pant. Finay, we mode the input of the process variabes as postsynaptic potentia inputs into specificay identified neurons of the controer. Our objective is to devise a forma synaptic weight update rue that when appied to the neurons of the controer, causes the controer to earn to perform the contro task. That the above probem differs from those previousy studied in feedback contro, can be discerned from the foowing observation. Traditiona feedback controers such as the proportiona-integra-derivative (PID) controer [] or its variants are designed to sove a contro probem in the continuous domain with few restrictions. The process variabe is a bounded continuous function of time, and so is the contro signa generated by the controer; there is itte ese that constrains these functions. In contrast, the contro output generated by the spiking neuron network controer is an ensembe of spike trains. The spike trains when convoved with the fixed convoution kerne referred to above, eads to a highy restricted and stereotyped signa. In particuar, it is easy to observe that given a kerne, there exists a bound C such that any non-zero contro signa f satisfies f > C informay, the controer has the choice between generating no output or an output arger than a fixed strength. This has immediate impications for the stabiity of the fixed point (determined by the set point) of the combined (the controer and the pant in cosed oop) dynamica system; the process variabe can at best be made to osciate around the set point. The overa goa of the paper is to demonstrate that deterministic spiking neuron based controers can be earned, and not to characterize a pre-given spike based controer. To our knowedge, this is the first deterministic spike based controer that has been demonstrated to earn a cassica task. The controer does operate very much ike a bang bang controer, and it has earned to operate that way. This work is a first

2 Output Force Output Force, Output Force Output Neurons Mode Left Force (+) Right Force (-) (a) Pant t = Υ (Past) Input t = (Present) (b) Controer Mode... (c) Modes Fig.. The hybrid dynamica system that modes the contro probem. For more detais, refer to the text. (a) Pant. The cart-poe pant has a state that can be changed by an externa force. The state is described by the vertica ange and the anguar veocity. The cart can ony move eft or right. For more detais, refer to Section III. (b) Controer. The proposed spiking neuron controer is a feedforward neuron network that takes the pant state as input and produces an output force to contro the pant. The synaptic update rues are set such that the weights on the synapses receiving the continuous process variabe inputs change ony when there are spikes generated by the output neurons of the controer. This is indicated by the vertica dotted ines. Perturbing the synaptic weights perturb the output spikes of the network in time (dotted arrows), which when convoved with the kerne creates a perturbation in the contro signa. The contro signa is optimized based on an error function that embodies the deviation of the process signa from its set point. (c) Modes. We present two modes, Mode and Mode, based on the number of output neurons of the controer and their corresponding force magnitude. A circe indicates an output neuron of the proposed controer shown in (b). The vertica arrow stemming from a circe denotes a force produced by the neuron with its ength representing the force magnitude. The onger the ength, the arger the force magnitude. Mode has two output neurons to generate a force in the eft direction (eft force) and a force in the right direction (right force) with both forces having the same magnitude. Mode has 4 or more output neurons with haf contributing to the eft force and haf right force, with forces having different magnitudes. In both modes, the number of output neurons and the force magnitudes are symmetric for the eft and right. The fina output force that woud be appied to the pant is shown at the top. It is generated by summing up the eft and right forces. Note that the eft force has a positive sign and the right force has a negative sign.... step toward contro of more compex dynamica systems such as winged fight or bipeda waking. Our objective is described schematicay in Figure. We consider a hybrid dynamica system constructed out of a cosed oop between a pant (we consider the cassica probem of the inverted penduum in this paper as described in Section III, but this coud be repaced with any we defined pant), and a network of spiking neurons that modes the controer. In the figure, the back vertica bars indicate the weights on spikes. That is, we virtuay assign weights to spikes (denoted by the height of the bars) instead of the corresponding synapse. The conceptua underpinnings of this are described in section IV. The vertica doube-headed arrows next to the back bars denote the perturbation of the corresponding weights. The four dashed arrows beginning at the second vertica ines from t = Υ (Past) denote the impact of the weight perturbations on the subsequent future spike times. We present two modes, Mode and Mode, based on the number of output neurons of the controer and their corresponding force magnitudes. Mode is defined as a network with two output neurons (for eft and right directions) with the same force magnitude, and Mode is defined as a network with four or more output neurons (two or more eft and two or more right) with distinct force magnitudes. The goa is to incrementay update the synaptic weights on the neurons of the network such that the network s output spike trains when convoved with the force kerne causes the process variabes of the pant to deviate as itte as possibe from predefined set points. The process variabes are in turn input into the spiking neuron network controer as postsynaptic potentias. Our approach is based on the observation that if (a) synaptic weights are ony updated at the times that the network generates spikes, and (b) one anayzes the past times at which the network generated spikes, one can then perform a perturbation anaysis that woud recommend a superior set of weights in the past. Since we can not reach into the past to change synaptic weights, the current synaptic weights of the network are updated to refect these improvements. The synaptic update rue is used to train the network from randomy generated initia conditions of the pant in an onine manner unti faiure. At faiure, the pant is reinitiaized to a different initia condition and the earning process continues. The remainder of the paper is structured as foows. Section describes the neuron mode. Section briefy describes the pant used in this paper, as we as the process variabes and their corresponding set points. Section 4 comprises the core of our contribution where the synaptic weight update rue is derived. Section 5 describes experimenta resuts from severa variations of the controer, and Sections 6 and 7 present reated work and concusions. II. NURON MODL We use a minor variation of the Spike Response Mode (SRM) [] for the neurons in our controer. The neuron receives continuous time process variabe inputs at its synapses. Athough our anaysis seamessy generaizes to postsynaptic potentias generated from afferent (incoming) spikes at synapses, we do not consider that here. The membrane potentia at the soma of the neuron is the synapticay weighted sum of postsynaptic potentias (PSPs) generated by the current vaues of the process variabes and afterhyperpoarizing potentias (AHPs) generated by the efferent (outgoing) spikes that have departed the soma of the neuron. The neuron generates a spike when the membrane potentia crosses the threshod Θ

3 from beow. Formay, the membrane potentia of a neuron at the current time is given by P (t) = i Γ w i x i (t) + η(t O k ). () where Γ is the set of synapses, w i is the weight of synapse i, x i (t) is the continuous process variabe input signa at synapse i, and t = is the current time (with positive t indicating past). Simiary, η is the prototypica after-hyperpoarizing potentia (AHP) eicited by an efferent spike of the neuron, t O k is the eapsed time since the departure of the kth most recent efferent spike, and F is the set of past efferent spikes of the neuron. We assume in addition that a efferent spikes that were generated earier than t = Υ in the past have no effect on the present membrane potentia of the neuron (See Figure ). The functiona form of the AHP of a spike that we have used (and this can be modified without affecting the anaysis) is η(t) = Re t/γ for < t Υ and otherwise () where R denotes the instantaneous fa in potentia after a spike and γ contros its rate of recovery. III. PLANT The pant we consider in this paper is the cassica contro probem of the cart-poe (aso known as the inverted penduum). The cart-poe comprises of an inverted rigid penduum, with the mass at the top. The penduum is fucrumed at its base to the cart which rests on a frictioness surface. Force can be appied to the cart to move it aong the horizonta axis. The contro probem is to appy forces to the cart to maintain the upright position of the penduum. The process variabes that we have considered in this paper are:, the anguar deviation of the penduum from the upright position, and, the anguar veocity of the penduum. The set points for the process variabes are =, =. The detais of the system dynamics can be found in []. A quantities of interest as presented in the next section, we have derived through numerica computations. IV. FDBACK CONTROL USING SPIKING NURONS As described above, the desired state of the pant is to maintain a zero vertica ange and a zero anguar veocity of the inverted penduum. These process variabes are input into different synapses of the controer neurons in the foowing form: the anges, and, and the anguar veocities, and. As described in Section 5, we have experimented both with the case where is a process variabe to be controed at the set point, and the case where it is not. The contro signa output of a neuron is generated by convoving the output spike trains of the neurons with a fixed force kerne κ(t) as defined in the next section. ven number of neurons are used to generate forces, the first set generating forces to the right and the second set to the eft. We anayze the genera case of mutipe neurons with different κ(t) force kernes coming together to constitute the fina contro signa. A. The rror Function The proposed spiking neuron based controer depicted in Fig can be formay modeed as foows. Consider a pant with process variabes represented by vector x (), x (),..., x D (), where D is the number of process variabes to be controed. The desired state of the pant (i.e., the set point of the process variabes) is represented by x (), x (),..., x D (). The error can then be defined by D i= (x i() x i (). The synaptic update rue that we derive next is based on minimizing this objective using gradient descent, which in the case of the fu set of process variabes and assuming set points of, reduces to: = D (x i ()). () i= A traditiona controer receives continuous time process signas from the pant and generates a continuous time contro output. The proposed controer, however, generates spike trains, one for each neuron, instead of a continuous output. The spike train output of each neuron j is then convoved with the kerne κ(t) = te t/τ f (4) to generate a force: F j = µ j i F j κ( j t O i ) (5) where τ f is the time constant, µ j is the magnitude assigned to neuron j, j t O i is the time eapsed since the generation of the i th most recent efferent spike of the output neuron j, and F j is the set of past spikes of output neuron j. The fina force F appied to the pant is F = j ±F j (6) where ± represents the direction of F j, + for neurons that push to the right and for those that push to the eft. B. Gradients of the rror function Our overa objective is to compute the gradient of the error with respect to the synaptic weights on the controer neurons. We do this in stages. We first compute the gradient with respect to the output spike times of the controer neurons. Appying chain rue, we then have ( j t O ) = ( j t O ) (7) where j t O is the time eapsed since the departure of the th most recent efferent (outgoing) spike of neuron j, and = i x i x i = i x i x i ( j t O ) = j κ( j t O ) j κ( j t O ) ( j t O ) = ±µ κ j t In q (8), xi is drawn as a numerica derivative from the pant: xi j xi F j. j t O (8) (9)

4 C. Perturbation anaysis Our goa now is to determine how perturbations in synaptic weights of the controer neurons transate to perturbations in the times of their output spikes. We achieve this by first assuming that synaptic weights are ony perturbed at the times of the output spikes (see Figure ). Consider the state of a neuron at the time of the generation of output spike t O. The membrane potentia of the neuron before perturbations of the weights on the input signas is given by Θ = i Γ w i, x i (t O ) + η(t O k t O ). () where Γ is the set of synapses of the neuron and w i, is the weight of synapse i immediatey prior to output spike. Note that we have repaced Θ with Θ to account for those output spikes that at the time of the generation of t O were ess than Υ od, but are now past that bound. If the synaptic weights were perturbed, this woud cause the output spike t O to be correspondingy perturbed according to Θ = i Γ(w i, + w i, )x i (t O + t O ) () + η(t O k t O + t O k t O ). () Using a first order Tayor approximation, we get Θ = ( i, + w i, ) x i (t i Γ(w O ) + x ) i t O t t O () + ( η(t O k t O ) + η ) ( t O k t O ) k t O ). (4) Combining q () and (), dropping higher order terms and rearranging, we get w i, x i (t O ) + η t O t O i Γ k t O ) k = x i w i, + η (5) t t O i Γ k t O ) We can now derive the fina set of quantities of interest from q (5). If we perturb the weight w i,, there wi ony be a direct effect of the perturbation since w i, does not impact spikes prior to t O. Therefore, we have t O w i, = i Γ x i (t O ) x i w i, + η. (6) t t O k t O ) If instead, we perturb w i,p where p > (so that t O p > t O ), there wi ony be an indirect effect of the perturbation through previousy generated spikes. Therefore, for p > we have the recursion η t O k t O k t O ) w i,p = w i,p x i w i, + η. (7) t t O i Γ k t O ) D. Learning rues Learning is accompished via gradient descent. The earning rue is a type of Spike Timing-Dependent Pasticity (STDP) [4]; the weight updates depend on spike times. The reason that the weights shoud be updated ony when there are spikes is as foows. If there are no spikes generated, the poe is in a safe kinematic range and thus no contro signa is necessary. This, in turn, indicates no need to update the weights. The weight updates are not independent. They get reated to each other due to the common error functiona on which gradient descent is performed. Appying chain rue, we get = w i,p t O F t O w i,p. (8) This is computed using q 7, 6, and 7. The weight modification rue for synapse i at t O p is defined as w i,p w i,p α w i,p (9) where α is the earning rate. Ceary we can not reach into the past to make these changes. We therefore institute a summed deayed update to the synapse at the current time. w i w i p F α w i,p () The weight update is performed immediatey after the generation of a spike by any of the output neurons. This way we are guaranteed that the weights on the synapses remain constant between any two successive spikes (on any neurons). A. Setup V. XPRIMNTS To demonstrate the efficacy of the proposed controer, we performed mutipe simuations. For these simuations, we defined a successfu earning event of the controer as having baanced the poe without faiure for one hour of simuation time. A faiure was defined as an event where the process variabes of the poe was not within a certain predefined range. For exampe, given the ms time step, the number of steps for a successfu run was 6 ( hour). The predefined range was [-.94,.94] for and [-.,.] for. Training of the controer was continued unti success. During the training phase, the controer earned by changing the synaptic weights. Once it had successfuy earned the task, we fixed the weights and tested the controer with random initia pant states to evauate its robustness. If the poe fe before training succeeded, we restarted training with random

5 Force(x.) 6 4 Force(x) 4 5 (a) Start condition before training (b) Start condition after training (c) Stabe condition after training Fig.. Snapshots of the pant state (top) and spike trains of the controer (bottom) for Mode with two output neurons before and after training. In a top figures, the bue dotted ine is the vertica ange, the green dash-dot is the anguar veocity, the cyan dashed ine is the error function, and the red soid ine is the force appied to the pant. (a) Before training, the synaptic weights are randomy chosen and fixed (no updates). As expected, the poe fas down quicky as the controer did not earn how to stabiize the pant. The spike train exhibits no particuar pattern. The fina force appied to the pant is scaed down fod (.) of the actua vaues for improved visuaization. (b) After training, the controer stabiizes the pant within a short amount of time ( ms). The spike trains are a bit dense but exhibit some patterns. The fina force appied to the pant is scaed down fod (.) of the actua vaues for improved visuaization. (c) In the stabe state, the poe osciates around the set point (, ) and the error function is aso at around. The trained controer behaves ike a bang-bang controer. This resuts from the patterns of the output spike trains. After one neuron fires three times, the other neuron aso fires three times, then the first neuron fires again, and so on. Compared to the start condition, the spike trains are sparse. The fina force appied to the pant is scaed down fod ( ) of the actua vaues for improved visuaization. It can be seen that the patterns of the spike trains are different in the start condition from those in the stabe state. In the start condition, one output neuron generates spikes continuousy whie the other does intermittenty. This is because the controer attempts to repeatedy push the pant in one direction. In the stabe state, however, both neurons produce spikes aternatey Mode Mode PID Faiure (a) Trajectories PID..4 Mode PID Mode (b) Zoomed in trajectories (c) Coverage of Mode (d) Coverage of PID Fig.. Trajectories and coverages of Mode with two output neurons. (a) Trajectories of the pant state (, ) for Mode after training and the PID controer over time with different initia settings. The trajectories start at two points (-.5,.4) and (-.5,.5). They are chosen to compare the performance of Mode against the PID controer. The bue soid ine shows the trajectory of the pant state for Mode with the initia pant state (-.5,.4). The green dotted ine shows the trajectory of the pant state for the PID controer with the same initia state. Whie the pant for Mode eventuay settes down, its trajectory is different from that of the PID controer. This demonstrates the fact that the proposed controer behaves differenty suggesting a nove contro mechanism. The red dashed ine represents the trajectory of the pant state for Mode with the initia state (-.5,.5). The cyan dash-dot is the trajectory for the PID controer with the same initia state. Whie Mode succeeds, the PID controer fais. (b) Trajectories in (a) zoomed in around the set point (, ). The pant for Mode osciates between (, -) and (, ) in the stabe condition. (c) Coverage of initia states (, ) with Mode controer. A green circe indicates a success whie a red x mark indicates a faiure for the corresponding initia state. (d) Coverage of PID controer. The number of dots covered (green circe) is 6 for Mode controer and for the PID controer. The proposed controer covers a arger area than the PID controer. initia weights since the same weights woud yied the same resuts of the faied run. The pant was configured as: haf-poe ength =.5 (m), poe mass m =. (kg), cart mass M = (kg), and gravity g = 9.8 (m/s ). The configuration for the controer was: time step = ms, threshod =, τ f = (ms), R =, γ =., and α =. The unit of R is the same as that of the membrane potentia. The magnitude of R was set arge to prevent a spike from being generated within 4-5 msec after a spike was generated by bringing down the membrane potentia dramaticay. We measured the firing rates of the output neurons in Hz (number of spikes per second). Specificay, the firing rate of a neuron is the tota number of spikes generated by the neuron divided by the tota running time of the simuation. In a the experiments, the same PID controer was used and its parameters were K P =, K I =, and K D =. B. Resuts We start by describing a mode that successfuy earned the contro response. Not surprisingy, when as a state variabe to contro was removed, the controer faied to earn regardess of whether the controer was based on Mode or Mode. To eaborate, athough the controer was abe to hod the poe upright for a short period in the training stage, it faied to hod

6 Force(x) Force(x) (a) Start condition for 4 output neurons (b) Stabe condition for 4 output neurons Force(x) Force(x) (c) Start condition for 6 output neurons (d) Stabe condition for 6 output neurons Force(x) Force(x) (e) Start condition for 8 output neurons (f) Stabe condition for 8 output neurons Fig. 4. Snapshots of the pant state (top) and the output spike trains of the controer (bottom) in the start and stabe condition for Mode with 4, 6, and 8 output neurons after training. In the top figures, the bue dotted ine and the green dash-dot represent the vertica ange and anguar veocity of the poe in radian ( ), respectivey. The red soid ine is the force appied to the cart and the cyan dashed ine is the error function. Note that the force appied to the pant is scaed down fod ( ) of the actua vaues for improved visuaization. For each spike trains snapshot, the force magnitude assigned to each output neuron is symmetric: big to sma and sma to big. (a), (b) 4 output neurons. For the spike trains snapshot, the force magnitude assigned to each output neuron is, 5, 5, and from top to bottom. (c), (d) 6 output neurons. The force magnitude assigned to each output neuron is,,,,, and from top to bottom. (e), (f) 8 output neurons. The force magnitude assigned to each output neuron is 5,, 5,,, 5,, and 5 from top to bottom.

7 (a) 4 neurons (b) 6 neurons (c) 8 neurons (d) PID Fig. 5. Stabiity coverages of the initia states (, ) of Mode controers. The PID coverage is reproduced from Fig. d for convenient visua comparison. A green circe indicates a success whie a red x indicates a faiure for the corresponding initia state. The 6 neuron controer covers a arger region than the 4 neuron controer, and the 8 covers a arger region than the 6. This impies that controers with more neurons are more robust. When compared to the PID controer, the proposed controer covers more in genera. speciay 6 and 8 neurons cover the whoe region of that of the PID. This indicates the proposed controer has better contro capabiity than the traditiona PID controer. the poe upright for hour. Surprisingy, the controer with either mode worked once we added. In what foows, we discuss the experimenta resuts for each mode. ) Mode : Fig. shows snapshots of the pant state (top) and the spike trains of the output neurons (bottom) of the proposed controer with output neurons before and after training. The controer received 4 continuous inputs (,,, ) from the pant and produced output spike trains at the two output neurons. The output spike trains correspond to the eft direction force and right direction force, respectivey. In the top panes, the bue dotted ine and the green dash-dot represent the time evoution of the vertica ange and anguar veocity of the poe in radian ( ) and rad/sec, respectivey. The red soid ine is the force appied to the cart and the cyan dashed ine is the error. The force magnitude assigned to each output neuron was. The average firing rates of the output neurons were.94hz and 4Hz. Fig. a shows a snapshot before training where the synaptic weights were randomy chosen and not updated. As shown in the top pane, the poe fe down shorty after the start of the simuation as the controer was untuned. This is further obvious from the bottom pane where the spike trains were randomy generated regardess of the state of the pant. After training, the poe successfuy stood upright for hour. The trained controer was tested on random initia conditions. Fig. b shows the start condition in the testing phase to demonstrate how the proposed controer behaved in controing the pant initiay. Fig. c shows the stabe state after a whie. From the figures, it can be seen that the patterns of the spike trains are different in the start condition versus the stabe state. In the start condition, one output neuron generates spikes continuousy whie the other does intermittenty. This is because the controer attempts to repeatedy push the pant in one direction. In the stabe state, however, both neurons produce spikes aternatey. Fig. shows the trajectories of the pant state (, ) over time with two different initia settings. Further testing was performed over severa initia states to determine the robustness and the coverage over initia states for the proposed controer with Mode as compared to the PID controer. In Fig. a, we can observe that the trajectory of the proposed controer is different from that of the PID controer. This impies that our controer behaves in a different manner in order to contro the pant suggesting a nove contro mechanism. Fig. b shows the trajectories in Fig. a zoomed in around the set point (, ) for better visuaization. Fig. c and Fig. d show the coverages of stabiity for Mode and the PID controer, respectivey. Athough they do not excusivey incude each other, Mode s coverage is arger than the PID. For exampe, the initia state (-.,.5) covered by Mode is not covered by the PID controer. ) Mode : We performed the same earning experiments above for 4, 6, and 8 output neurons with successivey arger force kernes. In this case, the force magnitude assigned to each output neuron was symmetric: pairs of equa magnitude for the eft and right force. For 4 output neurons, the force magnitude assigned to each output neuron was, 5, 5, and. For 6 output neurons, it was,,,,, and. For 8 output neurons, it was 5,, 5,,, 5,, and 5. Fig. 4 shows snapshots of the pant and the controer. It shows the start and stabe conditions after training. As shown in the figure, Mode controers achieved the objective of stabiizing the pant. As in the case of Mode, the spike trains exhibit reguar patterns; neurons fired aternatey periodicay. In genera, the spike train in the stabe state was sparser than that in the start condition. It can be observed that there are unnecessary spikes in the stabe state. To eaborate, since we have neurons generating arge as we as sma forces, we need ony the sma force neurons to fire in the stabe state. This can be mitigated by adding communication between output neurons so that they are aware of one another s spike trains. Fig. 5 shows the coverages of initia conditions that are controed. As shown in the figures, the stabiity coverage increases with the number of neurons in the controer. 4 has a arger region than, 6 has a arger region than 4, etc. Fig. 5d is reproduced from Fig. d to ease visua comparison with Mode s coverage. It is cear that Mode covers a much wider area than the PID. It shoud be noted that the coverage of 6 or 8 neurons subsumes the entire

8 PID area. This suggests that the proposed controer is more robust than the traditiona PID controer. VI. RLATD WORK Neura networks have been the too of choice for soving various probems since Rumehart et a. introduced the gradient descent based error backpropagation agorithm [5]. Simiar agorithms have aso been used for spiking neura networks [6], [7]. [8] introduced SpikeProp to sove the XOR probem by appying the error backpropagation agorithm to spiking neuron networks based on tempora coding or spike timing based coding [9]. Their supervised earning rue generates a desired pattern of spikes with the constraint that each output neuron be aowed to fire ony once in a prescribed time window. [] extended this rue to mutipe output spikes, abeit with the first output spike used in the error function. [] presented a spiking neura network based controer to reguate a robot s arms with 4-degrees of freedom. The neuron mode they used is the Izhikevich mode [] and the earning agorithm is Spike Timing-Dependent Pasticity (STDP) [4]. Their controer shows high firing rates due to rate-based coding. Recenty, Gerstner et a. [], [4] studied contro probems for motor systems using reinforcement earning. They used the actor-critic mode [], [5] to train the networks. [6] studied the perturbation anaysis [7] to revea how perturbations in the weights and times of the input spikes of a neuron transate to perturbations in the timing of its output spikes. Athough our proposed controer can be described to be most simiar to [6] and [8], unike these, ours does not require the prescription of the desired spike train and the input to the network is continuous. VII. CONCLUSION We have proposed a spiking neuron network controer and have appied it to the cassica cart-poe contro probem to demonstrate its efficacy. The derivation presented is genera and can be appied to any feedforward network. The primary advantage of our controer is that it has a arger region of stabiity as compared to the traditiona PID controer. Furthermore, our controer behaves in a manner different from the traditiona PID controer. As demonstrated in our experiments, the proposed controer succeeds in severa initia conditions where the PID controer fais. The proposed controer produced different trajectories than that of the PID controer. We presented two controer modes with different output neuron settings: two output neurons with the same force magnitude (Mode ) and 4 or more neurons with different force magnitude kernes (Mode ). From the experiments, we observe that more neurons with diverse force magnitudes can earn arger ranges and are thus more fexibe and robust. In particuar, the 6 or 8 output neuron controer performs substantiay better than the PID controer. In future work, we pan to add a kerne for fitering the inputs and sha consider other contro costs. The former can readiy be added to the current controer keeping the derivation the same. The atter requires using a more genera error function that incuded the number of contro spikes and other output statistics. One of the issues in Mode is that some spikes are produced redundanty. We can mitigate this by adding recurrent inhibitory connections among the output neurons. This wi ead to sparse spike trains which is more natura in bioogica systems as they signify higher energy efficiency. Finay, we pan to extend our controer to appy to the ocomotion of a fish with or more degrees of freedom which is a more reaistic and compex contro probem than the cart-poe. ACKNOWLDGMNT The authors woud ike to thank the Air Force Office of Scientific Research (Grant FA ) for their generous support of this research. RFRNCS [] N. Minorsky, Directiona stabiity of automaticay steered bodies, Journa of the American Society for Nava ngineers, vo. 4, no., pp. 8 9, 9. [] W. Gerstner and W. M. Kister, Spiking neuron modes: Singe neurons, popuations, pasticity. Cambridge university press,. [] C. W. Anderson, Strategy earning with mutiayer connectionist representations, in Proceedings of the Fourth Internationa Workshop on Machine Learning, 987, pp. 4. [4] S. Song, K. D. Mier, and L. F. Abbott, Competitive hebbian earning through spike-timing-dependent synaptic pasticity, Nature neuroscience, vo., no. 9, pp ,. [5] D.. Rumehart, G.. Hinton, and R. J. Wiiams, Learning representations by back-propagating errors, Cognitive modeing, vo. 5, p., 988. [6] F. Rieke, Spikes: exporing the neura code. MIT press, 999. [7] P. Dayan and L. F. Abbott, Theoretica Neuroscience: Computationa and Mathematica Modeing of Neura Systems. The MIT Press, 5. [8] S. M. Bohte, J. N. Kok, and H. La Poutre, rror-backpropagation in temporay encoded networks of spiking neurons, Neurocomputing, vo. 48, no., pp. 7 7,. [9] R. V. Forian, The chronotron: a neuron that earns to fire temporay precise spike patterns, PoS one, vo. 7, no. 8, p. e4,. [] O. Booij and H. tat Nguyen, A gradient descent rue for spiking neurons emitting mutipe spikes, INFORMATION PROCSSING LTTRS, vo. 95, no. 6, pp , 5. [] A. Bouganis and M. Shanahan, Training a spiking neura network to contro a 4-dof robotic arm based on spike timing-dependent pasticity, in Neura Networks (IJCNN), The Internationa Joint Conference on. I,, pp. 8. []. M. Izhikevich et a., Simpe mode of spiking neurons, I Transactions on neura networks, vo. 4, no. 6, pp ,. [] N. Frémaux, H. Sprekeer, and W. Gerstner, Reinforcement earning using a continuous time actor-critic framework with spiking neurons, PLoS computationa bioogy, vo. 9, no. 4, p. e4,. [4] G. Hennequin, T. P. Voges, and W. Gerstner, Optima contro of transient dynamics in baanced networks supports generation of compex movements, Neuron, vo. 8, no. 6, pp , 4. [5] A. G. Barto, R. S. Sutton, and C. W. Anderson, Neuronike adaptive eements that can sove difficut earning contro probems, Systems, Man and Cybernetics, I Transactions on, vo. Sep, no. 5, pp , 98. [6] A. Banerjee, Learning precise spike train-to-spike train transformations in mutiayer feedforward neurona networks, Neura Comput., vo. 8, no. 5, pp , May 6. [Onine]. Avaiabe: a 89 [7], On the phase-space dynamics of systems of spiking neurons. i: Mode and experiments, Neura Computation, vo., no., pp. 6 9,.