Delay Constrained Point to Multi-Point Scheduling in Wireless Fading Channels

Transcription

1 Delay Constrained Point to Multi-Point Scheduling in Wireless Fading Channels Nitin Salodkar School of Information Technology, IIT Bombay, Powai, Mumbai, India nitins@it.iitb.ac.in Abhay Karandikar Department of Electrical Engineering, IIT Bombay, Powai, Mumbai, India karandi@ee.iitb.ac.in V. S. Borkar School of Technology and Computer Science, Tata Institute of Fundamental Research, Homi Bhabha Road, Mumbai, India borkar@tifr.res.in Abstract In this paper we consider the problem of designing a centralized scheduler on the downlink for satisfying average delay constraints of users having time varying wireless channels. We assume that the information regarding the channel and the arrival processes of the users is not known to the scheduler. We formulate the problem as a constrained optimization problem where the objective is to maximize the system throughput under user specified delay constraints. We suggest an online throughput optimal algorithm that satisfies the user delay constraints. The algorithm is based on the stochastic gradient method, specifically, the Simultaneous Perturbation Stochastic Approximation (SPSA) algorithm. It attempts to arrive at the optimal scheduling probabilities of the users. Our simulation results show that the algorithm is able to satisfy the user delay requirements even for users perceiving extremely poor channel quality at the expense of very little loss in system throughput. The system throughput provided by our scheduler is much better than a delay satisfying heuristic scheduler described in the paper. I. INTRODUCTION Next generation wireless networks promise multi-megabit Internet access to the users. Traditionally voice traffic has been the dominant traffic in the wireless networks. But with the advent of data centric applications, it is expected that data traffic will dominate in the future wireless networks. For the users downloading data, the average delay suffered by the packets is an important Quality of Service (QoS) metric. Mobile wireless users perceive time varying channel quality. The channel quality across users might be quite diverse based on their locations. A centralized scheduler in the point to multi-point scenario, e.g., a base station on the downlink, can exploit this diversity by scheduling a user perceiving better channel quality. Data can be transmitted at a higher rate to such user while maintaining an acceptable Bit Error Ratio (BER). This results in higher system throughput []. Scheduling schemes that exploit such opportunities provided by multiuser diversity are called opportunistic scheduling schemes [2]. A. Related Work A pure opportunistic scheduler that always schedules the user perceiving the best channel can be unfair [2]. In [2], the authors consider the problem of opportunistic scheduling under different types of fairness requirements. Opportunistic scheduling under fairness constraints is also variously considered in [3], [4], [5]. In [6], the authors consider the problem of opportunistic scheduling with energy minimization on the uplink with rate constraints. The problem of power minimization under delay constraints for the point to point wireless link has been studied in [7]. In [8], the authors consider the problem of average delay minimization under average power constraints for a point to point wireless link. In [9], the authors consider the rate and power allocation policy for the Code Division Multiple Access (CDMA) forward channel for minimizing the delay. Stability results for the suggested Modified Largest Weighted Delay First (M-LWDF) scheme are proved. In [0], the authors prove the delay optimality of the Longest Queue Highest Possible Rate (LQHPR) policy under Poisson arrivals, exponentially distributed packet lengths and symmetric fading. A comprehensive survey of literature in the area of channel aware scheduling schemes under various constraints can be found in []. B. Our Contribution Most work on delay constrained scheduling with power minimization has been done for the point to point scenario. While the opportunistic scheduling schemes under fairness constraints have been considered for the point to multi-point scenario, the problem of delay constrainted scheduling in this scenario has not been addressed. In [9] and [0], the authors consider the problem of delay minimization for the point to multi-point scenario, while we consider the dual problem of throughput maximization under user specified delay constraints. The primary contribution of the paper is to propose a throughput optimal, delay constraint satisfying scheduler for the point to multi-point scenario. Our scheduler schedules the users with certain probabilities. We use the stochastic gradient method to determine the optimal user scheduling probabilities. Note that, it is not always possible to know the statistical characteristics of the channel or the arrival processes of the users. Our algorithm learns about the channel conditions and the arrival processes of the users and embeds this knowledge in the optimal scheduling probabilities. We study the performance of the algorithm under various simulation scenarios and compare the delay and the throughput characteristics of the algorithm with a delay satisfying heuristic scheduler and a pure opportunistic scheduler.

2 The rest of the paper is organized as follows. In Section II, we introduce the system model and formulate the problem as a constrained optimization problem. We propose the stochastic gradient based solution technique in Section III. Simulation setup and results are presented in Section IV. We conclude in Section V. II. SYSTEM MODEL AND PROBLEM FORMULATION A. System Model We consider a multiuser TDM system where a base station schedules N users on the downlink. Time is divided into slots of equal duration. Only one user can be scheduled in a time slot. The base station maintains a queue for each of the users. Packets arrive into the queue at the beginning of a time slot. The channel gain at the receiver is time varying. We assume a block fading model described in [2], where the channel gain is assumed to remain constant in a block, i.e., over a batch of symbols and is assumed to change in an Independently and Identically Distributed (IID) manner across blocks. Different users have independent channel gains. Let s(n) and r i (n) denote the signal transmitted by the base station and the signal received by a user i in time slot n. r i (n) can be expressed as, r i (n) = h i (n)s(n) + η i (n), () where the multiplicative factor h i (n) models the time varying channel gain. η i (n) denotes the Additive White Gaussian Noise (AWGN) at the receiver with Power Spectral Density (PSD) N0 2. For a Rayleigh channel, h i(n) is modeled as a circular symmetric Gaussian random variable with mean 0 and variance σ 2. Hence h i (n) 2 is an exponentially distributed random variable with density, { x } σ 2 exp σ 2, x > 0. (2) In this case, the channel capacity C i (n) for a user i in time slot n in packets/timeslot is expressed as [3], ( C i (n) = B log 2 + h i(n) 2 P N 0 B ), (3) where B is the bandwidth in Hz and P is the fixed maximum downlink transmission power. We assume a capacity achieving code, hence, the rate y i (n) at which the base station can transmit data to a user i in time slot n while maintaining an acceptable BER, is equal to the channel capacity C i (n). We assume that perfect channel state information (CSI) is available at the transmitter. The users specify their QoS requirements in terms of average packet delay constraints. Let x i (n) denote the instantaneous queue length of a user i in time slot n. Let a i (n) be the number of packets arriving into the user i queue in time slot n. The queue evolution equation for user i can be written as x i (n + ) = x i (n) + a i (n) I i (n)y i (n) (4) where I i (n) is an indicator function that is set to if user i is scheduled in time slot n, otherwise it is set to 0. y i (n) can be interpreted as the number of packets transmitted to a user i in one time slot n. B. Formulation as a Constrained Optimization Problem The objective of the system is to maximize the system throughput while also satisfying the average packet delay requirements of the users. If the system adopts a pure opportunistic policy, by always scheduling a user with the best channel gain, the throughput will be maximized, but the delay constraints may not be satisfied. Hence, the system has to, at times, schedule a user who does not have the best channel condition in order to meet its delay requirements. We consider a stationary scheme which attempts to determine the optimal probability of scheduling a user in a time slot. This probability is dependent on the users channel gain and the delay experienced. The average queue length achieved in the long run for a user i can be expressed as, x i = limsup M M M x i (n). (5) n= Let x = [x,...,x N ] T denote the vector of achieved queue lengths. Let a = [a,...,a N ] T denote the vector of the average arrival rates, a i being the average arrival rate for user i. Let θ i (n) be the probability with which a user is scheduled in a time slot n and θ(n) = [θ (n),..., θ N (n)] T be the corresponding scheduling probability vector. The throughput achieved by the system in a time slot n can be expressed as, T(n) = N θ i (n)y i (n). (6) i= The average throughput achieved by the system over a long period of time can be expressed as, T = liminf M M M T(n) (7) n= Let D = [D,..., D N ] T denote the vector of user specified packet delay constraints, D i being the delay constraint of user i. The average delay constraints can be converted into average queue length constraints using Little s law [4], q i = a i D i, (8) where q i is the average queue length constraint for a user i, q = [q,...,q N ] T being the vector of queue length constraints. The scheduling problem can be expressed as a constrained optimization problem, subject to, Maximize T, (9) x i q i for i =,..., N. (0) III. THE ONLINE THROUGHPUT OPTIMAL DELAY CONSTRAINT SATISFYING (O-TODCS) ALGORITHM In this section, we derive the online throughput optimal delay constraint satisfying (O-TODCS) algorithm. It is based on the stochastic gradient technique. To solve the constrained problem, we make use of the Lagrangian approach.

3 A. The Lagrangian Approach We use the Lagrangian approach [5] for converting the constrained problem in (9) and (0) into an unconstrained problem. Let λ = [λ,..., λ N ] T be the vector of Lagrange Multipliers (LMs). The Lagrangian can be expressed as, L(θ, λ) = T λ T (x q) () The optimal problem in (9) and (0), now, is to determine the saddle point of the Lagrangian, i.e., to determine optimal θ and λ such that the following saddle point optimality conditions are satisfied, L(θ, λ ) L(θ, λ ) L(θ, λ). (2) At the saddle point, the vectors θ and λ satisfy the following conditions, i.e., θ L(θ, λ ) = 0, θ=θ (3) and the complementary slackness condition, λ (x q) = 0. (4) Note that the Lagrangian in () is a time averaged function that cannot be determined apriori in an online implementation setup, hence, we cannot determine its partial gradient w.r.t θ. If the optimal LM vector λ is known, we can resort to iterative methods that improve their estimate of only the optimal θ. Since the optimal LM vector λ is also not known, we resort to primal-dual methods that determine both θ and λ iteratively [5]. In order to ensure convergence of the θ and λ iterates to optimal θ and λ, the iterations proceed at different timescales, i.e., the θ and λ values are updated at different rates [6]. The coupled iterative equations can be expressed as, and θ(n + ) = θ(n) + b(n) L(θ(n), λ(n)), (5) θ λ(n + ) = λ(n) + c(n)(x(n) q), (6) where {b(n)} and {c(n)} are positive gain sequences and λ(n) is the vector of LMs in time slot n. In a time slot n, define the immediate cost function g(θ(n), λ(n)), g(θ(n), λ(n)) def = θ(n) T y(n) λ(n) T (x(n) q). 2 (7) It should be noted that the Lagrangian in () can be expressed in terms of the immediate cost function as, L(θ, λ) = liminf M M M g(θ(n), λ(n)). n= 2 g is a function of the queue length x(n) and the achievable rate y(n) also, but for notational simplicity we do not show them. B. The Stochastic Gradient Technique We don t know the partial gradient w.r.t θ of the Lagrangian expressed in (5) and (6). Hence, we use the Simultaneous Perturbation Stochastic Approximation (SPSA) method [7] in order to approximate the gradient of the function L(θ, λ). This gradient approximation is formed by perturbing the elements of the vectors θ(n) and λ(n) in either a random manner or in a deterministic manner using Hadamard matrices [8]. We perform averaging of the perturbations in order to estimate the gradient of L(, ). Define fn (θ(n), λ(n)) and f2 n (θ(n), λ(n)) to be the two averaged perturbations of g(, ) expressed as, fn+ (θ(n), λ(n)) = f n [g(θ(n) (θ(n), λ(n)) + a(n) + c ] (n), λ(n) c 2 (n)) fn(θ(n), λ(n)), (8) and, fn+ 2 (θ(n), λ(n)) = f2 n [g(θ(n) (θ(n), λ(n)) + a(n) c ] (n), λ(n) + c 2 (n)) fn(θ(n), 2 λ(n)), (9) where c and c 2 are constants, c, c 2 (0, ] and (n) is an N dimensional perturbation sequence, i.e., either a random Bernoulli sequence or a deterministic Hadamard sequence. {a(n)} is a gain sequence. The estimated gradient w.r.t θ of the Lagrangian, n (θ(n), λ(n)), in a time slot n, can then be expressed as, n (θ(n), λ(n)) = f n(θ(n), λ(n)) f 2 n(θ(n), λ(n)) 2c (n) (20) Note that θ i [0, ]and i θ i =. To ensure these conditions as well as to ensure boundedness of the λ iterates in (6) and (2), we introduce projection functions π and π 2. π projects the θ iterates in [0, ] and π 2 projects the λ iterates in [0, L] for some L > 0. With these considerations and using the estimate of the gradient instead of the actual gradient, the modified θ iteration can be expressed as, θ(n + ) = π [ θ(n) + b(n) n (θ(n), λ(n)) ], (2) while the modified λ iteration can be expressed as, λ(n + ) = π 2 [ λ(n) + c(n)(x(n) q) ]. (22) The gain sequences {a(n)}, {b(n)}, {c(n)} in (8),(9),(2) and (22) have properties, a(n) = b(n) = c(n) = n n n (a(n) 2 + b(n) 2 + c(n) 2 ) < n b(n) a(n), c(n) 0. (23) b(n) The following O-TODCS algorithm based on the above equations is executed at the base station in every time slot n. The scheduler has knowledge of the user queues and since we assume perfect CSI at the base station, the scheduler also has information about the channel quality of each of the users in a time slot.

4 : Using the CSI, determine the rate at which data can be transmitted to each user. 2: Determine the number of packet arrivals in the user queue and the queue lengths of all the users. 3: Calculate the functions fn (, ), f2 n (, ) using (8) and (9). 4: Calculate the estimate of the gradient of L(, ) w.r.t θ, n (, ) using (20). 5: Calculate the scheduling probability vector θ(n) using (2). 6: Calculate the LM vector λ(n) using (22). 7: Schedule a user based on the probability distribution θ(n) and update the queue lengths. Algorithm : The Online Throughput Optimal Delay Satisfying (O-TODCS) Algorithm C. Convergence of the Online Algorithm We now provide a proof of convergence of the Algorithm. Given λ, the scheme in Algorithm is a stochastic gradient ascent scheme in the primals, i.e., the scheduling probabilities. To learn the correct LMs, we perform a gradient descent in the dual space. The directional gradient averaging, the primal maximization and the dual minimization are performed at different timescales, the properties of the gain sequences in (23) ensures this. The dual minimization is carried out at a slower timescale so that it sees the primal maximization as equilibriated while the primal maximization sees the dual minimization as almost constant [6]. Theorem : The iterates in Algorithm converge to the optimal probabilities and Lagrange Multipliers (θ, λ ). Proof: The proof proceeds on the lines of the proof given in [9]. We provide here a sketch of the proof. The dual minimization (λ iterations) being carried out on a slower timescale, the primal maximization (θ iterations) see the λs as constant. Let X (λ) = {θ : L(θ, λ) = G(λ) def = max θ L(θ, λ)}. It can be shown that the map λ G(λ) is concave. Hence X (λ) is closed convex and connected. The averaging property of stochastic approximation in the fn(, ), fn(, 2 ) iterations leads to the empirical gradient estimates being close to the true gradients by standard SPSA arguments. The θ iterations therefore converge to X (λ). (Strictly speaking, they converge to a small neighborhood thereof, because we are using an approximate gradient and not the exact one. We ignore this subtlety, which will introduce a small error component.) Since λ itself varies on a slower timescale, they track the sets X (λ(n)) by standard two timescale arguments [6]. Let denote the gradient in the λ variables. By the envelope theorem of mathematical economics [6], L(θ, λ) G(λ) for θ X (λ), where G is the subdifferential of G. Thus the λ iterations track the differential inclusion λ(t) G(λ(t)), and will therefore converge to the set of minima of G. Combining the above, (θ(n), λ(n)) converge to the saddle point of the Lagrangian. IV. SIMULATION SETUP AND RESULTS Before providing the details of the simulation setup, we discuss a heuristic scheduler that is also a delay satisfying algorithm. A. Heuristic Algorithm The heuristic scheduler works as follows. Let in a time slot n, ŷ i be the estimate of y i, the average rate at which data can be transmitted to a user i. ŷ i is calculated using exponential averaging as, ŷ i (n + ) = αŷ i (n) + ( α)y i (n), (24) where y i (n) is the rate at which data can be transmitted to a user i in time slot n and α (0, ). Then the heuristic scheduler schedules a user i in a time slot n such that, i = arg max j { (ˆxj (n) q j ) q j y j(n) ŷ j } (25) where ˆx j (n) is the average queue length of user j upto time slot n. The first term in the RHS in (25) is the normalized delay violation while the second term is the normalized channel gain. The scheduler schedules a user having the maximum product of the two quantities. B. Simulation Setup We perform simulations of the system using MATLAB. We simulate the system using 50 users divided into 0 groups for time slots of equal duration. During each time slot, we generate the arrivals for all the users using Poisson distribution with means specified in terms of packets/timeslot. Arrivals are generated in an IID manner across slots. We assume that the maximum number of arrivals that can occur in a time slot for a user is 8 packets. We simulate a Rayleigh channel for each of the users. Hence, h i (n) 2 in (3) is an exponentially distributed random variable with density expressed in (2). The mean of h i (n) 2, i.e., σi 2 is the average SNR for user i (expressed in dbs). We normalize the ratio P N 0 to. The rate at which the base station can transmit to a user i in time slot n is given by (3) and is upper bounded by 25 packets. The throughput achieved in a time slot is expressed in terms of packets/timeslot/hz. The users specify their average packet delay constraints in terms of number of timeslots. The users in a group have same values of the system parameters, i.e., the average arrival rate, the average SNR and delay constraints. We perform two experiments. In Experiment, we vary the arrival rate for Group and measure the user delay and system throughput, while in Experiment 2 we vary the average SNR for group 0 and measure the same quantities. For both the experiments, we compare the three schedulers, i.e., the, the pure opportunistic scheduler and

5 the heuristic scheduler. In each experiment, we consider 0 scenarios with different values of system parameters. Each scenario is run 0 times keeping the parameter values constant for all the 0 runs and the average of the 0 runs is considered. In Experiment, we maintain the parameters corresponding to Groups 2-0 constant for all the 0 scenarios as specified in Table I. The arrival rate of Group is increased from 0.00 packets/timeslot to 0.0 packets/timeslot over the 0 scenarios in steps of 0.00 packets/timeslot. In Experiment 2, we maintain the average SNR and delay constraints corresponding to Groups -9 constant for all the 0 scenarios as specified in Table I. The average SNR of Group 0 is decreased from 0 db to db over the 0 scenarios in steps of db. The arrival rates are maintained constant at 0.00 packets/timeslot for all the groups for all the simulation scenarios. We plot the simulation results for the three schedulers in Figures, 2, 3 and 4. User delay (timeslots) e Fig Arrival rate of single group (packets/timeslot) Single user delay variation with arrival rates TABLE I SIMULATION PARAMETER DETAILS Group Arrival Rate(pkts/ts) Average SNR(dB) Delay(ts) For the Algorithm, we set the parameters as, c = 0.0, c 2 = 0.05, a(n) = n, b(n) = 0.0 n, c(n) = 0.9 n. C. Results Figure depicts the variation of the delay of a user in Group against various arrival rates for Group. Figure 2 depicts the variation of the system throughput against various arrival rates for Group. In this experiment, Group has poor channel conditions, hence the pure opportunistic scheduler allocates a very little share of the bandwith to this group. From Figure, we observe that the pure opportunistic scheduler achieves a very high user delay. The and the heuristic scheduler satisfy the user delay requirements, but the system throughput provided by the heuristic scheduler is low than that of the as can be seen from Figure 2. The attempts to maximize the throughput while making the scheduling decision and is able to provide a higher system throughput for the same user delay constraints as compared to the heuristic scheduler for all arrival rates. Figure 3 depicts the variation of the delay of a user in Group 0 against various channel gains (average SNR) for Group 0. Figure 4 depicts variation of the system throughput against various channel gains (average SNR) for Group 0. From Figure 3, it can be seen that the O-TODCS scheduler is able to satisfy the user delay requirements even when it perceives extremely poor channel quality. The heuristic algorithm is also able to satisfy the user delay constraints as can be seen from Figure 3. But the is able to achieve a higher system throughput than the heuristic scheduler for all average SNR values as can be seen from Figure 4. D. Discussion The pure opportunistic scheduler is arrival process and delay agnostic. The emphasis is on maximizing the throughput. Users facing perennially bad channel quality are starved and hence their average delay is high. The allocates sufficient bandwidth even to the users experiencing poor channel quality and does not allow their average queue lengths to grow beyond their specified requirements. The emphasis is on satisfying the user specified delay requirements, but at the same time maximizing the system throughput because of which it provides a better throughput than the heuristic scheduler. It can be seen that the is able to satisfy the user delay even for users perceiving very bad channel quality as compared to the pure opportunistic scheduler at the expense of comparatively little loss in system throughput. The loss in system throughput in the O-TODCS scheduler is much less than that in the heuristic scheduler which is also a user delay satisfying scheduler. V. CONCLUSION In this paper we have considered the problem of designing a centralized downlink scheduler that satisfies user specified average delay constraints. We have assumed that the statistical information regarding the channel conditions and the arrival processes of the users is not known to the scheduler. We have formulated the problem as a constrained optimization problem where the objective is to maximize the average system throughput under the user specified average delay constraints. To solve the problem, we have designed a throughput optimal delay constraint satisfying scheduler based on the stochastic gradient scheme. We have made use of the SPSA algorithm and the two timescale formulation to obtain an iterative algorithm that determines the optimal user scheduling probabilities. We have proved the convergence

6 System throughput (packets/timeslot/hz) User delay (timeslots) System throughput (packets/timeslot/hz) Fig Arrival rate of single group (packets/timeslot) System throughput variation with arrival rates Fig Average SNR of single group (db) Single user delay variation with channel condition Fig. 4. Average SNR of single group (db) System throughput variation with channel condition of the algorithm. We have also compared the user delay and system throughput performance of the scheduler with a pure opportunistic scheduler and a delay satisfying heuristic scheduler under various simulation scenarios. The simulation results have shown that our algorithm is able to satisfy user delay requirements even for users perceiving extremely poor channel quality at the expense of very little loss in system throughput. The system throughput of our scheduler is much better than that of the delay satisfying heuristic scheduler. In this paper we have investigated the downlink delay constrained problem, where power efficiency is not a central issue. We are currently investigating the problem of power efficient, delay constrained scheduling on the uplink. REFERENCES [] R. Knopp and P. A. Humblet, Information Capacity and Power Control in Single-Cell Multiuser Communications, in Proceedings of IEEE ICC 995, vol., Seattle, USA, June 995, pp [2] X. Liu, E. Chong, and N. Shroff, Transmission Scheduling for Efficient Wireless Resource Utilization with Minimum-Performance Guarantees, in Proceedings of IEEE VTC 200, vol. 2, Oct. 200, pp [3] S. Kulkarni and C. Rosenberg, Opportunistic Scheduling Policies for Wireless Systems with Short Term Fairness Constraints, in Proceedings of IEEE GLOBECOM 2003, vol., Dec. 2003, pp [4] Y. Liu and E. Knightly, Opportunistic Fair Scheduling over Multiple Wireless Channels, in Proceedings of IEEE INFOCOM 2003, vol. 2, San Francisco, USA, Mar. 2003, pp [5] F. Berggren and R. Jantti, Asymptotically Fair Transmission Scheduling over Fading Channels, IEEE Transactions on Wireless Communication, vol. 3, no., pp , Jan [6] A. Bhorkar, A. Karandikar, and V. S. Borkar, Power Optimal Opportunistic Scheduling, accepted in IEEE Globecom [7] R. A. Berry and R. G. Gallager, Communication Over Fading Channels With Delay Constraints, IEEE Transactions on Information Theory, vol. 48, no. 5, May [8] M. Goyal, A. Kumar, and V. Gupta, Power Constrained and Delay Optimal Policies for Scheduling Transmissions over a Fading Channel, in Proceedings of IEEE INFOCOM 2003, San Francisco, USA, Mar. 2003, pp [9] M. Andrews, K. Kumaran, K. Ramanan, A. Stolyar, P. Whiting, and R. Vijayakumar. (2000, Apr.) CDMA Data QoS Scheduling on the Forward Link with Variable Channel Conditions. [Online]. Available: tm.ps [0] E. Yeh and A. Cohen, Delay Optimal Rate Allocation in Multiaccess Fading Communications, in Proceedings of IEEE Workshop on Multimedia Signal Processing, Oct. 2002, pp [] R. Berry and E. M. Yeh, Cross-Layer Wireless Resource Allocation, IEEE Signal Processing Magzine, vol. 2, no. 5, pp , Sept [2] H. Wang and N. Mandayam, Delay and Energy Constrained Dynamic Power Control, in Proceedings of IEEE GLOBECOM 200, vol. 2, 200, pp [3] D. Tse and P. Viswanath, Fundamentals of Wireless Communication. Cambridge University Press, [4] D. P. Bertsekas and R. Gallager, Data Networks. Prentice Hall, 987. [5] D. P. Bertsekas, Nonlinear Programming. Belmont, MA: Athena Scientific, 999. [6] V. S. Borkar, Stochastic Approximation with Two Time Scales, Systems and Control Letters, vol. 29, pp , 996. [7] J. C. Spall, Introduction to Stochastic Search and Optimization. Wiley, [8] S. Bhatnagar, M. C. Fu, S. I. Marcus, and I.-J. Wang, Two-Timescale Simultaneous Perturbation Stochastic Approximation Using Deterministic Perturbation Sequences, ACM Transactions on Modeling and Computer Simulation, vol. 3, pp , Apr [9] V. S. Borkar, An Actor-Critic Algorithm for Constrained Markov Decision Processes, Systems and Control Letters, vol. 54, pp , 2005.