Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation

Transcription

1 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation Parisa Mansourifard Joint work with: Prof. Bhaskar Krishnamachari (USC) and Prof. Tara Javidi (UCSD) Ming Hsieh Department of Electrical Engineering University of Southern California usc.edu American Control Conferece (ACC), Chicago, Illinois July 2, 2015

2 Introduction A network with uncertain available bandwidth, BW Demand (transmission rate) allocation, r? r > BW : Congestion r < BW : Under-utilization Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 1/ 18

3 Introduction A network with uncertain demands (users to be served), d Resource (capacity) allocation, C? C < d: Unsatisfied demands C > d: Over-utilization, e.g. energy consumption Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 2/ 18

4 General Formulation Uncertain demand (resource) Markovian Random process Allocated resource (demand) Action Goal of decision-maker Maximize expected reward or minimize expected cost Asymmetries Over-utilization cost vs. under-utilization cost Full observation vs. partial observation Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 3/ 18

5 Problem Formulation A Discrete-time real-state Markov process The finite horizon by T and the discrete time steps by t = 1, 2,..., T Known transition probabilities Objective: to select the sequential actions s.t. total expected discounted reward is maximized. Modeled as Partially Observable Markov Decision Process (POMDP) Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 4/ 18

6 Problem Formulation: Model POMDP State: B t M = [m, M] R, i.e. the state space. State transition: The transition probabilities of the actual states over time are shown m x, y M by Action: r t M p(x y) := P(B t = x B t 1 = y) Observed information: the event o t (r t ) O: -O F : o t (r t ) = {B t = i}, i [m, r t ) is the event of fully observing B t -O P : o t (r t ) = {B t r t } is the event of partial observing that B t is larger than or equal to the selected state Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 5/ 18

7 Problem Formulation: Model POMDP Reward: The immediate reward is a mapping R : M O R: R(B t, r t ) = q min(b t, r t ) - c u : over-utilization cost coefficient - c l : under-utilization cost coefficient - q: the gain unit Trade-offs Over-utilization cost vs. under-utilization cost { c u (r t B t ) if r t > B t c l (B t r t ) if r t B t, Immediate expected reward vs. earning information for future decisions Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 6/ 18

8 Problem Formulation: Example - An example tracking of a sample path of the actual states B t - With action sequence r t Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 7/ 18

9 Problem Formulation: Goal Optimal Policy The policy π specifies a sequence of actions r t to be taken. Goal: to maximize the total expected discounted reward in the infinite horizon, over all admissible policies π, given by max π Jπ T (s 0 ) = max π E[ β t R(B t ; r t ) s 0 ] t=0-0 β < 1: the discount factor - s 0 : the initial observed state - The optimal policy π opt : maximizing policy - Proof of existence [Bensoussan et al. 2007] Problem Solving Approach Dynamic programming (DP) Computationally intractable Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 8/ 18

10 Related Works A communication system where the transmitter must select the transmission rate in order to maximize the number of successfully transmitted bits [Laourine et al. 2010] Structure of optimal policies has been established for simpler related problems of optimizing transmissions over a two-state Gilbert-Elliott channel [Laourine et al. 2010], [Johnston et al. 2006] The optimal transmission policy for a Gillbert-Elliot channel with unknown statistics: e.g. [Wu et al. 2012] POMDP problem where the demand is a Markovian process, continuous state, only myopic policy (lower bound on optimal policy): [Bensoussan et al. 2007] multi-period newsvendor models in which the leftovers at each time step can be carried over to satisfy the future demand: e.g. [Bensoussan et al. 2008], [Chen et al. 2010] Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 9/ 18

11 Our Contributions Characterize optimal policy with only two parameters First work to present a sequence-based formulation New ordering of sequences Prove lower and upper bounds on the whole optimal sequence Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 10/ 18

12 Problem Solving Approach: Two Equivalent Value Iterations Belief-based: - Belief is defined as the probability density function of states, f t (x) - Existence of optimal policy is proved in Bensoussan et al Sequence-based: - Based on action sequences starting from each possible observed state - Find the best sequence to maximize the total expected discounted reward. Proposition: The optimal policy can also be perfectly characterized by only two parameters; s L : last observed state t L : time steps passed since the last observation Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 11/ 18

13 An Example Behaviour of Sequence-Based Policy a(i,.) = {a(i, 1), a(i, 2),...}: sequence of actions starting from state i a(i, t L ): action selected at t L time steps passed from last observed state i. a opt (i,.): optimal action sequence Figure : An example of executing an arbitrary policy (a(i,.) = i + a(0,.)) Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 12/ 18

14 Sequence-Based Value Iteration W (i): The expected cost-to-go after last observed state i W (i) = sup W (i; a(i,.)), a(i,t L ) [m,m], t L W (i; a(i,.)) = t L =1 a(i,tl ) j=m P t L i,a(i,1:t L 1),j dj t L 1 [ β τ 1 ((q + C l )a(i, τ) C l B(i, τ)) τ=1 + β t L 1 ((q + C u )j C u a(i, t L )) + β t L W (j)] Optimal Policy π opt : maps any state i to a sequence, a opt (i,.) = arg sup a(i,tl ) [m,m], t L W (i; a(i,.)). Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 13/ 18

15 Main Results: A Lower Bound Myopic Policy: At each time step maximizes only the immediate expected reward, ignores the impact on the future reward Myopic Sequence a myopic (i, t L ) = inf{r M : r j=m P t L i,a myopic (i,1:t L ),j dj = q + c l } q + c l + c u P t L i,a(i,1:t L 1),j : the probability of occurring the following event, - no reset at t = 1, 2,..., t L 1 passed from the last observed state s L = i - following the action sequence of a(i, 1 : t L ) = {a(i, 1),..., a(i, t L )} - reset to the actual state j at t L. Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 14/ 18

16 Main Results: A Lower Bound Theorem: Lower Bound on Optimal Sequence For FOSD-preserving transition probabilities, a opt (s L, t L ) a myopic (s L, t L ), t L Definition: First Order Stochastically Dominance (FOSD), f 1 s f 2 M M f 1 (j) f 2 (j), r M j=r j=r F 1 (r) F 2 (r) Definition: The transition probability p(x y) is FOSD-preserving if for any f 1 s f 2, M M f 1 (y)p(x y)dy s f 2 (y)p(x y)dy y=m y=m Proposition: Ordering For FOSD-preserving transition probabilities: a myopic (i, t L ) a myopic (j, t L ), i j Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 15/ 18

17 Main Results: An Upper Bound for IIMC Processes Definition: IIMC Continuous-State Process The transition probabilities for the state space M = R, satisfies the following: p(x y) = p(x + α y + α) α, x, y R Theorem: Upper Bound Sequence For IIMC processes and c l = 0, for any s L = i, a opt (i, t L ) a UB (i, t L ), t L : a UB (i, t L ) = inf{r [r l, r h ] : U = qβ 1 β (r h r l ) r h = sup{j : P t L i,a UB (i,1:t L ),j 0} r l = inf{j : P t L i,a UB (i,1:t L ),j 0} r P t L i,a UB (i,1:t j=r l L ),j dj = q + U q + c u + U } Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 16/ 18

18 Summary and Proposed Future Works Summary The tracking problem of Markovian random processes in which the decision-maker must select the best action at each time step in order to maximize the total expected discounted reward Optimal policy computationally intractable Sequece-based formulation Optimal policy can be characterized by only two parameter Upper and lower bound on the optimal sequences Future Works Deriving upper bound for general form of processes and relaxing the assumption of zero under-utilization Heuristic Percentile Threshold policy for general form of processes Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 17/ 18

19 Thank you for your attention Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 17/ 18

20 References A. Laourine and L. Tong, Betting on gilbert-elliot channels, Wireless Communications, IEEE Transactions on, vol. 9, no. 2, pp , L. A. Johnston and V. Krishnamurthy, Opportunistic file transfer over a fading channel: A pomdp search theory formulation with optimal threshold policies, Wireless Communications, IEEE Transactions on, vol. 5, no. 2, pp , Y. Wu and B. Krishnamachari, Online learning to optimize transmission over an unknown gilbert-elliott channel, in Modeling and Optimization in Mobile, Ad Hoc and Wireless Networks (WiOpt), th International Symposium on. IEEE, 2012, pp K. Arrow, T. Harris, J. Marshak, Optimal inventory policy, Econometrica 19 (3) (1951) X. Ding, M. L. Puterman, and A. Bisi, The censored newsvendor and the optimal acquisition of information, Operations Research, vol. 50, no. 3, pp , A. Bensoussan, M. Cakanyldrm, and S. P. Sethi, Technical notea note on the censored newsvendor and the optimal acquisition of information, Operations Research, vol. 57, no. 3, pp , A. Bensoussan, M. Cakanyldrm, and S. P. Sethi, On the optimal control of partially observed inventory systems, Comptes Rendus Mathematique, vol. 341, no. 7, pp , A. Bensoussan, M. Cakanyldrm, J. A. Minjarez-Sosa, A. Royal, and S. P. Sethi, Inventory problems with partially observed demands and lost sales, Journal of Optimization Theory and Applications, vol. 136, no. 3, pp , L. Chen, Bounds and heuristics for optimal bayesian inventory control with unobserved lost sales, Operations research, vol. 58, no. 2, pp , A. Bensoussan, M. Cakanyldrm, and S. P. Sethi, A multiperiod newsvendor problem with partially observed demand, Mathematics of Operations Research, vol. 32, no. 2, pp , Parisa Mansourifard, USC, ACC 2015 Tracking of Real-Valued Markovian Random Processes with Asymmetric Cost and Observation 18/ 18