Markov Decision Model for Perceptually. Optimized Video Scheduling

Transcription

1 Markov Decision Model for Percepually 1 Opimized Video Scheduling Chao Chen, Suden Member, IEEE, Rober W. Heah Jr., Fellow, IEEE, Alan C. Bovik, Fellow, IEEE, and Gusavo de Veciana, Fellow, IEEE, Absrac Transmiing video over slow fading wireless channels wih good percepual qualiy is a challenging ask because no ime-diversiy can be exploied o comba channel variaions, especially when he frequency diversiy and spaial diversiy is no available due o he wireless sysem implemenaion. While qualiy-scalable video coding echniques make video source-rae adapaion possible, deermining a good scheduling sraegy which selecively schedules video daa associaed wih differen layers is a challenging problem. For he bes performance of a wireless video sysem, he scheduler needs o consider he channel sae, he buffer sae and he percepual video qualiy a he receiver. In his paper, we propose a scheduling algorihm o opimize he percepual qualiy of scalably coded videos ransmied over slow fading channels. By modeling he dynamics of he channel as a Markov chain, we reduce he problem of dynamic video scheduling o a racable Markov decision problem over a finie sae space. We hen employ an infinie-horizon average-reward maximizaion algorihm o maximize he ime-average Muli-Scale Srucural SIMilariy (MS-SSIM) index which has been shown o correlae highly wih human judgmens of video qualiy. Simulaion resuls show ha he proposed MDP-based scheduling policy achieves significan percepual qualiy improvemen over scheduling mehods which do no explicily exploi he channel dynamics. Furhermore, we propose an on-line scheduling mehod which no only performs nearly as well as he MDP-based performance bu also has very low implemenaion complexiy. Index Terms Videos, Scheduling algorihm, Wireless communicaion, Image qualiy. The auhors are wih Deparmen of Elecrical and Compuer Engineering, The Universiy of Texas a Ausin, 1 Universiy Saion C0803, Ausin TX , USA chao.chen@uexas.edu This research was suppored in par by Inel Inc. and Cisco Corp. under he VAWN program.

2 2 I. INTRODUCTION Video ransmission over wireless channels is a challenging ask. The hroughpu of wireless channels varies over ime, making he delivery of real-ime video challenging due o igh delay consrains. In paricular, if he coherence ime of he channel is comparable o he delay consrain, hen he ime-diversiy of he channel canno be exploied. Tradiional channel coding mehods canno provide graceful visual qualiy degradaion of he received video signal when deep fading happens. Hence, adapive ransmission echniques such as muli-layer scheduling and link-adapaion should be employed. Furhermore, video packes are srucured. Due o he naure of predicive video coding algorihms, a video frame can be decoded only when is predicors are received a he receiver. Hence, he predicion srucure of he video codec enforces a parial order on he ransmissions of he video packes. Scalable video coding (SVC) is one approach for allowing flexible video ransmission over channels wih varying hroughpu [1], [2]. An SVC video encoder produces a layered video sream ha conains a base layer and several enhancemen layers. If he hroughpu is low, he ransmier can choose o ransmi he base layer only, which provides a moderae, bu accepable, degree of visual qualiy a he receiver. If he channel condiions improve, he ransmier can ransmi one, or more, enhancemen layers o furher improve he visual qualiy. Concepually, SVC provides a means o adap he daa rae for wireless video ransmission. The wireless ransmier can easily adap he daa rae by selecively scheduling video daa associaed wih various layers raher han re-encoding he video sequence ino a suiable rae. Designing scalable video scheduling algorihms for wireless channels is a complex ask. The scheduling policy depends no only on he channel condiion bu also on he receiver buffer sae. For example, if he receiver has successfully buffered base layer daa over many frames, he scheduler could choose o ransmi some enhancemen layer daa o improve he video qualiy even if he hroughpu is low. The scheduling policy also depends on he impac ha paricular video daa packe will have on he percepual video qualiy. The scheduler should assign higher prioriy o packes which could resul in higher percepual qualiy improvemens. An effecive percepual qualiy meric and an accurae raequaliy model are imporan for scheduling policy design. The objecive of his paper is o develop scheduling algorihms o maximize he receiver percepual video qualiy for scalable video ransmission over wireless channels. A. Conribuions In his paper, we assume ha video sequences are encoded by an H.264/SVC-compaible scalable video encoder. We employ a finie sae Markov chain (FSMC) o model he dynamics of he slow

3 3 fading channel. We also employ a rae-qualiy model o capure he relaionship beween he size of a video daa packe and is conribuion o he receiver percepual qualiy. We model he dynamic video ransmission sysem as a conrolled Markov sysem. The visual qualiy measured in ime-average MS-SSIM index is maximized by opimizing he scheduling policy via value ieraion (see e.g. [3]). The specific conribuions ha we make are as follows: 1) A racable MDP-based formulaion is proposed o design he opimal scheduling policy. Typical mobile users usually have available applicaion layer sorage space of several Gigabyes. Thus, he buffer size can be regarded as infinie. Because he performance of scheduling policies depend on he buffer sae, he policies needs o be opimized over a infiniely large sae space. By making some reasonable approximaions, we fix he scheduling policy excep a finie se. We prove ha opimizing he ransmission policy for his finie-sae se is equivalen o solving a semi-markov decision problem. Based on his resul, a value ieraion algorihm is used o opimize he scheduling policy. 2) Accurae predicion of visual qualiy is used as he opimizaion objecive. In mos video ransmission mehods, Peak Signal o Noise Raio (PSNR) is used as he opimizaion objecive. I is well known ha PSNR does no accuraely predic he percepual qualiy of videos in many insances [4] [5] [6]. In his paper, we employ an infinie-horizon average-reward maximizaion formulaion which is direcly relaed o he ime-average MS-SSIM index [7]. As shown in [8], he ime-average MS-SSIM index correlaes quie well wih human judgmens of visual qualiy. In his paper, he sysem sae is mapped o MS-SSIM index using a simple rae-qualiy model. Then, he MS-SSIM value of each sae is used as he sae-value in he value ieraion algorihm. 3) A simple and near-opimal scheduling mehod is proposed. Compuing he scheduling policy based on he MDP formulaion requires exra sysem resources. We devise a simple and nearopimal scheduling algorihm which, similar o he MDP-based policy, proacively ransmis daa for laer GOPs and dynamically schedules daa associaed wih differen layers. B. Relaed Work MDP-based sochasic conrol echniques have been proposed for video daa scheduling [9] [14]. In [9], adapive video ransmission over a packe erasure channel was sudied by modeling he buffer sae as a conrolled Markov chain. Laer, in [10], an MDP-based scheduling algorihm was proposed for video ransmission over packe loss neworks. This work was furher exended for wireless video sreaming in [11]. The wireless channel was modeled as a binary symmeric channel. This channel model is jusified for fas fading channels where he coherence ime is much less han he delay

4 4 consrain. In ha case, inerleaving can be applied wihou violaing he delay consrain and he channel will appears like a i.i.d channel. For slow fading channels, he bi error rae canno be modeled as a consan. In [12] [13], a reinforcemen learning framework was proposed for wireless video ransmission. Their algorihm was based on MDP wih discouned-reward maximizaion formulaion. The ransmier learns he characerisics of he channel and he video sequence during he ransmission process. The scheduling policy is updaed according o he learned characerisics. In our previous work [14], an infinie-horizon average-reward maximizaion formulaion was proposed. A very simple rae-qualiy model was employed o differeniae he imporance of differen layers. The difference beween frames, however, was no incorporaed in he rae-qualiy model. In addiion, PSNR, insead of ime-average MS-SSIM index, was used as he opimizaion objecive. MDP-based mehods have been used o solve oher problems in adapive video ransmission. In [15], an MDP formulaion was proposed for adapive video playou and scheduling for single layer videos. A wo-sae channel model was used o represen good and bad channel condiions. The conroller adaps he playou speed according o he receiver buffer sae and channel sae in order o opimize he PSNR of he signal a he receiver. In [16], an MDP-based formulaion was inroduced for he problem of real-ime encoder rae-conrol. The derived opimal rae-conrol policy adaps he encoding bi rae according o he channel condiion and video rae-qualiy characerisics. In [17], muli-user scheduling and rae adapaion in wireless local-area neworks (WLAN) were sudied. The users in a WLAN access he available resources in a decenralized manner. The muli-user scheduling problem was formulaed as a compeiive Markov decision process [18] and a Nash-equilibrium policy was compued via a value ieraion algorihm. Among all he menioned work, he mos closely relaed work o ours are [10] and [12] which focus on single user scalable video ransmission. The differences from our work are summarized as follows: The video scheduling algorihm proposed in [10] was developed for packe loss neworks. In a packe loss nework, he hroughpu of each ransmission slo is limied by congesion raher han by deep signal fading in he ransmission medium. Hence, in [10], he channel was assumed o be ime-invarian. Each packe was assumed o be los or delayed independenly of he oher packes. Based on hese assumpions, he scheduling policies for differen video packes can be facorized. Thus he policy opimizaion problem was grealy simplified. For wireless channels, he channel sae is ime varying and hus he packe losses are no independen across coheren periods, hence scheduling policy facorizaion is no possible. In [12], Zhang e al. sudied single user video ransmission over wireless channels. The conrol

5 5 policy was learned on-line hrough reinforcemen learning. The major drawback of reinforcemen learning is ha i akes ime o learn from he wrong scheduling acions. The scheduler may cause bad visual qualiy during he learning period. As shown in [12] and [19], wih an acceleraed on-line learning algorihm, he scheduling policy converges afer 25 frames are ransmied. In our work, he channel dynamics are assumed o be known. Indeed, he finie sae Markov model can be obained analyically [20] or by measuremen [21]. The channel model is combined wih a simple rae-qualiy model o form a model-based MDP formulaion. The scheduling policy hus can be derived off-line. In [12] [16] [19], a discouned-reward maximizaion formulaion is employed o rade off he visual qualiy of recen and laer frames. The discouning facor needs o be chosen heurisically and affecs he performance of he derived scheduling policy. In our work, an average-reward maximizaion formulaion is proposed. This formulaion is naurally relaed o he ime-average MS-SSIM index which correlaes well wih human objecive judgemens of visual qualiy [8]. C. Noaion Used A and a are examples of a marix and a vecor, respecively. A is a se. A is he cardinaliy of se A. 1 is he uni vecor of all-ones and 0 is he zero vecor. max{a, b} and min{a, b} are he componenwise maximum and minimum of vecor a and b, respecively. 1( ) is he indicaor funcion. is he ceiling funcion. P( ) is he probabiliy measure and E[ ] is he expecaion. II. SYSTEM MODEL In his secion, we firs describe he wireless video sysem o be considered. Then, we presen our video codec configuraions and inroduce he rae-qualiy model based on MS-SSIM index. Finally, we presen he Markov channel model o be used in he sequel. A. Sysem Overview We consider a ime-sloed scalable video ransmission sysem over slow fading wireless channels. A video sequence is encoded wih a qualiy-scalable video encoder and sored in a video server. The video server ransmis video daa o a mobile user via a wireless ransmier. Each slo, he server sends some video daa upon he requess of a scheduler equipped on he wireless ransmier. These daa are packeized a he wireless ransmier for physical layer ransmission. The scheduler operaes according o a scheduling policy which maps he channel and buffer sae o he scheduling acion(see Fig. 1).

6 6 We assume ha he channel beween he video server and he wireless ransmier is no he boleneck of he link. Thus, in he perspecive of he wireless ransmier, he whole video sequence is accessible. We also assume ha he physical layer channel sae informaion is available a he ransmier and ha he modulaion and coding scheme (MCS) is deermined by a given physical layer link-adapaion policy. B. Video Codec Configuraion We assume ha video sequences are encoded by an H.264/SVC-compaible scalable video encoder. The duraion of each frame ΔT is called a frame slo. The video frames are uniformly pariioned ino Groups of Picures (GOPs). Every GOP has L GOP frames. The firs frame in a GOP is an I frame while he oher frames are P frames. Every frame is encoded ino L layers. The firs layer is he base layer; The oher layers are enhancemen layers. Every enhancemen layer of a frame is predicively encoded using he lower layers of he frame. The base layer of a P frame is predicively encoded using he base layer of is preceding frame. The base layer of an I frame is encoded independenly (see Fig. 2). Each frame has a playou deadline a he receiver. In he following, frames whose deadlines have expired are called expired frames. The oher frames are called unexpired frames. The firs unexpired frame is called he curren frame. A any ime, he frames are indexed relaive o he curren frame as shown in Fig. 2. The video daa in he lh layer of he fh frame is called he (f,l)h video daa uni. We adop he predicion srucure in Fig. 2 raher han he Hierarchical B srucure because no srucural delay is inroduced [1]. Specifically, in he Hierarchical B predicion srucure, he encoding order differs from he display order, hus, he ransmission of a frame mus be delayed unil all necessary predicors are received. Also, due o he ime-varying naure of he wireless channels, he adapive ransmier mus drop some enhancemen layers when channel hroughpu is low. So, if he enhancemen layers are used o predic oher frames as is he case in he Hierarchical B srucure, dropped enhancemen layers can give rise o error propagaion and unpredicable visual qualiy degradaion. A he possible cos of lower compression efficiency, he predicion srucure ha we use will eliminaes error propagaion arising from enhancemen layer losses, since here will be no iner-frame predicion among enhancemen layers.

7 7 C. Rae-Qualiy Model The rae-qualiy model characerizes he relaionship beween he size of a video daa uni and he visual qualiy improvemen when i is correcly received. We adop a simple model. For each P frame, le rl P be he amoun of daa in he lh layer daa uni. For each I frame, le rl I be he amoun of daa in he lh layer daa uni. Define q l o be he visual qualiy incremen afer he lh layer is correcly received, given all is predicors are also received. This model implies ha he visual qualiy improvemen incurred by he enhancemen layers in one frame does no depend on wheher he enhancemen layers of oher frames are received. This is rue for he predicion srucure given in Secion II-B since here is no error propagaion due o losses of enhancemen layers. Convenional image qualiy measures such as he PSNR reflec absolue signal fideliy bu wihou accouning for percepual visual qualiy. Recenly, a variey of models ha accuraely predic percepual video qualiy have been proposed [7], [22] [25]. In our formulaion, we adop MS-SSIM index as he visual qualiy measure [7], since i has been shown o correlae quie well wih percepual visual qualiy and i is of reasonable compuaional complexiy [8]. The MS-SSIM index of a video sequence ranges from 0 o 1. The larger he index, he beer he qualiy. In our rae-qualiy model, he qualiy incremen q l is measured using MS-SSIM index. Therefore, he quaniy q l [0, 1]. Larger values of q l mean larger qualiy improvemen can be achieved by ransmiing he lh layer daa unis. In a real video sequence, rae-qualiy characerisics vary from frame o frame. For simpliciy, we use he average value of he measured rae-qualiy characerisics as esimaes of rl I, rp l and q l. In Fig. 3, he daa raes and MS-SSIM values of wo video es sequences, Foreman and Paris, are shown. These wo sequences are widely used in visual qualiy assessmen. Foreman has higher emporal complexiy and Paris has higher spaial complexiy [26]. As shown in Fig. 3, our proposed model is a good fi for he rae-qualiy characerisics. D. Channel Model We focus on scheduling for a slow fading channel. By slow fading, we mean ha he coherence ime of he channel is less han he duraion of a GOP and larger han he duraion of a frame. Assuming he mobile users are moving in a 1.5m/s walking speed and he carrier frequency is 2GHz, he Doppler spread is abou 10Hz. The coherence ime is abou 100ms. A ypical GOP duraion is abou 1 second and a frame slo is abou 30ms. Hence, for pedesrian video users, wireless channels are slow fading. As he channel sae is sable during each frame slo, he scheduling decision is made in a frame-byframe basis. A he beginning of each frame slo, a frame is played ou. Then, he wireless ransmier

8 8 schedules video daa unis for ransmission according o a scheduling acion. The scheduling acion is defined as an ordered collecion of video daa unis u = { (f 1,l 1 ), (f 2,l 2 ),, (f u,l u ) }. When a scheduling acion u is aken, he daa unis conained in u are ransmied sequenially. A physical layer, each scheduled daa uni is packeized ino physical layer packes and each packe is repeaedly ransmied, i.e., if errors occur, unil acknowledged. The MCSs used in ransmiing daa packes is deermined by a link-adapion policy. In his paper, we focus on scheduling policy design and assume ha he link-adapaion policy is given. In [20] and [27], i is shown ha he firs-order FSMC can be uilized o accuraely describe he firs-order channel sae ransiion probabiliies for Rayleigh fading channels. Firs-order FSMC models have also been validaed in [21] and [28] by channel measuremens of urban area wireless channels. In his paper, we employ a firs-order FSMC o describe he dynamics of he channel sae. I should be noed ha, as poined in [29], a firs-order FSMC is no sufficien o describe high-order channel sae disribuions. Generally, he auocorrelaion funcion (ACF) of a firs-order FSMC is exponenially decreasing and he ACF of a Rayleigh fading channel is a zeroh-order Bessel funcion of he firs kind. To model he higher order dynamics of he wireless channel, a he cos of higher complexiy, a higher order Markov channel model can be applied. A he physical layer, in he h frame slo, he ransmission bi rae R is deermined by he MCS and he packe error rae p is deermined by boh he channel sae and he MCS. Under he given link adapaion mehod, he chosen MCS is a funcion of he channel sae. Thus, here is a one-o-one mapping from channel sae o he uple (R,p ). Due o he Markov propery of he channel sae, (R,p ) can also be modeled by an FSMC. The channel sae space is C = { } C 1,..., C C, where C i =(R i,p i ) is he ih channel sae. The sae ransiion marix P c is a C C marix wih enry P c i,j = P(C j C i ) being he ransiion probabiliy from sae (R i,p i ) o (R j,p j ). III. PROBLEM FORMULATION In his secion, we define he scheduler s sae space and he policies o be considered. Then, we show how o simplify he scheduling problem o a finie-sae Markov decision problem using reasonable approximaions. An infinie-horizon average-reward maximizaion MDP formulaion is proposed o opimize he scheduling policy so as o improve he ime-average MS-SSIM index a he receiver.

9 9 A. Scheduling Policy and Sae Space Considering all he possible scheduling acions makes defining he scheduling policy and represening he buffer sae unmanageably complicaed. If we do no apply any consrain on he scheduling acions, he receiver buffer sae could look like Fig. 4. On he one hand, o represen he buffer sae, he frame index and he layer index of each received daa uni need o be recorded. Because he number of received daa unis is no bounded, we canno represen all possible buffer saes using a finie-dimension vecor space. On he oher hand, he scheduling acions which give rise o he buffer sae in Fig. 4 canno provide opimal visual qualiy a he receiver. As shown in Fig. 4, some video daa unis are ransmied before heir predicors. If heir predicors are no received before heir playou deadlines, hese unis are undecodable and useless. In his paper, by applying reasonable consrains on he scheduling acions, we concenrae on hose scheduling sraegies which are possible o presen good performances. Specifically, we consider he scheduling policies which comply wih he following consrains. Consrain 1: The scheduler always schedules a daa uni laer han is predicors in he predicion srucure. Consrain 2: The amoun of video daa scheduled in he h slo is jus larger han R ΔT, i.e., he amoun of daa which can be ransmied in he slo. Consrain 3: The scheduler never schedules more enhancemen layer daa unis for laer P frames han sooner P frames in he same GOP. Consrain 1 is applied o make sure ha he ransmission order is compaible wih he predicion order given in Secion II-B, since a daa uni can be decoded only when is predicors are received. Consrain 2 forces he ransmier o keep busy ransmiing daa during he enire slo. Wih Consrain 3, a any ime and for all he P frames wihin a GOP, he scheduler does no sacrifice he qualiy of he frames which will be displayed sooner for he frames o be displayed laer by ransmiing more enhancemen layer daa for he laer. Because he opimizaion objecive is ime-average MS-SSIM index, he qualiy of each P frame is equally imporan. Transmiing more enhancemen layer daa for laer frames does no help o improve he ime-average MS-SSIM value. Noe ha, alhough he P frames wihin a GOP are equally imporan for ime-average MS-SSIM index, he frames in differen GOPs are no. As discussed in Secion I, when he channel hroughpu is very low, i is beneficial o sacrifice P frames in curren GOP for ransmiing he I frame of he nex GOP. To differeniae he imporance of curren and fuure GOPs, we pariion he daa unis of he unexpired frames ino hree ses: I pre, I and I pos. The se I conains he unexpired daa unis of he

10 10 firs unexpired I frame, I pre conains daa unis before he firs unexpired I frame, and I pos conains he remaining unexpired daa unis (see Fig. 5). In he following, we define he buffer sae spaces for he hree ses as B I, B pre and B pos, respecively. The overall buffer sae space is B = B pre B I B pos. B I A he h slo, he sae of I is defined as B I =(f I,b I ), where f I {1,,L GOP } is he frame index of he firs unexpired I frame and b I is he number of he received daa unis of I. B pre According o Consrain 3, he number of daa unis received in I pre is decreasing wih respec o he frame index. Hence, we only need o record he number of received daa unis for each layer. We define he buffer sae space of I pre B pre I pre. =(b pre 1,b pre 2,,b pre L as a L-dimensional vecor ), where bpre is he number of he received daa unis in lh layer of l B pos I is noed ha Consrain 3 is only applied wihin each GOP. Hence, we canno define he sae space of I pos in he similar way as we did for I pre. In principle, we need o record he number of received daa unis for each frame in I pos. In ha case, we canno define a vecor space wih fixed number of dimensions o represen he sae of I pos. Hence, for simpliciy, we exend Consrain 3 o all he frames in I pos. In oher words, he scheduler never ransmis more enhancemen layers for laer frames han sooner frames in I pos. Similar o I pre,we define he buffer sae space of I pos as a L-dimensional vecor B pos where b pos l is he number of he received daa unis in lh layer of I pos. =(b pos 1,b pos 2,,b pos L ), Remark By exending Consrain 3 o all he GOPs in I pos, we acually rule ou he opion of ransmiing more daa unis for he I frames in laer GOPs of I pos. This may poenially degrade performance bu his is negligible. Transmiing more daa unis for laer I frame is necessary only when he channel sae is bad hroughou he whole GOP. However, as we assumed in Secion II-D, he coherence ime of he channel is much less han he duraion of a GOP. The channel is less likely o be bad hroughou he whole GOP. Hence, qualiy degradaion is less likely o occur. When no video daa unis for he curren frame are received, he decoder canno coninue decoding he frame. The video sream will experience an inerrupion unil all he necessary base layer daa unis for decoding he curren frame are received. We define an inerrupion sae N ir which is he se of he unreceived base layer daa unis for decoding he curren frame a slo (see Fig. 5(b)). When he curren frame is decodable, he inerrupion sae N ir =. N ir conains a mos L GOP 1 daa unis because I frames resynchronize he decoding process and erminae he inerrupion. Because

11 11 every daa uni is ransmied only when all is predicors are received, he daa unis needed for decoding he curren frame mus be composed of a sequence of consecuive base layer daa unis, i.e., N ir = {( N ir +1, 1), ( N ir represened by he number N ir +2, 1),, (0, 1)}. Hence, he inerrupion sae can be simply = N ir. The sysem sae S is defined as he produc of he channel sae, he buffer sae and he inerrupion sae. A slo, he sysem sae is S =(C,B,N ir ), where B =(B pre,b I,B pos ). For each sae S S, we define a feasible conrol se U S which conains all he scheduling acions complying wih all he hree consrains. A any ime, he sae S conains all he informaion abou he receiver buffer and he channel. The ransmier mus decide which acion in U S o ake in order o maximize he ime-average MS-SSIM index value. We define he scheduling policy μ( ) as he mapping from he sysem sae S o an acion in U S. In he following secions, we show how o opimize he scheduling policy μ( ). B. Policy Simplificaion Because he receiver buffer size is regarded as essenially infinie, he sae space B pos is herefore infinie. Opimizing he scheduling policy over his infinie-sae space is inracable. In he following, we reduce he sae space o a finie one by reasonable approximaions. When many frames are buffered a he receiver, he scheduler can ransmi more enhancemen layers because here is enough ime before he frames are played ou. Based on his observaion, we define a window W which conains he daa unis wihin he firs W unexpired frames. The scheduler is resriced o work as follows: If all he daa in W are all received and N ir =, he scheduler schedules as many enhancemen layers as possible. Oherwise, he scheduler only focuses on scheduling he video daa in W and N ir. By using he window W, alhough he sae space is sill infinie, we fix he scheduling acions ouside a finie sae se. In oher words, i is only necessary o find he opimal acions for a finiesae se. The window size W provides a radeoff beween complexiy and opimaliy. The larger he window, he less consrained he conrol policy bu he higher complexiy. I should be noed ha he window defined here is differen from he sliding window defined in [10] and [19]. Our scheduling policy allows he ransmier o ransmi he daa unis ouside he window. C. Transiion Probabiliy Le S =(C,B,N ir ) and U S be he sysem sae and he corresponding feasible conrol se a slo, where C =(R,p ) and B =(B pre,b I,B pos ). A he beginning of each slo, one frame is

12 12 decoded and played ou. We le B + =(B pre+,b I+,B pos+ ) denoe he buffer sae righ afer he firs frame is displayed. If f I =1, i.e., he decoded frame is an I frame, hen he frame se I becomes he nex I frame, i.e., he L GOP h frame (see Fig. 6(a)). Hence, buffer sae is ( ) L B I+ = L GOP, 1(b pos l L GOP ), (1) where L l=1 1(bpos l l=1 L GOP ) is he number of received layers in he nex I frame. Meanwhile, I pre becomes he firs L GOP 1 frames and I pos conains he frames whose index is larger han L GOP. Thus, we have B pre+ = min { B pos, (L GOP 1)1 } (2) and B pos+ =max { B pos L GOP 1, 0 }. (3) If he decoded frame is no an I frame, he frame se I pos will no be affeced and he buffer sae B pos and does no change (see Fig. 6(b)). B pre B pre+ becomes =max{b pre 1, 0}. (4) Summarizing (1), (2), (3) and (4), we have min { B pos B pre+, (L GOP 1)1 } if f I =1, = max {B pre 1, 0} if f I 1, ( L GOP, ) L B I+ l=1 1(bpos l L GOP ) if f I =1, = (f I 1,l I ) if f I 1, B pos+ = max { B pos L GOP 1, 0 } if f I =1, B pos if f I 1. Afer he firs frame is displayed, he ransmier begins o sequenially ransmi he collecion of video daa unis indicaed by he acion u = μ(s )={(f 1,l 1 ),, (f u,l u )}. Le Δu = {(f 1,l 1 ),, (f n,l n )} denoe he compleely received daa unis by he end of he slo, where n is he number of received daa unis. Among he daa unis in Δu, le ΔB pre =(Δb pre 1, Δb pre 2,, Δb pre ) be he number of newly received daa unis for each layer in frame se I pre. Similarly, we denoe ΔB pos = (Δb pos 1, Δb pos 2,, Δb pos ) as he number of newly received daa unis for each layer in frame se Ipos L L

13 13 and Δl I as he number of received daa unis for I. A he beginning of he ( +1)h slo, we have he following sae ransiion relaionship B pre +1 = B pre+ +ΔB pre, (5) B I +1 =(f I+,l I+ +Δl I ), (6) B pos +1 = B pos+ +ΔB pos. (7) As for he inerrupion sae N ir, afer he firs frame is played ou, he second frame becomes he curren frame. If he displayed frame is an I frame, N ir+ = 1(b I =0). Here, 1(b I =0)indicaes wheher he base layer of he displayed frame is received. If he displayed frame is he las frame of a GOP, hen N ir+ =0. If he displayed frame is neiher an I frame nor he las frame in a GOP, we have N ir+ = N ir are removed from N ir+. Thus, B I + 1(b pre 1 =0). A he end of he slo, he daa unis in N ir+ which are also in Δu N+1 ir = N ir+ N ir+ Δu. (8) The amoun of video daa in Δu, denoed by R(B, Δu ), can be esimaed according o buffer sae and he rae-qualiy model inroduced in Secion II-C. Specifically, for each daa uni in Δu,we firs deermine wheher i belongs o an I frame or a P frame according o B I and hen esimae he amoun of daa by he rae-qualiy model. The se Δu records he compleely ransmied daa unis up o (f n,l n )h daa uni. However, daa uni (f n+1,l n+1) is only parially received. Denoing he amoun of daa in uni (f n+1,l n+1) by R(B I, Δu ), he amoun of received daa is a leas R(B I, Δu ) and a mos R(B I, Δu )+ R(B I, Δu ). Assuming he physical layer packe lengh is L PHY, here is ΔT R N = L PHY packe ransmissions during a ime slo ΔT. The number of successfully ransmied packes is a leas N l = R(BI,Δu) L PHY and is less han N h = R(BI,Δu)+ R(B I,Δu) L PHY. As assumed in Secion II-D, he channel sae is consan over each slo. Thus, he packe losses are independen wihin each slo. The number of successful packe ransmissions in a slo is disribued binomially. Hence, he sae ransiion probabiliy from S =(C,B,N ir [ Nh 1 ( N P μ (S S +1 )= n =N l n ) o S +1 =(C +1,B +1,N ir +1) is ) p N n (1 p ) n ] P(C C +1 ), (9) where he firs muliplicaive erm is he ransiion probabiliy of he receiver buffer sae from (B,,N ir ) o (B +1,N+1) ir and he second erm is he ransiion probabiliy of he channel sae from C o C +1.

14 14 D. Opimizaion Objecive A he beginning of each ime slo, he firs frame in he window is played ou and he MS-SSIM index is Q(S )= L q l 1 l (S ), (10) l=1 where 1 l (S ) is he indicaor of wheher he lh layer of he displayed frame is received a sae S. The quaniy q l is he rae-qualiy model parameer defined in Secion II-C. Because of he way ha he MS-SSIM index is defined, he quaniy Q(S ) is also bounded in [0, 1]. Our aim is o find he opimal policy μ ( ) which maximizes he ime-average MS-SSIM index, i.e., { N 1 } 1 J μ = lim N N E μ Q(S ). (11) E. Finie Sae Problem Formulaion Using he window defined in Secion III-B, we fix he ransmission policy ouside a finie sae se. The buffer sae space, however, is sill infinie and he sysem sae evolves in his infinie sae space. In he following, we show how o simplify his infinie sae space problem o a finie-sae problem. Noe ha we only need o opimize he policy μ( ) when some of he video daa in he window have no been received or N ir 0. We formally define his finie se S W as follows: =0 S W = {( C, B, N ir) ( C, B, N ir) S, V(B) V W or N ir 0 }, (12) where V W denoes he se of video daa unis in W and V(B) is he se of buffered video daa unis. We define anoher subse of S as follows S W = {( C, B, N ir) ( C, B, N ir) S, V W V(B) and N ir =0 }. (13) For all he saes in S W, he video daa in W and N ir are all received. Noe ha, because he ransmier always ransmis he video daa unis in W and N ir wih higher prioriy, S W and S W form a pariion of sae space S. In oher words, we have S W S W = S and S W S W =. Given a policy μ( ), he sysem sae ransis as a conrolled Markov chain in se S W and as an unconrolled Markov chain in se S W. Because he ransmission rae is finie, he number of saes in S W which can be reached from S W in one sep is also finie. We formally define his se of saes as follows S Δ = {S S S W ; S S W, s.., P μ (S S ) > 0}. (14) Once he sysem moves ino he se S W, he sysem sae his a sae in S Δ and hen says in S W for some ime. During his period, he decoded video qualiy is always ˆQ = L l=1 q l, because all

15 15 he daa unis in W are received. The dynamics of he sysem when i moves ino se S W affecs he performance of he sysem. Generally, he longer i says in S W, he beer he performance is. Alhough he scheduling policy in S W is fixed as described in Secion III-B, he conrol policy in S W deermines how frequenly he sysem sae will hi S W and hus also affecs he sysem performance. In he following, we denoe he sysem under a given policy μ as sysem A μ. Le T μ (S) be he expeced ime spen by A μ in S W afer i eners S W a sae S S Δ. Le P T μ (S S) denoe he probabiliy ha A μ jumps back o S W a sae S S W afer i eners S W a sae S. To find he opimal policy, we define a finie-sae sysem Ãμ as follows: Definiion 1: A sysem Ãμ is called he simplified sysem of he original sysem A μ if i has he following dynamics: 1) The sysem is a semi-markov process over sae space S = S W S Δ. A any sae S S, he visual qualiy is Q(S) as in (10). A any sae in S W, he sysem acs according o he policy μ; 2) When he sysem jumps o a sae S S Δ, i spends T μ (S) slos in S wih video qualiy L l=1 q l for each slo. Then, he sysem ransis o a sae S S W wih probabiliy P T μ (S S). I should be noed ha à μ is no coupled wih he original sysem A μ. I jus shares some properies wih he original sysem (see Fig. 7). The following heorem relaes he visual qualiy under Ãμ and ha of A μ. Theorem 1: If he jump chain of he original sysem A μ is posiive recurren, hen he ime-average MS-SSIM index of A μ is he same as he simplified sysem Ãμ. Proof Skech: If he jump chain is posiive recurren, he jump from S W o S Δ can pariion he Markov process ino i.i.d segmens. We only need o opimize he policy μ o maximize he average qualiy in each segmen. Every segmen consiss of wo consecuive subsegmens. During he firs subsegmen, S S W. In he oher subsegmen, S S W. Because every sae in S W has he same visual qualiy L l=1 q l, we can absrac he firs subsegmen as a single sae wih ransiion probabiliy P T μ (S S). This simplified sysem provides he same average qualiy as he original sysem. For a deailed proof, see he echnical repor [30] Remark The posiive recurren condiion for he jump chain means ha he average hroughpu of he channel is neiher oo large nor oo small relaive o he average daa rae of he video. If he average hroughpu of he channel is very large, he receiver buffer can always buffer enough frames and he dynamic scheduling is unnecessary. If he average channel hroughpu is oo small, he channel canno suppor he video sream and dynamic scheduling canno help eiher. As indicaed by Theorem 1, given any policy μ, he visual qualiy of A μ is he same as Ãμ. Thus,

16 16 we can opimize our policy wih respec o Ãμ which has a finie-sae space, and a sandard policy opimizaion algorihm can by applied. IV. POLICY OPTIMIZATION In he following, we show how o compue he parameers T μ and P T μ. Then, we presen a value ieraion algorihm o find ou he opimal scheduling policy. A. Compuaion of T μ and P T μ Before we can apply an MDP algorihm o opimize he policy, we need o compue T μ (S) and P T μ (S S) for every sae S S Δ and S S W. Boh T μ (S) and P T μ (S S) only involve dynamics of he sysem when he sae S S W. Because he scheduling policy is fixed in S W, we can simplify he sae represenaion furhermore. Le B W be he number of buffered packes ouside he window. Noed ha, when he sysem moves in S W, he sysem always schedules as many enhancemen layer daa unis as possible. In addiion, when he sysem is in sae se S W, we always have N ir =. Hence, B W and f I conain all he informaion abou he buffer sae (B,N ir ). We can furher simplify he sae represenaion (C,B,N ir ) o (C,B W,f I ) when S S W. All he saes in S W correspond o some saes wih B W 0. All he saes in S W correspond o some saes wih B W < 0. When he sysem evolves in S W, a he beginning of a slo, he sae B W firs decreases by ΔBd W (S ) when he curren frame is displayed. Then, he ransmier schedules as many enhancemen layer daa unis as possible. A he end of he slo, B W increases by ΔB W i (S ). Because he quaniy ΔB W (S )=ΔB W i (S ) ΔB W d (S ) only depends on sae S, he sae B W varies like a random walk bu wih Markovian sep-size ΔB W. This process can be described by a quasi-birh-deah process (QBDP). Hence, deermining T μ and P T μ is acually he hiing ime problem of he QBDP. The problem for coninuous ime QBDP was essenially solved in [31, p. 96]. The discree ime case can also be solved similarly. Deails on how o compue T μ and P T μ are found in he echnical repor [30]. B. Deermining Opimal Policy via Value Ieraion Given T μ and P T μ, he opimal policy for an MDP can be deermined for he simplified sysem Ãμ, which is also he opimal policy of A. Le S ini be any sae in S = S W S Δ. The hiing ime o sae S ini can pariion he process ino i.i.d cycles. Opimizing he policy μ( ) in he cycles maximizes he ime-average MS-SSIM index of he sysem. Similar o he derivaion in [3, p. 441], his is equivalen

17 17 o an average-reward maximizaion problem wih sage-reward g(s) τ(s)λ, where λ is he expeced average-reward of each cycle and Q(S) : S S W g(s) = T μ (S) L l=1 q l : S S Δ, 1 : S S W τ(s) = T μ (S) : S S Δ. Le us denoe by h(s) he average reward-o-go in each cycle when he sysem sars a sae S. Then we have he following Bellman s equaion array: h(s) =g(s) τ(s)λ + S S W S P μ (S S)h(S ), (15) where h(s ini )=0. To find he opimal policy, he sandard value ieraion algorihm can be applied [3, p. 430]. V. PERFORMANCE EVALUATION AND NEAR-OPTIMAL POLICY DESIGN In his secion, we firs es he MDP-based scheduling policy by simulaions and show is superioriy o hose scheduling mehods which does no explicily explore he buffer-channel informaion. Then, we propose a simple scheduling policy which presens near-opimal performance. A. Performance evaluaion The proposed dynamic scheduling algorihm was evaluaed on he es sequences of foreman, bus, flower, mobile and Paris [26]. These video sequences were encoded using H.264/SVC reference sofware JSVM [32] ino 3 layers. The GOP lengh was se as L GOP =16. The encoding parameers and rae-qualiy model parameers are lised in Table I. The parameers rl I and rl P are measured in megabis and q l is measured in MS-SSIM index. The quanizaion parameers (QP ) for base layers were chosen such ha he base layer qualiy is abou 0.90 o 0.91 MS-SSIM value which is of moderae bu accepable visual qualiy. The QP s of he enhancemen layers were chosen such ha he hird layer provides MS-SSIM visual qualiy predicion of 0.95 o 0.96 and he bi raes of he wo enhancemen layers are roughly he same. The Lagrangian mulipliers for moion esimaion and mode decision were se as QP 2.

18 18 We employ a 4-sae Markov channel model o es he performance of he proposed scheduling algorihm. The sae ransiion marix is and he seady sae disribuion is P c 5 5 = π= [0.15, 0.60, 0.20, 0.05]. Le us denoe he hroughpu of channel sae (R i,p i ) by r i = R i (1 p i ). The sae parameers R i and p i are configured such ha r 1 <R1 V <r 2 <R2 V <r 3 <R3 V <r 4 in which Ri V is he average video daa rae up o he ih layer. Hence, he channel hroughpu flucuaes among he average rae of each layer. According he seady sae disribuion, he average hroughpu of he channel is higher han he base layer bu no enough o suppor he firs enhancemen layer. We simulaed 40 ransmissions of each sequence over his channel model. To conceal errors, every los frame was reconsruced by copying is preceding frame. To demonsrae he advanages of he proposed algorihm, hree scheduling algorihms were esed over he same channel realizaions. They are summarized as follows: H1 H2 H3 This policy only schedules he base layer. I is he mos conservaive ransmission sraegy. This policy always schedules as many enhancemen layers as possible. I preferenially maximizes he visual qualiy of recen frames. This policy decides dynamically how many enhancemen layers o ransmi. Specifically, when he insananeous channel hroughpu is lower han he average daa rae up o he firs enhancemen layer, he policy behaves like H1. If he channel hroughpu is higher han he average daa rae up o he second enhancemen layer, he policy acs like H2. Oherwise, he policy schedules he video daa up o he firs enhancemen layer. The average visual qualiy measured using MS-SSIM index is shown in Table II. I is observed ha he proposed scheduling algorihm ouperforms all of he oher policies by in MS-SSIM value, which is percepually significan. As can be seen in Table I, o increase he MS-SSIM value by 0.02 approximaely requires doubling he video bi rae. Thus, he proposed scheduling algorihm provides very significan performance improvemens over oher scheduling policies.

19 19 B. Near-opimal scheduling policy design Alhough he MDP-based scheduling policy has he opimal performance among all he policies we consider, he off-line compuaion of MDP policies requires exra sysem resources. This moivae us o design a simple on-line scheduling policy which presens similar performance as he MDP-based policy. The simulaion resuls show ha, by dynamically schedules daa associaed wih differen layers, H3 achieves much beer performance han oher heurisics. Bu, for he sequence Paris, which has large I frames, H3 s performance is much worse. This is mainly because H3 only ransmis video frames sequenially wihou proacively ransmiing daa of laer GOPs. In he following, we propose a scheduling scheme which, similar o he MDP-based policy, no only dynamically schedules daa unis associaed wih differen layers bu also allocae ransmissions for laer GOPs. A he h slo, he proposed policy firs esimaes he amoun of daa which can be sen before he + τh ime slo as D = τ 1 n=1 [r ρ n 1 + r avg (1 ρ n 1 )]. Here, ρ is he subdominan eigenvalue of P c, which represens he emporal correlaion of he channel condiion. Such a correlaion parameer could be easily measured. The quaniy r is he hroughpu of he curren slo and r avg is he average hroughpu of he channel. Again, hese parameers can be measured. If b I =0, correcly sending I o he receiver is criical and hus we se τ = f I.Ifb I > 0, we care abou he ransmissions wihin he coherence ime and hus we se τ = 1/ ln(1/ρ), where 1/ ln(1/ρ) is roughly he relaxaion ime of channel variaions. Le D l be he amoun of unreceived daa conained for he firs l layers of he nex τ frames. The proposed scheduling policy operaes as follows: If D <D 2, he policy only schedules he base layer daa unis; If D 2 D <D 3, he policy only schedules he firs wo layers; Oherwise, he policy schedules daa unis from all he hree layers. If D <D 1, he policy schedules as much of base layer daa for I as possible. If D 1 <D <D 2, he policy schedules up o 50% of he ransmissions for I. Oherwise, he scheduler policy ransmis he frames sequenially wihou proacively ransmi daa for I. The parameers of he proposed policy, i.e., ρ, r and r avg can be easily measured and his policy is very simple o implemen. The performance of his policy was esed over he simulaed Markov channel models wih differen emporal correlaion parameer ρ. The simulaion resuls are summarized in Table III and Table IV. For mos esed sequences, he gap beween he proposed policy and he MDP-based policy ranges from o 0.01.

20 20 VI. CONCLUSIONS We have developed dynamic scheduling for efficien scalable video ransmission in wireless channels. By modeling he wireless channel as a Markov chain, an infinie-horizon average-reward maximizaion formulaion is proposed o maximize he visual qualiy prediced by MS-SSIM index. To reduce he sae space o a finie one, we employ a window o fix he scheduling policy when all he daa wihin he window are received. I is shown ha he scheduling policy opimizaion problem is equivalen o finding he opimal conrol policy for a conrolled semi-markov process over a finie-sae space. Simulaion resuls demonsrae he superioriy of he scheduling policy obained by he proposed MDP-based formulaion. Furher, a simple scheduling policy is proposed and presens near-opimal performances. REFERENCES [1] H. Schwarz, D. Marpe, and T. Wiegand, Overview of he scalable video coding exension of he H.264/AVC sandard, IEEE Trans. Circuis Sys. Video Technol., vol. 17, no. 9, pp , Sep [2] W. Li, Overview of fine granulariy scalabiliy in MPEG-4 video sandard, IEEE Trans. Circuis Sys. Video Technol., vol. 11, no. 3, pp , Mar [3] D. Bersekas, Dynamic Programming and Opimal Conrol, 3rd ed. Ahena Scienific, 2005, vol. 2. [4] Z. Wang and A. C. Bovik, Mean squared error: Love i or leave i? A new look a signal fideliy measures, IEEE Signal Process. Mag., vol. 26, no. 1, pp , Jan [5] B. Girod, Wha s wrong wih mean-squared error, Digial Images and Human Vision (A. B. Wason, ed.), pp , [6] A. M. Eskicioglu and P. S. Fisher, Image qualiy measures and heir performance, IEEE Trans. Commun., vol. 43, no. 12, pp , Dec [7] Z. Wang, E. P. Simoncelli, and A. C. Bovik, Muliscale srucural similariy for image qualiy assessmen, in Conference Record of he Thiry-Sevenh Asilomar Conference on Signals, Sysems and Compuers, vol. 2, Nov. 2003, pp [8] K. Seshadrinahan, R. Soundararajan, A. C. Bovik, and L. K. Cormack, Sudy of subjecive and objecive qualiy assessmen of video, IEEE Trans. Image Process., vol. 19, no. 6, pp , Jun [9] M. Podolsky, S. McCanne, and M. Veerli, Sof ARQ for layered sreaming media, Technical Repor, Compuer Science Division, Universiy of California, Berkeley, vol. UCB/CSD , [10] P. A. Chou and Z. Miao, Rae-disorion opimized sreaming of packeized media, IEEE Trans. Mulimedia, vol. 8, no. 2, pp , Apr [11] J. Chakareski, P. A. Chou, and B. Aazhang, Compuing rae-disorion opimized policies for sreaming media o wireless cliens, in Proceedings of Daa Compression Conference, 2002, pp [12] Y. Zhang, F. Fu, and M. van der Schaar, On-line learning and opimizaion for wireless video ransmission, IEEE Trans. Signal Process., vol. 58, no. 6, pp , Jun [13] F. Fu and M. van der Schaar, A new sysemaic framework for auonomous cross-layer opimizaion, IEEE Trans. Veh. Technol., vol. 58, no. 4, pp , May [14] C. Chen, R. W. Heah, A. C. Bovik, and G. de Veciana, Adapive policies for real-ime video ransmission: a Markov decision process framework, in 18h IEEE Inernaional Conference on Image Processing, Sep

21 21 [15] Y. Li, A. Markopoulou, J. Aposolopoulos, and N. Bambos, Conen-aware playou and packe scheduling for video sreaming over wireless links, IEEE Trans. Mulimedia, vol. 10, no. 5, pp , Aug [16] J. Cabrera, A. Orega, and J. I. Ronda, Sochasic rae-conrol of video coders for wireless channels, IEEE Trans. Circuis Sys. Video Technol., vol. 12, no. 6, pp , Jun [17] J. W. Huang, H. Mansour, and V. Krishnamurhy, A dynamical games approach o ransmission-rae adapaion in mulimedia WLAN, IEEE Trans. Signal Process., vol. 58, no. 7, pp , Jul [18] J. Filar and K. Vrieze, Compeiive Markov Decision Processes. New York: Springer-Verlag, [19] F. Fu and M. Van Der Schaar, A sysemaic framework for dynamically opimizing muli-user wireless video ransmission, IEEE J. Sel. Areas Commun., vol. 28, no. 3, pp , Apr [20] Q. Zhang and S. A. Kassam, Finie-sae Markov model for Rayleigh fading channels, IEEE Trans. Commun., vol. 47, no. 11, pp , Nov [21] H.-P. Lin and M.-J. Tseng, Two-layer mulisae Markov model for modeling a 1.8 GHz narrow-band wireless propagaion channel in urban Taipei ciy, IEEE Trans. Veh. Technol., vol. 54, no. 2, pp , Mar [22] M. H. Pinson and S. Wolf, A new sandardized mehod for objecively measuring video qualiy, IEEE Trans. Broadcas., vol. 50, no. 3, pp , Sep [23] D. M. Chandler and S. S. Hemami, VSNR: A wavele-based visual signal-o-noise raio for naural images, IEEE Trans. Image Process., vol. 16, no. 9, pp , Sep [24] A. K. Moorhy and A. C. Bovik, Visual imporance pooling for image qualiy assessmen, IEEE J. Sel. Topics Signal Process., vol. 3, no. 2, pp , Apr [25] Z. Wang and Q. Li, Video qualiy assessmen using a saisical model of human visual speed percepion, Journal of he Opical Sociey of America, vol. A 24, B61-B69, Jul [26] Tes sequences [On line]. Available: hp://race.eas.asu.edu/yuv/. [27] H. S. Wang and P.-C. Chang, On verifying he firs-order Markovian assumpion for a Rayleigh fading channel model, IEEE Trans. Veh. Technol., vol. 45, no. 2, pp , May [28] T. Su, H. Ling, and W. J. Vogel, Markov modeling of slow fading in wireless mobile channels a 1.9 GHz, IEEE Trans. Anennas Propag., vol. 46, no. 6, pp , Jun [29] C. C. Tan and N. C. Beaulieu, On firs-order Markov modeling for he Rayleigh fading channel, IEEE Trans. Commun., vol. 48, no. 12, pp , Dec [30] Technical Repor [On line]. Available: hps://webspace.uexas.edu/cc39488/pdf/repor.pdf. [31] M. Neus, Marix-Geomeric Soluions in Sochasic Models: An Algorihm Approach. The Johns Hopkins Universiy Press, [32] J. Reichel, S. Schwarz, and W. M, Join scalable video model 11 (JSVM 11), Join Video Team, Doc. JVT-X202, Jul

22 22 TABLE I THE ENCODING PARAMETERS AND RATE-QUALITY MODEL PARAMETERS OF THE TESTED SEQUENCES. sequences Layer 1 (base layer) Layer 2 Layer 3 QP r I 1 r P 1 q 1 QP r I 2 r P 2 q 2 QP r I 3 r P 3 q 3 foreman bus flower mobile Paris TABLE II THE PERFORMANCE OF THE MDP-BASED POLICY. Paris mobile flower bus foreman MDP-based Policy H H H TABLE III THE PERFORMANCE OF THE NEAR-OPTIMAL POLICY. ρ = AND τ =2. Paris mobile flower bus foreman MDP-based Policy Near-opimal Policy TABLE IV THE PERFORMANCE OF THE NEAR-OPTIMAL POLICY. ρ = AND τ =4. Paris mobile flower bus foreman MDP-based Policy Near-opimal Policy

23 23 Channel and Receiver Buffer Sae Requess Video Server Requesed Daa Scheduler Transmier Markov Channel Receiver Fig. 1. Dynamic Scheduling for Video Transmission -1,3 0,3 1,3 2,3 3,3 4,3-1,2 0,2 1,2 2,2 3,2 4,2-1,1 0,1 1,1 2,1 3,1 4,1 Curren Frame Expired Frames Unexpired Frames Fig. 2. Encoder Predicion Srucure when L =3