Encoder and Decoder Side Global and Local Motion Estimation for Distributed Video Coding

Transcription

1 Encoder and Decoder Sde Global and Local Moton Estmaton for Dstrbuted Vdeo Codng Frederc Dufaux, Touradj Ebrahm MultMeda Sgnal Processng Group Ecole Polytechnque Fédérale de Lausanne (EPFL) CH-1015 Lausanne, Swtzerland Abstract In ths paper, we propose a new Dstrbuted Vdeo Codng (DVC) archtecture where moton estmaton s performed both at the encoder and decoder, effectvely combnng global and local moton models. We show that the proposed approach mproves sgnfcantly the qualty of Sde Informaton (SI), especally for sequences wth complex moton patterns. In turn, t leads to rate-dstorton gans of up to 1 db when compared to the state-of-the-art DISCOVER DVC codec. I. INTRODUCTION Dgtal meda content s omnpresent n nowadays nformaton socety where fast and effcent access to nformaton s paramount. Ths evoluton has been possble thanks to the rapd and remarkable progresses n vdeo codng technologes. Over the last two decades, standardzaton efforts n MPEG and ITU-T have led to new technologes wth ever mprovng codng performance. The resultng standards adopted throughout the years have been a cornerstone of the dgtal age experence, and the state-of-the-art H.264/AVC [1] s the latest outcome of ths process. In all MPEG and ITU-T vdeo codng schemes, the encoder s n charge of explotng the source statstcs to acheve the most effcent compresson, resultng n an asymmetrc computatonal load where the encoder s sgnfcantly more complex than the decoder. For nstance, the compresson gans acheved by H.264/AVC are the result of an extensve analyss at the encoder n order to better represent the vdeo sgnal. Namely, the encoder can choose from an ever growng number of codng modes, mprovng codng effcency at the cost of much ncreased encodng complexty. Whle ths ncremental progresson has brought sgnfcant achevements n the past, one may wonder for how long ths path wll contnue to produce mproved performance. It s therefore a good tme to reflect on the feld of vdeo compresson. More specfcally, t s legtmate to ask whether we are reachng a plateau n performance wth the current standard archtecture, or whether they are new promsng theores and technologes whch may brng sgnfcant breakthroughs n the near future. Dstrbuted Vdeo Codng (DVC) has more recently emerged as a new codng paradgm. DVC fnds ts theoretcal MMSP 10, October 4-6, 2010, Sant-Malo, France /10/$ IEEE. foundaton n the Slepan-Wolf [2] and Wyner-Zv [3] theorems. These results remaned unexploted untl the early 2000 s when the frst practcal DVC schemes have been proposed [4][5]. The DVC paradgm presents a number of advantages: flexble parttonng of the computatonal complexty between the encoder and decoder, error reslence, codec-ndependent scalablty and mult-vew codng. DVC has ganed a lot of nterest over the last few years. Overvews of recent developments are presented n [6][7]. Most of the research actvtes on DVC have so far focused on outperformng conventonal codng solutons under the specfc constrant of very low encodng complexty, ths feature beng appealng for a number of upcomng up-lnk applcatons. For nstance, the DISCOVER DVC codec [8] has reported some of best codng performance results at the moment. In partcular, t consstently outperforms H.264/AVC Intra. For scenes wth smple and unform moton, t even outperforms H.264/AVC No Moton. However, for more complex scenes, ts performance typcally remans lower than H.264/AVC No Moton. Nonetheless, the encodng complexty advantage offered by DVC may be very shortlved due to exponentally ncreasng computng power as predcted by Moore s law. In summary, substantal gans have been obtaned wth conventonal codng by contnuously addng more effcent analyss at the encoder, and hence resultng n a complex encoder smple decoder arrangement. Conversely, new DVC codec desgns have manly focused on the opposte extreme, proposng advanced tools at the decoder, creatng a smple encoder complex decoder framework. Challengng current thnkng and as an avenue towards further codng advances, t has been proposed n [9] to develop a new class of codec archtecture where both encoder and decoder are peers of equal mportance and share the workload burden. Thanks to Moore s law, whch has been a drvng force of technologcal changes n the late 20 th and early 21 st centures, such a complex encoder complex decoder archtecture may quckly become acceptable for varous applcaton domans, as long as t provdes a compellng value proposton and enable new products and servces. A smlar path s taken n [10], whch consders H.264/AVC and proposes to perform moton estmaton both at the encoder and decoder for the codng of B frames. The

2 scheme saves on the transmsson of moton vectors and often acheves better predcton, leadng to codng gans. In [11][12], DVC schemes are presented dscardng the low encodng complexty constrant and performng moton estmaton both at the encoder and decoder. Ths paper follows a smlar drecton. Gven that the effectveness of DVC strongly depends on the correlaton between the Sde Informaton (SI) and the Wyner-Zv (WZ) frame, we propose to perform moton estmaton both at the encoder and decoder, combnng global and local moton models. We show that the proposed archtecture leads to better SI, resultng n codng gans. Although we do not explore the subject n ths paper, t s mportant to underlne that the proposed archtecture also preserves the strong error reslence feature of DVC. Ths paper s structured as follow. Related works are frst revewed n Sec. II. We then present the overall proposed system n Sec. III. The process to generate SI s descrbed n more detals n Sec. IV. Next, the performance of the proposed approach s assessed n Sec. V. Fnally, Sec. VI draws concluson and outlnes future perspectves. II. RELATED WORK In DVC, the qualty of the SI to approxmate the WZ frame has a great mpact on codng effcency. Most commonly, SI s estmated at the decoder by lnear nterpolaton of moton vectors between consecutve reference frames. Varous technques have been proposed n order to mprove SI generaton. Spatal smoothng and moton vectors refnement has been proposed n [13]. By provdng moton felds closer to the true moton n the scene, the method results n better predcton. Moton compensated forward and backward extrapolaton s ntroduced n [14], generatng two SI whch are exploted n the decodng process. Moton estmaton wth sub-pxel accuracy s consdered n [15]. In [16], decoded btplanes of the WZ frame are exploted to refne moton vectors. Several nterpolaton modes are also ntroduced. Moton compensated temporal nterpolaton s teratvely mproved n [17] based on a partally decoded WZ frame. In [18], an teratve technque s proposed based on multple SI and moton refnement. Based on error probablty, the turbo decoder then determne whch SI to select for each block. In the same way, a partally decoded WZ frame s used to mprove SI generaton n [19]. An enhanced moton compensated temporal nterpolaton method s also ntroduced. In [11], a pxel doman DVC scheme s ntroduced, combnng low-complexty encoder-sde btplane moton estmaton wth decoder-sde moton compensated frame nterpolaton. Improvements are shown for sequences wth fast and complex moton. Fnally, [12] presents a DVC scheme where both the encoder and decoder cooperate to perform moton estmaton. Results show that t reduces overall computatonal complexty whle mprovng codng effcency. III. PROPOSED SYSTEM In ths paper, we more specfcally consder the DISCOVER DVC codec [8]. Input frames are frst splt nto Group of Pctures (GOP). Key frames, correspondng to the frst frame of each GOP, are conventonally encoded usng H.264/AVC [1]. In turn, WZ frames undergo a DCT transform followed by unform quantzaton. The quantzed values are then splt nto btplanes whch go through a Low-Densty Party-Check Accumulate (LDPCA) encoder. At the decoder, SI approxmatng the WZ frames s generated from the prevously decoded reference frames. SI s then used n the LDPCA decoder, along wth the party bts of the WZ frames requested va a feedback channel, n order to reconstruct the btplanes, and subsequently the decoded vdeo sequence. Hereafter, wthout loss of generalty, we more specfcally consder a GOP sze of 2. Namely, odd and even frames are coded as key and WZ frames respectvely. In most DVC schemes, SI s generated at the decoder by moton compensated nterpolaton or extrapolaton of prevously decoded reference frames. Henceforth, t s very challengng to estmate an SI closely approxmatng the WZ frame, especally for scenes wth complex moton. Moreover, such schemes assume that moton remans unform n-between the reference frames, although ths hypothess often does not hold for complcated scenes. In contrast, n the proposed DVC archtecture, moton estmaton and compensaton s performed both at the encoder and decoder sdes. Moreover, we consder the combnaton of global and local moton representatons. The resultng codec s llustrated n Fg. 1. A. Overvew of SI Processes at the Encoder and Decoder The followng operatons are performed at the encoder sde. Frst, frame-based global moton estmaton and compensaton s performed between the current WZ frame and prevous and next decoded key frames. It results n a frst SI, denoted GMC SI. In parallel, block-based local moton estmaton and compensaton s appled between the prevous and next decoded key frames, resultng n the MCTI SI. Fnally, for each 16x16 MacroBlock (MB) of the WZ frame, the optmal predcton mode s determned by selectng the best approxmaton between GMC and MCTI SI. The global moton parameters used n GMC, as well as the optmal MB mode selectons, are transmtted to the decoder as supplementary nformaton. At the decoder sde, the global moton parameters are used to generate the GMC SI. Conversely, local moton estmaton and compensaton s performed agan n order to produce the MCTI SI. Both SI are then fused usng the optmal MB mode decsons n order to create the fnal SI.

3 Fg. 1. Proposed DVC archtecture. As we wll show hereafter, ths DVC archtecture allows mprovng the SI generaton process by explotng both global and local moton representatons and selectng optmal MB mode decsons. Moreover, the btrate needed to transmt supplementary nformaton, namely the global moton parameters and the MB mode selectons, remans margnal. Addtonally, the proposed archtecture preserves one of the essental features of DVC, namely the absence of a predcton loop. Ths prevents drfts n the presence of transmsson errors and, along wth the bult-n jont source-channel codng structure, mples strong error reslence. However, the rate-dstorton performance gan s obtaned at the expense of ncreased computatonal complexty at the encoder sde, hence breakng the low encodng complexty target commonly assumed n DVC research works. IV. SIDE INFORMATION GENERATION We now descrbe n more detals the SI generaton process takng place both n the encoder and decoder. A. Global Moton Compensaton Usng GMC, the global moton wthn the scene s modeled by a perspectve transform, also known as homography. Ths model s vald whenever the scene can be approxmated by a planar surface. More specfcally, two transforms H 1 and H 2 are computed between the WZ and prevous key frame on the one hand, and between the WZ and next key frame on the other hand, as llustrated n Fg. 2. The GMC SI s then obtaned by averagng both backward and forward predctons. Fg. 2. Sde nformaton wth the proposed GMC. Note that, as the process s performed at the encoder, the WZ frame to be approxmated s avalable and can be drectly exploted n the mnmzaton process, as explctly detaled hereafter. More precsely, the perspectve transform s defned as a0 + a2x + a3 y a1 + a4x + a5 y u = v = a6x + a7 y + 1 a6x + a7 y + 1 where (u,v ) denotes the pxel locaton n the WZ frame, (x,y ) the correspondng poston n the prevous or next key frame, and a 0, a 1,, a 7 the parameters of the transform. The moton parameters are estmated by the global moton estmaton technque ntroduced n [20]. More specfcally, they are obtaned by mnmzng the expresson E = N = 1 ρ ( Iˆ( u, v ) I( x, y )

4 where Î(u,v ) and I(x,y ) represent the mage pxel values of the WZ frame and prevous or next key frame. In order to ncrease robustness to outlers, a truncated quadratc robust estmator s chosen for the metrc ρ. The summaton s carred over N pars of pxels wthn the mage boundares. Ths nonlnear problem s solved usng the Levenberg-Marquardt gradent descent algorthm to teratvely estmate the parameters. For each WZ frame, two set of parameters are calculated at the encoder, defnng the backward and forward transform respectvely. These global moton parameters are then transmtted to the decoder. An effcent way to represent ths nformaton conssts n transmttng the dfferental coordnates of four ponts n the mage defnng the perspectve transform [21]. Hence, the assocated btrate overhead remans margnal. B. Moton Compensated Temporal Interpolaton An alternatve SI s also generated by the conventonal MCTI [13]. In ths case, moton estmaton s frst performed between the prevous and next decoded key frames. More specfcally, block-based moton vectors are computed by block matchng. Spatal moton smoothng [13] s then appled n order to further mprove performance. Unlke most DVC schemes, n the proposed approach, MCTI s appled both n the encoder and the decoder (see Fg. 1). On the one hand, ths archtecture allows selectng optmal MB modes at the encoder sde for SI generaton. On the other hand, moton vectors, whch would otherwse represent a sgnfcant btrate overhead, do not need to be transmtted. C. Mode Selecton In prevously proposed DVC codecs, the SI s most commonly generated solely at the decoder sde, greatly lmtng ts effectveness. In the proposed archtecture, two SI are avalable, based on GMC and MCTI respectvely. By performng mode selecton at the encoder, the WZ frame to be approxmated s avalable and can be exploted to determne the optmal predctor. Straghtforwardly, the latter s determned by selectng the best approxmaton between GMC and MCTI SI. More precsely, the mode selecton s made on a MB bass, where each MB corresponds to 16x16 pxels. One bt per MB s transmtted n order to sgnal the mode. On the one hand, ths allows for local adaptaton of the predcton, talored to the scene content. On the other hand, the resultng btrate overhead to transmt the MB decsons s neglgble. For nstance, for a vdeo at QCIF resoluton and 15 fps wth a GOP of 2, the supplementary btrate s only 99 bts per WZ frame or kbps. V. EXPERIMENTAL RESULTS Performance of the proposed approach s now assessed, takng the DISCOVER DVC scheme as reference [8]. The four test sequences Foreman, Soccer, Coastguard and Hall Montor, at QCIF resoluton and 15 fps, are used for smulatons. Note that only the lumnance component s processed. We adopt the same test condtons as n [22]. A. Sde Informaton The qualty of the SI resultng from the proposed approach s frst assessed. For ths purpose, we compare the SI obtaned usng three approaches: GMC, MCTI, and combned GMC- MCTI, as descrbed n Sec. IV.A, IV.B and IV.C respectvely. Fg. 3 shows PSNR as a functon of the frame number for Foreman (correspondng to the rate-dstorton pont wth quantzaton matrx Q=8 [22]). Clearly, the proposed combned GMC-MCTI leads to hgher SI PSNR. Straghtforwardly, the gan s larger on the part of the scene exhbtng swft camera moton. Fg. 3. Sde Informaton PSNR for Foreman. Table I shows the correspondng SI average PSNR values for the four test sequences (correspondng to the ratedstorton pont wth quantzaton matrx Q=8 [22]). We observe that the proposed combned GMC-MCTI consstently outperforms conventonal MCTI. The gan s more sgnfcant for sequences wth complex moton such as Foreman and Soccer. In opposton, the gan s neglgble for Hall Montor, as the scene s manly statc and n ths case MCTI s performng very well. It can also be notced that GMC alone s usually less effcent than MCTI, except for Soccer. Clearly, Table I shows that the performance mprovement s the result of the optmal combnaton of GMC and MCTI. TABLE I SIDE INFORMATION AVERAGE PSNR Sequence GMC MCTI combned GMC-MCTI Foreman Soccer Coastguard Hall Fg. 4 shows the vsual qualty of the SI obtaned by MCTI and the proposed combned GMC-MCTI for a sample frame of Soccer. In ths example, MCTI leads to very obvous dstortons, due to the complexty of the moton n the scene whch makes truthful nterpolaton very challengng. In contrast, the vsual qualty s dramatcally enhanced when usng the proposed technque. Ths mprovement results from

5 the encoder sde moton estmaton whch s capable of accurately modelng the moton n the scene. a) b) VI. CONCLUSION Most of the research actvtes on DVC have so far focused on outperformng conventonal codng solutons under the specfc constrant of very low encodng complexty. In ths paper, we dscard ths constrant and put forward a complex encoder complex decoder desgn. More specfcally, we propose a new DVC archtecture where moton estmaton s performed both at the encoder and decoder. Furthermore, we effectvely combne global and local moton representatons. Expermental results show that the proposed approach mproves sgnfcantly the qualty of SI when compared to common MCTI. In terms of rate-dstorton, we report gans of up to 1 db when compared to the state-ofthe-art DISCOVER DVC codec. Improvements are more mportant for sequences wth complex and fast moton. As future research actvtes, we wll further explore DVC schemes wth complex encoder complex decoder characterstcs, wth the goal to outperform conventonal MPEG or ITU-T vdeo codng schemes. a) Fg. 4. Sde Informaton vsual qualty for Soccer : a) MCTI (PSNR = db), b) combned GMC-MCTI (PSNR = db). B. Rate Dstorton Fnally, we evaluate the rate-dstorton performance of the proposed SI generaton scheme. More precsely, we compare the two cases: DISCOVER [8] usng common MCTI [13], and the proposed DVC archtecture wth combned GMC-MCTI. We also show the performance of two H.264/AVC schemes, H.264/AVC Intra and H.264/AVC no moton, commonly used as reference n DVC. In the former varant, all frames are Intra coded and no temporal correlaton s exploted. In the latter varant, an IB BI structure s used to explot temporal redundancy, but wthout performng moton estmaton (.e. all moton vectors are zero). The rate-dstorton results are shown n Fg. 5 for the three sequences Foreman, Soccer and Coastguard. The proposed technque consstently outperforms DISCOVER. Agan, the mprovement s more mportant for Foreman and Soccer whch exhbt fast moton. For these two sequences, the gan reaches up to 1 db n the hgher btrate range, but remans around 0.5 db n the lower btrate range. For Foreman, the proposed approach s now always outperformng H.264/AVC Intra, whch s not the case wth DISCOVER, and s gettng closer to H.264/AVC no moton. For Soccer, despte the performance mprovement, the proposed scheme remans notably nferor to both H.264/AVC varants. Fnally, both DVC approaches outperform conventonal schemes for Coastguard, as the unform moton n ths scene s well captured n the SI generaton process. b) c) Fg. 5. Rate-dstorton performance evaluaton comparng the proposed combned GMC-MCTI, DISOCVER wth MCTI, as well as H.264/AVC Intra and H.264/AVC no moton: a) Foreman, b) Soccer, c) Coastguard.

6 ACKNOWLEDGMENT The authors would lke to acknowledge the use of the DISCOVER codec, a software whch started from the IST WZ software developed at the Image Group from Insttuto Superor Técnco (IST) of Lsbon by Catarna Brtes, João Ascenso and Fernando Perera. REFERENCES [1] T. Wegand, G.J. Sullvan, G. Bjøntegaard, and A. Luthra, Overvew of the H.264/AVC Vdeo Codng Standard, IEEE Trans. on Crcuts and Systems for Vdeo Technology, vol. 13, no. 7, July [2] J. Slepan and J. Wolf, Noseless Codng of Correlated Informaton Sources, IEEE Trans. on Informaton Theory, vol. 19, no. 4, July [3] A. Wyner and J. Zv, The Rate-Dstorton Functon for Source Codng wth Sde Informaton at the Decoder, IEEE Trans. on Informaton Theory, vol. 22, no. 1, January [4] R. Purt and K. Ramchandran, PRISM: A new robust vdeo codng archtecture based on dstrbuted compresson prncples, n Proc. Allerton Conference on Communcaton, Control and Computng, Allerton, IL, USA, October [5] A. Aaron, R. Thang, and B. Grod, Wyner-Zv codng of moton vdeo, n Proc. Aslomar Conference on Sgnals and Systems, Pacfc Grove, CA, USA, November [6] C. Gullemot, F. Perera, L. Torres, T. Ebrahm, R. Leonard and J. Ostermann, Dstrbuted Monovew and Multvew Vdeo Codng, IEEE Sgnal Processng Magazne, vol. 24, no. 5, [7] F. Dufaux, W. Gao, S. Tubaro, A. Vetro, Dstrbuted Vdeo Codng: Trends and Perspectves, EURASIP Journal on Image and Vdeo Processng (Specal ssue on DVC), vol. 2009, [8] X. Artgas, J. Ascenso, M. Dala, S. Klomp, D. Kubasov, M. Ouaret, The Dscover Codec: Archtecture, Technques and Evaluaton, n Proc. of Pcture Codng Symposum, Lsboa, Portugal, November [9] F. Perera, Vdeo Compresson: Stll Evoluton or Tme for Revoluton?, Keynote talk at the 10th Internatonal Workshop on Image Analyss for Multmeda Interactve Servces (WIAMIS), London, UK, May [10] S. Klomp, M. Munderloh, Y. Vats, J. Ostermann, Decoder-Sde Block Moton Estmaton for H.264 / MPEG-4 AVC Based Vdeo Codng, n Proc. IEEE Internatonal Symposum on Crcuts and Systems, Tape, Tawan, May [11] T. Clerckx, A. Munteanu, J. Cornels, and P. Schelkens, Dstrbuted vdeo codng wth shared encoder/decoder complexty, n Proc. IEEE Internatonal Conference on Image Processng, San Antono, TX, September [12] H. Chen and E. Stenbach, Flexble dstrbuton of computatonal complexty between the encoder and the decoder n dstrbuted vdeo codng, n Proc. IEEE Internatonal Conference on Multmeda & Expo, Hannover, Germany, June [13] J. Ascenso, C. Brtes and F. Perera, Improvng Frame Interpolaton wth Spatal Moton Smoothng for Pxel Doman Dstrbuted Vdeo Codng, n Proc. 5th EURASIP Conference on Speech and Image Processng, Multmeda Communcatons and Servces, Smolence, Slovak Republc, July [14] K. Msra, S. Karande, H. Radha, Mult-hypothess based Dstrbuted Vdeo Codng usng LDPC codes, n Proc. Allerton Conference on Commun, Control and Computng, Allerton, IL, USA, September [15] L. We, Y. Zhao, A. Wang, Improved Sde-Informaton n Dstrbuted Vdeo Codng, n Proc. Internatonal Conference on Innovatve Computng, Informaton and Control, Bejng, Chna, August- September [16] J. Ascenso, C. Brtes, F. Perera, Moton Compensated Refnement for Low Complexty Pxel Based Dstrbuted Vdeo Codng, n Proc. IEEE Internatonal Conference on Advanced Vdeo and Sgnal Based Survellance, Como, Italy, September [17] X. Artgas, L. Torres, Iteratve Generaton of Moton-Compensated Sde Informaton for Dstrbuted Vdeo Codng, n Proc. IEEE Internatonal Conference on Image Processng, Genova, Italy, September [18] W. A. R. J. Weerakkody, W. A. C. Fernando, J. L. Martnez, P. Cuenca, F. Qules, An Iteratve Refnement Technque for Sde Informaton Generaton n DVC, n Proc. IEEE Internatonal Conference on Multmeda and Expo, Bejng, Chna, July [19] S. Ye, M. Ouaret, F. Dufaux and T. Ebrahm, Improved Sde Informaton Generaton wth Iteratve Decodng and Frame Interpolaton for Dstrbuted Vdeo Codng, n Proc. IEEE Internatonal Conference on Image Processng, San Dego, CA, October [20] F. Dufaux and J. Konrad, Effcent, Robust, and Fast Global Moton Estmaton for Vdeo Codng, IEEE Trans. on Image Processng, vol. 9, no.3, March [21] T. Ebrahm, F. Dufaux, and Y. Nakaya, MPEG-4 Natural Vdeo II, n A. Pur and T. Chen, edtors, Multmeda Systems, Standards, and Networks, Marcel Dekker, 2000, ch. 9, pp [22]