A Perceptual Based Motion Compensation Technique for Video Coding

Transcription

1 A Perceptual Based Moton Compensaton Technque for Vdeo Codng Amn Bantaleb, Sad ader-esfahan, Alreza asr Avanak School of Electrcal and Computer Engneerng Unverst of Tehran, Tehran , Iran Abstract Moton estmaton s one of the mportant procedures n the all vdeo encoders. Most of the complet of the vdeo coder depends on the complet of the moton estmaton step. The orgnal moton estmaton algorthm has a remarkable complet and therefore man mprovements were proposed to enhance the crude verson of the moton estmaton. The basc dea of man of these works were to optmze some dstorton functon for mean squared error (MSE) or sum of absolute dfference (SAD) n block matchng But t s shown that these metrcs do not conclude the qualt as t s, on the other hand, the are not compatble wth the human vsual sstem (HVS). In ths paper we eplored the usage of the mage qualt metrcs n the vdeo codng and more specfc n the moton estmaton. We have utlzed the perceptual mage qualt metrcs nstead of MSE or SAD n the block based moton estmaton. Three dfferent metrcs have used: structural smlart or SSIM, comple wavelet structural smlart or CW-SSIM, vsual nformaton fdelt or VIF. Epermental results showed that usage of the qualt crterons can mprove the compresson rate whle the qualt remans f and thus better qualt n coded vdeo at the same bt budget. Kewords-Vdeo Codng; Structural Smlart; Moton Compensaton; Vsual Informaton Fdelt. I. ITRODUCTIO In the few past ears, the vdeo compresson and codng paradgm has been under a lot of attenton and the subject of etensve research. Applcatons such as vdeo broadcastng, vdeo transmsson over the networks and vdeo streamng on the nternet as well as the move ndustr, eplored the necesst of the compresson of the vdeo sgnals further and further. In the orgnal H.6 vdeo codng protocol the man part of the encoder s moton estmaton stage. A smple block matchng s performed on the current frame (or reference mage) and the net frame (or target mage). SAD s used as the dstorton measure for block matchng. e. the block from the net frame wth the smallest SAD between the reference block s selected as the best match. There s usuall a search range for the blocks to be compared to the reference block. In the smplest case the search range can be a block wth sde equal to 3 tmes bgger than the reference block [], [], [3]. Fgure shows a tpcal dagram of the moton estmaton. After the ntroducton of H.6, researchers had consumed much effort to mprove the performance of H.6 vdeo coder. Several mprovements were added to the crude verson of the moton estmaton for mprovng the compresson rato and reducng the computatonal complet [3]-[]. H.63 added more features such as supportng more pcture formats, dfferent GOB (group of blocks) structure, unrestrcted moton vectors, P and B frames mode, advanced predcton mode and also half-pel moton estmaton whch allows the resoluton of moton vectors to be half pel [6]. There s a bref comparson between H.6 and H.63 n [3]. H.63+ and H.63++ were ntroduced after H.63 and H.64 and MPEG4 are the last verson of the coders wth man optonal modes [7]. Block matchng moton estmaton algorthms such as the three step search [0], [] and the damond search [3], [4] algorthms are currentl beng used n vdeo codng schemes as alternatves to full search algorthms. Man mprovements to reduce the complet of these methods proposed. In the almost all of the mentoned vdeo codng standards, there s a lack of consderaton of the human percepton sstem n the moton estmaton procedure. Usng a SAD or MSE crteron would perhaps lead to less computatonal complet but t s not optmal n case of codng rate or qualt of the compressed vdeo. In the moton compensaton of H.64 the best matchng blocks and the best predcton modes are chosen b Lagrange cost functon whose dstorton functon s SAD. It has been shown that MSE s not a relable qualt crteron when we are dealng wth perceptuall mportant sgnals such as mages and vdeos [5]-[8]. A. C. Bovc and hs team proposed new qualt metrcs for mages and vdeos such as SSIM, CW-SSIM and VIF that are based on human vsual sstem. An overvew of mage qualt assessment (IQA) can be found n [6]. In ths paper, we utlzed the usage of the mage qualt metrcs nstead of SAD or MSE n the moton estmaton and compensaton of the vdeo. We used SSIM [7], CW-SSIM [8] and VIF [5] as qualt crteron n moton estmaton and eplored the effect and trade-offs of utlzaton of these metrcs. The rest of ths paper s organzed as follows: secton II s a bref ntroducton to mage qualt metrcs commonl used n qualt assessment contet. Secton III contans the specfcatons of the dataset. Secton IV s dedcated to eplanng our method and secton V descrbes the smulaton results whle secton VI s concluson and secton VII contans the references.

2 mages and. Local structural smlart s usuall formulated as: S( ) l( ). c( ). s( ) C C ( ).( C C C3 ).( C 3 ) (4) Fgure. Moton vector & moton compensaton II. UTILIZED PERCEPTUAL QUALITY METRIC As mentoned before, t s proven that wthout modelng the human vson sstem one cannot ntroduce an acceptable qualt crteron for mages and ths was the ke dea to the generaton of HVS based qualt metrcs. In ths secton we brefl ntroduce man crterons for calculate the qualt of mages and vdeos. A. Sum of Absolute Dfference The most common block dstorton measure (BDM) s sum of absolute dfference whch gves almost smlar performance of usng MSE but wth much less computatons. Suppose the block sze s and we want to calculate SAD of a block located at () n the current frame F t matched aganst a block wth a dsplacement (u,v) from () n prevous frame F t-, then SAD s smpl defned as: SAD 0 j0 ( u, v) F (, j) F t t ( u, j v) SAD s used n moton compensaton manl because of ts smplct and lttle computatonal cost mpled b SAD to the whole encodng procedure. B. Mean Squared Error Due to ts smplct, MSE (mean square error) has been the domnant quanttatve performance metrc n the feld of sgnal processng for man ears. But t s shown that there s a bg lack of accurac n MSE when dealng wth perceptuall mportant sgnals such as speech, mages and vdeo sgnals. For those applcatons newl developed perceptual qualt metrcs are used. The MSE and PSR between two 8-bt mage sgnals and are formulated as: MSE ( ) 55 PSR 0log (3) MSE Where, are the gra level ntenst values of the pels. C. Structural Smlart SSIM (structural smlart) s a popular and state of the art qualt metrc n the feld of mage processng. We manl use the prmtve formulaton of the Wang and Bovc [6], [7] n whch, the target applcaton s qualt assessment of the mages. SSIM s a functon of lumnance, contrast and a local structure functon of () () Where μ and μ are (respectvel) the local sample means of and, σ and σ are (respectvel) the local sample standard devatons of and, and σ s the sample cross correlaton of and after removng ther means. The tems C, C, and C 3 are small postve constants that stablze each term, so that near-zero sample means, varances, or correlatons do not lead to numercal nstablt. The entre SSIM metrc between the orgnal and the reference mage s calculated b averagng the local SSIM all over the mage. As mentoned before, we are to check the performance of the vdeo coder wth dfferent dstorton functons. We utlzed the dea that MSE s not good enough to be used as qualt metrc and test the performance for varous crterons. The man dea s to use SSIM to search for the best matchng block n moton estmaton of the vdeo. Although the applcaton of SSIM shows better performance than MSE and SAD (better block matchng and thus better compresson) but we go further to mprove ts performance. We eplored the usage of CW-SSIM and VIF. A bref overvew of both s gven below. D. Comple Wavelet SSIM As we saw above, SSIM measures the qualt b comparng the structures n the mage. As we know, small geometrc dstortons are not structural, so a new wavelet doman verson of SSIM was ntroduced to overcome these artfacts [5]. CW-SSIM s usuall formulated as [6]: ~ S ( c, c ) m~ ( c, c ). ~ p( c, c ) c c, c c, K K. c c. c. c *, *, K K Where n the prevous equaton, n the comple wavelet doman, c { c,,,..., } and c { c,,,..., } are respectvel two sets of wavelet coeffcents from the same spatal locaton n the same wavelet subbands of the two mages beng compared. K s a small postve constant used for stablzng. m ~ ( c, c ) s the SSIM nde appled to the magntude of the coeffcents. ~ p ( c c ) s calculated b montorng the dfference between phases of c & c. Frst, CW-SSIM s calculated for each subband of the wavelet decomposton and then average of these values elds an overall CW-SSIM metrc for the entre of the mage. More detals are avalable n [8]. (5)

3 E. Vsual Informaton Fdelt VIF quantfes the smlart of two mages usng a communcaton framework. It attempts to relate the sgnal fdelt to the amount of the nformaton that s shared between the two sgnals (orgnal and nos verson or dstorted verson). Ths shared nformaton s quantfed usng the concept of mutual nformaton whch s wdel used n nformaton theor. Suppose that we are to compare the qualt of two sgnals (mages), a reference and a nos verson (n the current moton estmaton applcaton one sgnal s a block of pels from reference frame and second s a block of pels from net frame). Let us denote the reference sgnal b C and the dstorted one b D. E s the perceved verson of the source sgnal (C) b the neurons of the HVS (human vsual sstem) and F s perceved verson of D. We can wrte the followng equatons: d gc v, e c n, f d n (6) In these equatons, c, and d are random vectors etracted from the same locaton of the same wavelet subband n the reference and dstorted mages, respectvel. g s a scalar determnstc gan factor and v s an ndependent addtve zero-mean whte Gaussan nose. Ths model s a general model but has good performance almost everwhere. See [5] and [6] for more detals. In recever, the n s used to model the vsual dstorton as a statonar, zero-mean, addtve whte Gaussan nose process n the wavelet transform doman. The reference mage s modeled b a wavelet doman Gaussan scale mture (GSM), whch s a good model for natural mages []. Then c can be modeled as c z u where u s a zero-mean Gaussan vector and z s an ndependent scalar random varable. Accordng to [5] the VIF s computed b: c; f z) C; F z) (7) VIF C; E z) c ; e z ) where s the nde of local coeffcents patches, wth all subbands nclude. More nformaton about VIF can be found n [5], [6]. III. DATA The vdeo frames we used as the dataset are the standard Foreman vdeo sequences. The format of the frames s standard CIF format. Fgure shows some sample frames used n the smulaton. We have used the frst frame as the reference frame and the second frame as the target frame n the smulaton procedure. Fgure. Sample frames used for smulatons IV. METHOD We modfed the conventonal moton estmaton method wth the usage of SSIM (and also CW-SSIM and VIF) as the smlart crteron for block matchng. Macro blocks are selected wth a sde equal to 6 pels (6 6 blocks). The comparson between our method and the conventonal method s done n the full search approach. e. the search regon s a block from the target frame. The best matched block n the search area s a 6 6 block wth the bggest smlart to the selected 6 6 block from the reference frame. Bggest smlart s nterpreted as bggest SSIM or bggest CW-SSIM or VIF. After that best match block s found, moton vectors are calculated and moton compensaton s done usng the moton vectors. The smulaton results and trade-offs wll be eplaned n the followng secton. V. RESULTS Moton Compensaton for Foreman frames wth dfferent qualt crterons s smulated usng a. GHz P.4 dual core processor. Table shows the results of the smulaton. As we can see from the table, VIF shows the best performance of compresson. The constructed frame usng moton vectors obtaned b VIF s the most smlar to the target frame. Ths smlart s computed b means of MSE, SSIM, VIF and also btplane error. Btplane error s the average Hammng dstance between the reference frame and the target frame when each of them s consdered as a sequence of bnar bts (Each pel s a 8 bt bnar and the sequence can be obtaned b puttng these pel values n seral order). The trade-off between the computatonal complet and qualt of the compressed vdeo can be observed n the table. Although the usage of VIF shows better performance but ts computatonal complet s bgger than the other methods. It seems that SSIM s the knee pont of the trade-off. The performance s acceptable whle the complet s not much bgger that MSE. The usage of VIF or CW-SSIM would be helpful n stuatons that the complet s not of much concern such as n dstrbuted vdeo codng wth moton estmaton n the decoder [6]. Fgure 3 shows the Foreman reference frame and target frame obtaned b varous metrcs. Fgure 3.a s the target

4 frame. Fgure 3.b, 3.d, 3.e, 3.f, 3.g are moton compensaton output frames usng SAD, MSE, SSIM, CW-SSIM, VIF qualt crterons respectvel. Fgure 3.c s the constructed target frame wth SAD dstorton measure and a block of sze 8 (nstead of 6; we put ths fgure to sketch the poor performance of SAD). As we can see from the results the usage of qualt assessment of the mages n the moton compensaton can help us to manage the vdeo compresson n case of requred computatonal complet and codng performance. Another nterestng pont s that the usage of SSIM, CW- SSIM and VIF more than reducng the average btplane error, would cause n a remarkable decrease n the error of the most sgnfcant bts between the reference and target frames. Table eplans ths fact. The left sded bts of the bnar representaton of the pels have less error after the compresson when we use the HVS based qualt crterons n the moton estmaton. The ke pont here s the fact that n man lossless and loss compresson technques that onl deal wth the bt sequence (and the nature of the sgnal s not such mportant), more precse mportant btplanes results n better codng. Ths s mportant because n almost all vdeo coders and standards there s bt codng procedure (source codng stage) more than the moton compensaton stage. TABLE I. COMPARISO BETWEE TARGET FRAME AD COSTRUCTED FRAME USIG MOTIO COMPESATIO Comparson between target frame and constructed Smulaton Qualt frame usng moton compensaton Tme (Sec.) SAD MSE=3.55, SSIM=0.55, VIF=0.09, dst= MSE MSE=3.33, SSIM=0.57, VIF=0.0, dst= SSIM MSE=3.6, SSIM=0.8, VIF=0.50, dst= CWSSIM MSE=0.4, SSIM=0.88, VIF=0.58, dst= VIF MSE=7., SSIM=0.9, VIF=0.63, dst=0.4 7 TABLE II. COMPARISO BETWEE TARGET FRAME AD COSTRUCTED FRAME USIG MOTIO COMPESATIO, BITPLAE HAMMIG DISTACES Qualt Comparson between target frame and constructed frame usng moton compensaton, btplane Hammng dstances SAD [0.64, 0.03, 0.3, 0.387, 0.48, 0.453, 0.366, 0.47] MSE [0.48, 0.89, 0.304, 0.364, 0.4, 0.448, 0.467, 0.473] SSIM [0.07, 0.04, 0.5, 0.03, 0.99, 0.43, 0.465, 0.475] CWSSIM [0.03, 0.040, 0., 0.95, 0.8, 0.40, 0.467, 0.478] VIF [0.09, 0.037, 0.095, 0.8, 0.73, 0.39, 0.463, 0.47] Fgure 3. Comparson of the target frame and reconstructed frame usng moton estmaton: a) target frame b) constructon usng SAD c) SAD (8 8 macro block) d) MSE e) SSIM f) CW-SSIM g) VIF

5 VI. COCLUSIO AD FUTURE WORKS In ths paper, we frst revew the recent advances n the vdeo codng and specall moton compensaton procedure then we state a bref overvew about the mage qualt metrcs. After that, we eplan our method that s the usage of the HVS based metrcs n the vdeo compresson and more precsel n moton compensaton. We eamne the legtmac of our approach through the smulaton. Results revled ths fact that usage of these metrcs can make a remarkable mprovement n the compresson performance, especall on the moton compensaton and source codng stages. VII. REFERECES [] A. M. Tekalp, Dgtal Vdeo Processng, Prentce Hall, Sgnal Processng Seres, 995. [] Introducton to Moton Pcture Codng and the CCITT Algorthm, Inmos Lmted, Februar, 990. [3] G. Ashraf and M.. Chong, Performance analss of H.6 and H.63 vdeo codng algorthms, IEEE Proceedngs of Internatonal Smposum on Consumer Electroncs, ISCE 97, Sngapore, Dec [4] Sh. Uramoto, A. Takabatake, M. Suzuk, H. Sakura and M. Yoshmoto, A half-pel precson moton estmaton processor for TSC-resoluton vdeo, IEEE Conference on Custom Integrated Crcuts, 993. [5] T. Tovonen and J. Hekkla, Effcent method for half pel block moton estmatonusng block dfferentals, VLBV Internatonal Workshop, Sep. 003, Madrd, Span. [6] Y. G. Lee, J. H. Lee and J. B. Ra, Fast half pel moton estmaton based on drectonal search and a lnear model, Proceedngs of SPIE, IEEE Vsual Communcatons and Image Processng, vol. 550, pp , 003. [7] T. Ebrahm and C. Horne, MPEG4 natural vdeo codng, an overvew, IEEE Sgnal Processng: Image Communcaton, vol. 5, pp , 000. [8] B. Grod, E. Stenbach and. Farber, Comparson of H.63 and H.6 vdeo compresson standards, Standards and Common Interfaces for Vdeo Informaton Sstems, SPIE, vol. CR60, Oct. 995, Phladelpha, USA. [9] I. E. G. Rchardson, H.64 and MPEG-4 Vdeo Compresson, England, John Wle and Sons, 003. [0] K. R. amudur and A. J, Computaton and performance tradeoffs n moton estmaton, Internatonal Conference on Informaton Technolog: Codng and Computng, ITCC 0, Aprl 00. [] T. Y. Kuo, J. Chaldabhongse and C. C. Ja Kuo, Fast moton vector search for overlapped block moton compensaton (OBMC), Conference on Sgnals, Sstems and Computers, ACSSC, vol., pp , 996. [] E. Chan and S. Panchanathan, Revew of block matchng based moton estmaton algorthms for vdeo compresson, IEEE CCECE/CCGEI 993. [3] J. Y. Tham, S. Ranganath, M. Ranganath and A. Al Kasm, A novel unrestrcted center based damond search algorthm for block moton estmaton, IEEE Transactons on Crcuts and Sstems for Vdeo Technolog, vol. 8, o. 4, Aug [4] H. Ja and L. Zhang, A new cross damond search algorthm for block moton estmaton, IEEE ICASSP 004. [5] H. R. Shekh and A. C. Bovc, "Image nformaton and vsual qualt," IEEE Transactons on Image Processng, Vol. 5, o., Feb [6] Z. Wang and A. C. Bovk, Mean squared Error: love t or leave t?, IEEE Sgnal Processng Magazne, Jan [7] Z. Wang, A. C. Bovk, H. R. Shekh, and E. P. Smoncell, Image qualt assessment: from error vsblt to structural smlart, IEEE Trans. Image Processng, vol. 3, pp , Apr [8] Z. Wang and E.P. Smoncell, Translaton nsenstve mage smlart n comple wavelet doman, n Proc. IEEE Int. Conf. Acoustcs, Speech, Sgnal Processng, Mar. 005, pp [9] Z. Wang, H. R. Shekh and A. C. Bovk, Objectve Vdeo Qualt Assessment, The Handbook of Vdeo Databases: Desgn and Applcatons, CRC Press, Florda, USA, p , 003. [0] Z. Wang, L. Lu, A. C. Bovk, Vdeo qualt assessment based on structural dstorton measurement, IEEE Sgnal Processng Magazne, vol. 9, pp. -3, 004. [] C. Cheung and L. Po, Adjustable partal dstorton search algorthm for fast block moton estmaton, IEEE Trans. on Crcuts and Sstems for Vdeo Technolog, vol. 3, no., Jan [] J. Portlla, V. Strela, M. Wanwrght, and E.P. Smoncell, Image denosng usng scale mtures of Gaussans n wavelet doman, IEEE Trans. Image Processng, vol., no., pp , ov [3] H. Zhang, X. Tan and Y. Chen, A Vdeo Structural Smlart Qualt Metrc Based on a Jont Spatal-Temporal Vsual Attenton Model, Journal of Zhejang Unv., Vol., pp , Sprnger, 009. [4] K. Seshadrnathan and A. C. Bovk, A Structural Smlart Metrc for Vdeo Based on Moton Models, IEEE Int. Conference on Acoustcs, Speech and Sgnal Processng, pp , ICASSP 007. [5] A. Ortega and K. Ramchandran, Rate-dstorton methods for mage and vdeo compresson, IEEE Sgnal Processng Magezne, pp. 3-50, ov [6] J.Ascenso,C. Brtes,and F. Perera, Improvng Frame Interpolaton wth Spatal Moton Smoothng for Pel Doman Dstrbuted Vdeo Codng, EURASIP Conference on Speech and Image Processng, Multmeda Communcatons and Servces, Smolence, Slovak Republc, June 005.