Motion Vector Recovery for Real-Time H.264 Video Streams

Transcription

1 Internatonal J. of Recent Trends n Engneerng and Technology, Vol., No., May 2 Moton Vector Recovery for Real-Tme H.264 Vdeo Streams Kavsh Seth, Tummala Rajesh, V. Kamakot 2, and S. Srnvasan Dept. of Electrcal Engg., Indan Insttute of Technology Madras, Chenna, Inda Emal: {kavshseth, trajeshreddy}@gmal.com, srn@ee.tm.ac.n 2 Dept. of Computer Sc. and Engg., Indan Insttute of Technology Madras, Chenna, Inda Emal: veezh@gmal.com Abstract Among the varous network protocols that can be used to stream the vdeo data, RTP over UDP s the best to do wth real tme streamng n H.264 based vdeo streams. Vdeos transmtted over a communcaton channel are hghly prone to errors; t can become crtcal when UDP s used. In such cases real tme error concealment becomes an mportant aspect. A subclass of the error concealment s the moton vector recovery whch s used to conceal errors at the decoder sde. Lagrange Interpolaton s the fastest and a popular technque for the moton vector recovery. Ths paper proposes a new system archtecture whch enables the RTP-UDP based real tme vdeo streamng as well as the Lagrange nterpolaton based real tme moton vector recovery n H.264 coded vdeo streams. A completely open source H.264 vdeo codec called FFmpeg s chosen to mplement the proposed system. Proposed mplementaton was tested aganst the dfferent standard benchmark vdeo sequences and the qualty of the recovered vdeos was measured at the decoder sde usng varous qualty measurement metrcs. Expermental results show that the real tme moton vector recovery does not ntroduce any notceable dfference or latency durng dsplay of the recovered vdeo. Index Terms Dgtal Vdeo, Moton Vector, Error Concealment, H.264, UDP, RTP I. INTRODUCTION Streamng of vdeos s a very common mode of vdeo communcaton today. Its low cost, convenence and worldwde reach have made t a hugely popular mode of transmsson. The vdeos can ether be a pre-recorded vdeo sequence or a lve vdeo stream. The vdeos captured are raw vdeos, whch take up lot of storage space. Vdeo compresson technologes have to be wdely employed n vdeo communcatons systems n order to meet the channel bandwdth requrements. The H.264 s currently one of the latest and most popular vdeo codng standard []. Compared to prevous codng standards, t s able to delver hgher vdeo qualty for a gven compresson rato, and better compresson rato for the same vdeo qualty. Because of ths, varatons of H.264 are used n many applcatons ncludng HD-DVD, Blu-ray, Pod vdeo, HDTV broadcasts, and most recently n streamng meda. The compressed vdeos are sent n the form of packets for streamng meda. The packets sent are hghly prone to erroneous transmsson. The packet may be damaged or may not be receved at all. Such errors are lkely to damage a Group of Blocks (GOB) of data n the decoded frames for block-based codng schemes such as H.264. Error also propagates due to hgh correlaton between neghborng frames and degrades the qualty of successve frames. The Real Tme Protocol (RTP) over User Datagram Protocol (UDP) s the recommended and commonly used mechansm employed whle streamng the H.264 meda format [2]. Varous approaches have been used to acheve error reslence n order to deal wth the above problem. A nce overvew of such methods s gven n [], [4]. One of the ways to overcome ths problem s the mplementaton of Error Concealment (EC) at the decoder sde. Moton Vector Recovery (MVR) s one way of EC whch uses several mathematcal technques to recover the erroneous moton felds. Among the varous MVR technques reported n the lterature, the Lagrange Interpolaton (LAGI) s the fastest and a popular MVR technque whch produces the hgh qualty of the recovered vdeo [5]. FFmpeg s a comprehensve multmeda encodng and decodng lbrary that conssts of numerous audo, vdeo, and contaner formats [6]. Ths paper proposes a new system archtecture whch enables the real tme vdeo streamng as well as the real tme MVR. The proposed archtecture s mplemented n both FFmpeg coder and decoder. A RTP packet encapsulaton/decapsulaton module s added to the FFMpeg codec, whch packs/unpacks the Network Abstracton Layer (NAL) packets [2], [7] nto sngle NAL unt type RTP packets. A UDP socket program s used at both coder and decoder sdes to stream the RTP packets over UDP. A LAGI based MVR technque s mplemented at decoder sde. The proposed system mplementaton was tested aganst the dfferent benchmark vdeo sequences. Qualty of the receved vdeos can be measured usng varous qualty measurement standards such as Peak Sgnal to Nose Rato (PSNR) and Vdeo Qualty Evaluaton Metrc (VQM) [8] tools. The expermental secton presents a bref analyss of whch qualty measurement parameter s best suted for the streamng vdeo analyss. Expermental results show that the proposed mplementaton does not ntroduce any latency or degradaton n the qualty of the recovered vdeo whle streamng and performng the MVR smultaneously n real tme. 6

2 Internatonal J. of Recent Trends n Engneerng and Technology, Vol., No., May 2 Rest of the paper s organzed as follows: a bref ntroducton of the H.264 codec s presented n Secton II. An overvew of real tme streamng for H.264 vdeos s gven n Secton III. The MVR detals are covered n Secton IV. A system archtecture for real tme streamng and real tme MVR s proposed n Secton V. Expermental results are presented n Secton VI and last secton concludes the paper. II. H.264 CODEC The H.264 vdeo codec s an effcent codng scheme that covers all forms of dgtal compressed vdeo rangng from low bt-rate Internet streamng applcatons to HDTV broadcast and Dgtal Cnema applcatons. Compared to the current state of technology, the H.264 standard s shown to yeld same qualty of mages as that produced by current state-of-the-art standards wth a savngs of 5% on the btrate over the other standards. For example, the H.264 s reported to have acheved the same qualty of mages usng a btrate of.5 Mbt/s compared to what was acheved usng the MPEG 2 standard at.5 Mbt/s [9]. The codec specfcaton [] tself dstngushes conceptually between a vdeo codng layer (VCL) and a NAL. The VCL performs the sgnal processng part of the codec namely, mechansms such as transform, quantzaton, and moton compensated predcton; and a loop flter. It follows the general concept of most of today s vdeo codecs, a Macro Block (MB) based coder that uses nter pcture predcton wth moton compensaton and transform codng of the resdual sgnal. The VCL encoder outputs slces: a bt strng that contans the MB data of an nteger number of MBs, and the nformaton of the slce header (contanng the spatal address of the frst MB n the slce, the ntal quantzaton parameter, and smlar nformaton). The NAL encoder encapsulates the slce output of the VCL encoder nto NAL unts, whch are sutable for transmsson over packet networks or use n packet orented multplex envronments. A NAL unt conssts of a one-byte header and the payload byte strng. The header ndcates the type of the NAL unt, presence of bt errors or syntax volatons n the NAL unt payload, and nformaton regardng the relatve mportance of the NAL unt for the decodng process. III. REAL TIME STREAMING FOR H.264 VIDEOS The Internet Protocol (IP) [] s a packet-basednetwork transport protocol upon whch the nternet s bult. IP packets are encapsulated n lower, hardwarelevel protocols for delvery over varous networks (Ethernet, etc), and they encapsulate hgher transport- and applcaton-level protocols for streamng and other applcatons. In the case of streamng H.264 vdeos over IP networks, multple protocols, such as RTP and UDP, are carred n the IP payload, each wth ts own header and payload that recursvely carres another protocol packet. For example, H.264 vdeo data that s carred n an RTP packet whch n turn s carred n a UDP packet whch n turn s carred n an IP packet. The UDP sends Fgure. Frame wth Lost Macro block the meda stream as a seres of small packets. Ths s smple and effcent; however, there s no mechansm wthn the protocol to guarantee delvery. It s up to the recevng applcaton to detect loss or corrupton and recover data usng error correcton technques. If data s lost, the stream may suffer a dropout. The RTP was developed for carrage of real tme data over IP networks [], [2]. RTP s a natve nternet protocol, desgned for and fttng well n the general sute of IP protocols. RTP does not provde any multplexng capablty. Rather, each meda stream s carred n a separate RTP stream and reles on underlyng encapsulaton, typcally UDP, to provde multplexng over an IP network. Because of ths, there s no need for an explct de-multplexer on the clent ether. Each RTP stream must carry tmng nformaton that s used at the clent sde to synchronze streams when necessary. IV. MOTION VECTOR RECOVERY IN H.264 Among the exstng MVR algorthms reported n lterature, the LAGI [5] s the most popular technques used to recover the lost MVs n H.264 encoded vdeo. The computaton cost of the LAGI based MVR technque s lower than most other nterpolaton functons and hence ths technque becomes a good choce for real tme vdeo streamng applcatons. Ths secton presents a MVR method that s based on LAGI formula. For n + gven ponts ( x, y ) and =, K, n by sutable choce of parameters, we can consttute the nterpolaton functon f (x) such that f ( x ) = y. The formula for LAGI functon s as follows: f = yl + yl + L + ynln () Where L, K, Ln denote the parameters n the LAGI formula. They can be computed from the n + gven ponts, by the followng formula: 7

3 Internatonal J. of Recent Trends n Engneerng and Technology, Vol., No., May 2 x Table I the correspondng coordnates of each mv x x x 2 x L y = f (x) MV MV MV 2 MV ( x x x x = ( x x x x + x xn ) x x ) + The Lagrangan bass functons are calculated as follows: Where, ( ) ( x x 2 ) ( x x )( x x 2 )( x x ) n (2) ( ) ( x x 2 ) (4) ( x x )( x x 2 )( x x ) ( ) ( x x ) (5) 2 ( x 2 x )( x 2 x )( x 2 x ) L ( ) ( x x 2 ) = ( x x )( x x )( x x 2 ) (e) The Lagrangan polynomal s formed as follows: f = MVL + MVL + L + MVL (7) The H.264 standard dvdes every frame nto several Macro Blocks (MBs). Each MB s assocated wth to 6 Moton Vectors (MVs) ensurng backward compatblty wth prevous standards. Fg. shows a H.264 frame segment wth 9 MBs denoted by F m, n, where m and n denote the spatal locaton of the MB wthn the frame. Each MB s assocated wth 6 MVs. In Fg., let F m, n denote the lost MB. As n the case of many MVR algorthms, t s assumed that ether two of the vertcally adjacent or two of the horzontally adjacent MBs of the lost MB are correctly decoded [5], []. In Fg., t s assumed wthout loss of generalty that both the horzontally adjacent MBs of F m, n are error-free. In ths case, the lost MVs of F m, n are recovered row-by-row. Let MV ( ) denote the correct MVs that belong to a partcular row of the horzontally adjacent MBs of F m, n as shown n Fg.. Let V ( ) represent the MVs of the rows of F m, n that need to be recovered. (6) Fgure. Proposed decoder archtecture The procedure to recover the one row of MVs of F m, n s descrbed as follows:. The correct neghborng MVs MV, K, MV are used to compute Lagrange bass functons by substtutng these values n (6). 2. By substtutng n (7), the four ( x, y) pars shown n Table I, a thrd degree LAGI polynomal s formed.. The lost MVs are computed by substtutng the values of MV and x n (7). A smlar procedure s followed f the vertcally adjacent frames of F m, n are error-free. In ths case, the lost MVs of F m, n are recovered column-by-column usng the correct MVs of F m, n and F m, n+ as shown n Fg.. V. PROPOSED ARCHITECTURE FFmpeg provdes the complete package to encode/decode most of the popular encoded vdeos. It s a computer program that can record, convert and stream dgtal audo and vdeo n numerous formats [6]. FFmpeg s a command lne tool that s composed of a collecton of free software and open source lbrares. It ncludes lbavcodec, an audo/vdeo codec lbrary used by several other projects, and lbavformat, an audo/vdeo contaner mux and demux lbrary. On careful examnaton of the lbavcodec source code, the fle used by FFmpeg to decode the H.264 vdeos was found to be h264.c.the name of the project comes from the MPEG vdeo standards group, together wth FF for fast forward. FFmpeg s developed under Lnux, but t can be compled under most operatng systems, ncludng Apple Inc. Mac OS X, Mcrosoft Wndows and AmgaOS. The proposed system archtecture at FFmpeg encoder sde conssts of a H.264 encoder, a RTP encapsulaton module and a UDP server. Ths UDP server uses a socket program to transmt any H.264 vdeo as the UDP packets over a predetermned port known both to the server as well as to the clent. Table II table showng the psnr values for varous vdeo sequences Sequence PSNR (db) wthout error wth MVR wth error Akyo Coastguard Fgure 2. Proposed encoder archtecture Foreman

4 Internatonal J. of Recent Trends n Engneerng and Technology, Vol., No., May 2 Table III table showng the vqm values for varous vdeo sequences Sequence VQM wthout error wth MVR wth error Akyo Coastguard Foreman The block dagram of FFmpeg encoder sde archtecture s presented n Fg. 2. The system archtecture at FFmpeg decoder sde conssts of a H.264 decoder, a RTP decapsulaton module and a UDP clent whch keeps lstenng on same port used by socket program run on server end. The block dagram of FFmpeg decoder sde archtecture s presented n Fg.. Once UDP clent receves the H.264 lve stream, t forwards ts payload (UDP payload s a sngle NAL unt type RTP packets) to the RTP decapsulaton module. Note that the RTP encapsulaton/decapsulaton modules can deal only wth sngle NAL unt type of packets. RTP decapsulaton module gves out the NAL packets to the H.264 decoder wth MVR capablty. The RTP decapsulaton module also performs a check on erroneous or mssng packets based on the RTP sequence number and tmestamp and t reports the check results to the decoder to perform MVR. In order to ntroduce MVR as a part of decodng process, a LAGI based MVR module s ncorporated wth the moton compensaton module of the FFmpeg decoder. VI. EXPERIMENTAL RESULTS After mplementng the proposed system archtecture n FFmpeg codec, the mplementaton s tested wth three dfferent standard benchmark vdeo sequences the Akyo, the Foreman, and the Coastguard. All of them are Quarter Common Intermedate Format (QCIF) raw sequences and the total number of frames n each sequence s 7. In order to conduct the experments, errors are delberately ntroduced nto these sequences before streamng them to decoder. Note that 5% MB loss randomly s ntroduced n all the P frames of each sequence and the Quantzaton Parameter (QP) s set to 24. Fgure 4. PSNR of the Akyo sequence wth error and after MVR Fgure 5. The frame n the Akyo sequence wthout error Fgure 6. The error frame n the Akyo sequence The complexty and cost of subjectve qualty measurement make t attractve to be able to measure qualty automatcally usng an algorthm. The most wdely used measure s Peak Sgnal to Nose Rato (PSNR) but the lmtatons of ths metrc have led to many efforts to develop more sophstcated measures lke Vdeo Qualty Evaluaton Metrc (VQM) that approxmate the response of real human observers. VQM s a modfed dscrete cosne transform based VQM based on Watsons proposal [4], whch explots the property of vsual percepton. A detaled account of how t s mplemented can be found at [8]. The values of PSNR and VQM have been calculated usng a Vdeo Qualty Measurement tool called MSU [8]. Fgure 7. The error frame n the Akyo sequence after MVR The PSNR values for the above mentoned vdeo sequences are gven n Table II. The PSNR graph for the Akyo sequence wth error and after MVR s shown n Fg. 4. The frames of the Akyo sequence wthout error, wth error and after MVR are shown n Fg. 5, 6 and 7 respectvely. The VQM values are used to quantfy the qualty of the vdeo streams. The VQM values reflects the dfferent characterstcs of a vdeo stream e.g. latency, frames 9

5 Internatonal J. of Recent Trends n Engneerng and Technology, Vol., No., May 2 drops, vsual qualty of a frame, resoluton etc. Hgher the value of VQM of a gven vdeo stream ndcates the bad qualty. On the other hand, PSNR values are used to access only the vsual qualty of frames and doesn t consder other parameters lke latency, frame drops etc. The VQM values for the dfferent vdeo sequences are gven n Table III. It s clear from the Table III that the VQM values of the recovered vdeos fall wthn pretty much acceptable range. Ths ndcates that the streamng qualty on the proposed mplementaton s qute good. The VQM graph for the Akyo sequence wth error and after MVR s shown n Fg. 8. Though PSNR and VQM can both be used to measure vdeo qualty, VQM performances much better n stuatons when PSNR fals. The lght computaton and memory load make t even more attractve for wde applcatons [8]. CONCLUSIONS Vdeo streamng usng the RTP over the UDP has been successfully tested and reported n ths paper. The MVR usng the Lagrangan nterpolaton technque has been ntegrated nto FFmpeg and tested for varous erroneous H.264 vdeo sequences. The MVR algorthm also gves good results wth real tme vdeo wthout any notceable dfference or latency durng dsplay. Also the analyss of the receved vdeos through dfferent qualty measurement metrcs. shows that the VQM s a better way to measure qualty than PSNR. REFERENCES [] ITU-T Recommendaton: H.264 Advanced vdeo codng for generc audovsual servces, May 2. [2] S. Wenger, M. M. Hannuksela, T. Stockhammer, M. Westerlund, D. Snger, RTP Payload Format for H.264 Vdeo: RFC 984, February 25. [] Y. Wang and Q. F. Zhu, Error control and concealment for vdeo communcaton: a revew, Proc. IEEE, vol. 86, no. 5, May 998 [4] Y. K. Wang, M. M. Hannuksela, V. Varsa, A. Hourunranta, and M. Gabbouj, The error concealment feature n the H.26L test model, Proc. IEEE Int. Conf. Image Processng,,pp , 22. [5] J. H. Zheng and L. P. Chau, Moton vector recovery algorthm for dgtal vdeo usng Lagrange Interpolaton, IEEE Trans. Broadcastng, vol. 49, No. 4, pp. 8-89, Dec 2. [6] FFmpeg, [7] Ian E. G. Rchardson, H.264 and MPEG-4 Vdeo Compresson: Wley, 2. [8] MSU, Vdeo Qualty Measurement Tools: VQM Reference Manual, [9] A. Luthra, G. J. Sullvan, and T. Wegand, Specal Issue on H.264/AVC, IEEE Transactons Crcuts and Systems on Vdeo Technology, vol., no. 7, pp , July 2. [] J. Postel, Internet Protocol: RFC 79, September 98. [] H. Schulzrnne, S. Casner, R. Frederck, and V. Jacobson, RTP: A Transport Protocol for Real-Tme Applcatons: STD 64, RFC 55, July 2. [2] H. Schulzrnne, S. Casner, R. Frederck, and V. Jacobson, RTP: A Transport Protocol for Real-Tme Applcatons: RFC 889, January 996. [] Kavsh Seth, M. Komsetty, V. Anand, V. Kamakot, and S. Srnvasan, VLSI mplementaton of Moton Vector Recovery Algorthms for H.264 Vdeo Codecs," th IEEE/VSI VLSI Desgn and Test Symposum, Bangalore (Inda), July 29. [4] A. B. Watson, Image data compresson havng mnmum perceptual error: US Patent 5,629, Fgure 8. VQM of the Akyo sequence wth error and after MVR 4