Image Characteristic Based Rate Control Algorithm for HEVC

Transcription

1 Image Characteristic Based Rate Control Algorithm or HEVC Mayan Fei, Zongju Peng*, Weiguo Chen, Fen Chen Faculty o Inormation Science and Engineering, Ningbo University, Ningbo 352 China *pengzongju@26.com; my75@63.com Abstract: - Rate control plays an important role in high quality video coding. In this paper, a rate control algorithm based on video characteristics is proposed. Firstly, we put orward a novel bit allocation algorithm or intra rame by analyzing the relationship among the image characteristic, bit per pixel and quantization step. The image gradient is used as image characteristic. In rame layer, dierent bit allocation model are applied according to rame type and assign the rational bit to rame according the complexity o video content. In largest coding unit (LCU layer, it also chooses dierent strategies o bit allocation according to the type o LCU. To be more speciic, the bit o a LCU without previously coded collocated LCU is assigned on the basis o its image complexity. Otherwise, the bit o a LCU is assigned by using the gradient o residual belonging to collocated LCU. Experimental results show that the proposed algorithm is better than the state-o-the-art rate control algorithm in terms o the accuracy o rate control and the coding quality, the average o PSNR has been improved by 0.5dB. Maximal PSNR gaining can reach 0.67dB. Key-Words: - high eiciency video coding, rate control, image characteristic, largest coding unit Introduction With the continuous development o video compression and broadband network, highdeinition video related applications are becoming particularly popular. However, the inormation in high deinition video is huge, only by compressing original signal using video coding can them be eectively stored and transerred. To satisy the increasing demands o storing and transmitting high deinition video contents over the Internet, high eiciency video coding (HEVC standard is proposed in early 203 by the Joint Collaborative Team on Video Coding (JCT-VC []. It only require a hal o the bit-rate in H.264/AVC encoder to achieve the same subjective visual quality [2]. The improvement o compression ratio is largely due to the act that HEVC adopted some new techniques such as the expanded prediction and transorm block sizes with a lexible coding structure. Those new techniques pose new challenges in developing an eicient rate control or HEVC. In video communication applications, video data is transmitted over a limited network bandwidth. To make the most eective use o the limited network bandwidth while maintaining optimized visual quality o reconstruction video, rate control play a crucial role or video compression and communication applications [3]. Superabundant or over-reduced bits stream will be produced in video encoder without an eective rate control. Accordingly, There are many rate control have been proposed or various video coding standards in past ew years. For example, the test model near-term (TMN or H.263, the joint model (JM or H.264, etc. However, research on HEVC rate control is still let dormant. A ew scholars have tried to extend some rate controls in previous video coding standards to the early version o the HEVC reerence sotware [4]-[5]. The authors in [4] introduced the quadratic method o H.264 into the HEVC reerence sotware. The authors in [5] studied the ρ-domain method in HEVC. Nevertheless, due to the dierent coding structure o HEVC rom the previous video coding standards, the rate control used in previous video coding standards are not suitable or HEVC. Thus, it is urgent need to make a speciic rate control on HEVC. Recently, some research o rate control on HEVC had been done [6]-[0]. Li et al. [6][7] proposed the R-λ algorithm. It has better rate control perormance, but it also has some drawbacks. i.e., or intra rames, the process o rame layer bit allocation is without considering the characteristics o video content and largest coding unit (LCU layer bit allocation is based on the idea o average allocation, not considering characteristics o rames. For B/P rames, the weights o rames in a group o picture (GOP are predeined and not so adaptive. Wang et al. [8] proposed a rate-gop based rame level rate control scheme or HEVC. Though this scheme acquired a better R-D perormance, it neglected the characteristic o image. Meddeb et al. [9] proposed a region-o-interest (ROI based rate E-ISSN: Volume 2, 206

2 control scheme or video conerence application. More bits are allocated to ROIs which are obtained by a ace detection algorithm. Wang et al. [0] modiied the R-λ model by using the image gradient to represent the image complexity, and adjusted the bit o LCU according to the complexity o LCU. But it is only applicable to the structure o all intra rame case. With respect to the above shortcomings, we propose a rate control algorithm which is based on image characteristics. In rame layer, we present a bit allocation strategy or I-rame according to its image complexity, a proper number o bits are allocated to an I-rame, and a Rate-Complexity-QP (R-C-Q model which is used to calculate the quantization step based on both its image complexity and pre-assigned bits or I-rame. In addition, we also develop a bit allocation strategy or P/B rame in accordance with its image complexity. In LCU layer, we modiy the strategy o bit allocation according to dierent types o LCU. Experimental results demonstrate that the proposed rate control algorithm can achieve better coding quality than that o the state-o-the-art rate control strategy with less bit rate mismatching and deviation o PSNR. The rest o the paper is organized as ollows. In Section 2, we briely describe the image complexity measurement. In Sections 3 and 4, we describe the rate control strategies or intra and inter rames, respectively. The experiment results are analyzed in Section 5. The conclusion is given in Section 6. 2 Image Complexity Measurement In general, the encoding bits o each rame are mainly rom quantized residual inormation and entropy coding. In [], it has been demonstrated that there is a very strong linear relationship between the encoding bit rate and the percentage o zeros among the quantized transorm coeicients. That is, the relationship can be characterized as R = θ ( ρ N zero = θ ( N = α N non _ zero where R is the encoding bits o a rame, ρ is the percentage o zeros among the quantized transorm coeicients. The larger value o ρ, the less the residual inormation. θ and a are two model parameters. N, N zero and N non_zero are the total number o pixels in a rame, the numbers o zeros and the ( numbers o non-zeros among the quantized transorm coeicients o one rame, respectively. Since HEVC utilizes the spatial correlation o the coding unit in the process o intra prediction [2], it resulted that the number o residual signal directly related to image complexity [3]. In addition, nonzeros coeicients are rom the quantized transorm o residual inormation. We can assume that relationship between the video texture complexity and N non_zero can be deined as: C = k N non _ zero (2 where K is model parameter. Combining ( and (2 gives: R = β C (3 where β is model parameter, R and C are coding bits and the complexity o image, respectively. At present, there are a variety o image complexity measurement methods [4][0]. For example, absolute error and gradient o image are oten used to describe the image complexity. We adopt image gradient to measure the complexity o image [0]. Gradient is calculated as: C = I I + I I H H W ( i, j i+, j i, j i, j+ (4 W i= 0 j= 0 where I i,j is the (i, jth luminance value in image. H and W are the height and width o image, respectively. To veriy the relationship between R and C in the ormula (3, we conducted a series o experiments. Fig. shows the relationship between complexity o image and coded bits with sequences BQSquare and RaceHorseC when quantization parameter is 22. From the Fig., obvious linear relationship between R and C has been presented. The coded bits o rames are changing simultaneously with the complexity o image. Bits 2.3 x Image complexity bits Frame index Image complexity E-ISSN: Volume 2, 206

3 Bits 7 x bits (a Image complexity Frame index (b Fig. The relationship between image complexity and bits. (a BQSquare, (b RaceHorseC. 3 Rate Control Strategy or Intra Frame The rate control algorithm in HEVC can be roughly divided into two processes: bit allocation and quantization parameter calculation. The process o bit allocation is based on some strategy to assign the target bits which is set by users to group o picture (GOP layer, rame layer and largest coding unit layer. The process o quantization parameter calculation is also divided into two steps: the irst step is to calculate the corresponding Lagrange multiplier according to the pre-assigned bits. Another step is to calculate the quantization parameter by using the Lagrange multiplier rom the irst step. The purpose is making the output rate to get close the target rate. 3. Frame Layer Bit Allocation or Intra Frame In the encoding process, intra rame is critical, the coding quality o which directly aects subsequent coding rames. However, with the diversiication o video content, various types o movement, ranging rom a single small target movement to more than one big moving targets, even to camera movement, might occur. In a relatively short period o time, intra rame may be used as reerence or subsequent coding rame. But with drastic movement o objects in sequence or the camera shit, an intra-rame might be no longer used as a reerence rame or subsequent coding. I the bits allocated to intra rames are not enough, the quality o subsequent coding rames will deteriorate. I the bits or intra rame are excessive, the quality o intra rame has improved. However, i intra rame are not reerenced as a long term or subsequent rame, the kind o allocation strategy will cause the waste o bits, i the wasted bits o intra rame are assigned to Image complexity subsequent coding rame, it is will relieve shortage o bits or ollow-up rame. Based on the above analysis, we can see the bit allocation o intra rame is very essential. To tackle the above mentioned problems, we introduce image characteristics into the intra rame bit allocation process. The allocation strategy is as ollows: C ( i β IntraBits = α Bitavg (5 bpp Bit avg bpp = W H Rtar Bitavg = (7 where IntraBits is the pre-assigned bits o intra rame, α and β are model parameters. C i is the image complexity o i-th rame, Bit avg is the average bit, R tar is target bit rate, is rame rate, bpp is bit per pixel, W and H are width and height o image, respectively. In order to veriy the proposed bit allocation strategy or intra rame which incorporates image complexity, we have done some experiments. Firstly, we encoded the irst 300 rames o our standard WVGA ( test sequences and our standard WQVGA ( test sequences using intra-pattern with dierent quantization (22, 27, 32, 37. Secondly, the bit rate o encoded is used as the target bit rate o the rate control, experimental conditions keep the same except the rate control is enable. Ater coding, we itted the data which contain image complexity and coded bit, and ound that the result is very close to the strategy which is proposed by us, the correlation coeicient is up to Fig.2 Intra Bit Allocation Model 3.2 LCU Layer Bit Allocation or Intra Frame (6 E-ISSN: Volume 2, 206

4 When the rame level IntraBits is allocated, the proposed LCU level bit allocation strategy is based on the image complexity o each LCU in the current rame. IntraBits BitH CodedSilce TLCU = CLCUCur (8 C i NotCodedLCUs LCU where Bit H is the estimated header bits, which is obtained by averaging the header bits o the previous picture belonging to the same picture level. Coded Slice is the number o bits coded in current rame. C LCU is the image complexity o LCU. 3.3 Intra Frame R-C-Q Control Model In order to determine the relationship among bit rate o I rame, image complexity measured by gradient and quantization parameter (QP, we conducted a two-step exploration. In the irst step, a series o sequences are encoded with ixed quantization parameter and the distribution o generated bits and image complexity on the rame level or dierent sequences are illustrated in Fig.3. From Fig.3, a strong linear relationship has been presented between image complexity and coded bits. bpp bpp Frame Complexity (a Frame Complexity (b Fig.3 The relationship between rame complexity and bit rate at ixed QP=22. (a BlowingBubbles, (b BasketballDrill. In addition, or various QPs, similar linear relationship exists as well. Thereore, it is sae that we express R, C and QP relationship in the ollowing manner: R( Ci, QP = α Ci ( QP (9 where R(C i, QP is I rame bit rate, a is model parameter, C i is the image complexity o i-th rame calculated by (4. In the second step, we try to ind out what the unction o QP should be. It is well known that QP and Q step have the ollowing relationship: ( 4/6 Q 2 QP step = (0 where QP is the quantization parameter, Q step is the quantization step. According to (9 and (0, it can be derived that RC ( i, Qstep = α ( Qstep ( C i For a particular rame in the video sequence, the complexity o image is ixed. In order to determine the relation model between I rame bit rate and quantization step at a ixed image complexity, we choose the irst 300 rames o our standard WVGA ( test sequences and our standard WQVGA ( test sequences whose video eatures such as texture complexity and movement degree vary and encode them as I rame. Each sequence is encoded using QP ranging rom 22 to 48 at an interval o 4, and the statistic data o the 0th rame o each sequence are collected to ind out the rate-complexity-quantization (R-C-Q relationship. Fig.4 shows the distribution o bit rate and QP o the 0th rame in some sequence at dierent QPs. Q step and bpp/c in the Fig.4 are quantization step, the result o bpp divided image complexity measured by gradient, respectively. From Fig.4, we can observe that I rame bit rate its power unction with Q step very well at all quantization levels, which is expressed as ollows: bpp β = α ( Qstep (2 Ci where Q step is quantization step, α and β are model parameters. Based on our experiments, we have ound that typical values o β are in the range o to , and α in the range o to For simplicity, the moderate parameters are E-ISSN: Volume 2, 206

5 initialized as α = , β = During our parameter updating procedure, the value o β is ixed and the parameter α is updated by using: (3 where ε is a orgetting actor with typical value o ε = 0.5, C i is the image complexity o i-th rame calculated by (4. (a Fig.4 the relationship between Q step and bpp/c at dierent Q step. (a Class C, (b Class D. 4 Rate Control Strategy or P/B Frame 4. Frame Layer Bit Allocation Strategy In K003, hierarchical bit allocation algorithm is used or rate control. We use simple low delay coding structure to illustrate the rationale, which is shown in Fig.5. 4n+, 4n+3 and 4n+5 belong to the irst level; 4n+2 and 4n+6 belong to the middle level; 4n+4 and 4n+8 belong to the last level. The rame level target bit is assigned according to the weight o each rame, without considering the complexity o the image characteristic between rames. This might lead to the side eect o allocation too many bits to smooth areas while complex areas lack in bits. In order to overcome this shortcoming, we use the ormula (4 to calculate the complexity o the current rame, and the bit allocation strategy or P/B rame is deined as: (4 where T GOP is target bits o a GOP, N Slice is the GOP size, C Slice is the complexity o image, Coded GOP is the number o bits in the current GOP, ω SliceCur is the weight o picture level bit allocation or current picture. (b Fig.5 Pictures level under low delay coding structure 4.2 LCU Layer Bit Allocation Strategy In K003, LCU layer bit rate allocation is calculated according to the prediction error (in orm o mean absolute dierence (MAD o collocated LCU in the previous coded rame belonging to the same level, as Fig.5 shows. For example, i the current coding LCU is C which belongs to 4n+7, the MAD o the P in 4n+5 is used as the reerence o C. In addition, MAD is calculated by: (5 E-ISSN: Volume 2, 206

6 where N pixels is the number o pixels in the current LCU, P org and P pred are the pixel value o the original signal and the pixel value o the predicted signal, respectively. However, the prediction signal is inaccurate. On the one hand, the error prone P pred is optimal in the LCU sense rather than globally optimal. To be more speciic, the predicted LCU is obtained by traversing only available modes at depth zero and selecting the one leading to minimal rate distortion (RD cost, rather than by the normal LCU recursive coding strategy. However, the normal LCU recursive coding strategy checks the available modes at all depths and inds the optimal predicted LCU. On the another hand, We can t ind an accurate prediction block with a once motion estimation or LCU due to the act that the size o LCU is 64 64, and video content exist diverse motion object. Furthermore, in the process o inter rame coding, since HEVC has introduced a new coding technique which names merge mode on the basis o SKIP mode in H.264. SKIP or MERGE mode is used in the most o region within a rame, and this area is just not produce residual. Those areas only need a small amount o bits to transmit the index o encoding mode. Through the above analysis, we use the gradient o residual which is acquired ater accurately motion estimation o LCU to substitute MAD. Let res i,j be the pixel value o (i, j position in residual rame, and deine ε i as the gradient o residual LCU. The ormula is as ollows: H W ε = ( res res + res res i i, j i+, j i, j i, j+ H W i= 0 j= 0 (6 where W is width o LCU, H is height o LCU. Thus, the bit allocation strategy on the LCU layer is modiied to: T T Bit Coded Slice H Silce LCU = εi i NotCodedLCUs ε LCUCur (7 5 Experimental Results In order to veriy the proposed rate control algorithm, we conducted extensive experiments on HEVC reerence sotware HM.0. The experiments were conducted to the irst 300 rames o sixteen standard test sequences except the sequence o Kimono and ParkScene were coded 240 rames. The GOP size is set to 4. The motion search range is 64. The RDO and rate control are enabled. We use the test bit rates adopted by JCTVC-A204 [4] as the target bit rate and the rest o conigurations are detailed in [5]. In order to evaluate the accuracy o the proposed rate control algorithm, the bitrate mismatch is deined as ollows RCM = R R (8 actual t arg et where R actual and R target denote the actual coding bit rate and target bit rate, respectively. The rate control accuracy grows with the decreasing o the absolute value o RCM. We also used the standard deviation o the PSNR to measure the luctuation o the quality. PSNR std_deviation is deined as: PSNR std _ deviation = N i= ( PSNR PSNR i N 2 (9 where PSNR is the average o PSNR, PSNR i is the PSNR o the i-th rame, N is the number o rames to be encoded. A smaller value o PSNR std_deviation, indicates the scheme with a smaller o luctuation o the quality. Tables and 2 show the results o the experiment. From the table, we can observe that our proposed method is better than reerence [7]. The maximal and minimal values o RCM are 8.49kbits and 0.00kbits with our proposed method, respectively. While [7] brought about the maximal and minimal values o RCM are 52.04kbits and 0.0kbits, respectively. This means that image complexity is preerable to optimize the rame level bit allocation in the rame layer. The bit rate mismatch o Traic has increased slightly, the reason is that the sequence o Traic is taken by a ixed camera on the highway and the image characteristic is very similar in a GOP, leading to improved ineective. Compared with [7], the bit rate mismatch o Kimono is smaller duo to our proposed algorithm considered the image complexity in the process o bit allocation. In addition, the content o Kimono is that a lady walk on the road and the background is pretty complexity. Our proposed method will rational adjust the bit rate according to the dierent parts o the image complexity. E-ISSN: Volume 2, 206

7 Sequence type class A class B class C class D class E Table The results o bit rate mismatch Sequence Target bit Actual bit rate(kb/s RCM(kb/s rate(kb/s [7] Proposed [7] Proposed Traic PeopleOnStreet Cactus BasketballDrive Kimono ParkScene BasketballDrill BQMall PartyScene RaceHorsesC BasketballPass BlowingBubbles BQSquare RaceHorses FourPeople Johnny From the table 2, we can observe that the PSNR o our proposed method achieves a gain o up to 0.67dB, and average PSNR gain is 0.5dB compared with [7]. Although the PSNR o some sequence has decreased compared with the reerence method, but the deviation o the PSNR is small, and thereore the visual quality is not aected greatly. The reason why PSNR o PartyScene has increased by 0.67dB is that its complexity texture, the reerence method doesn t consider the image characteristic in the process o bit allocation, causing the poor coding quality. However, our proposed method gives ull consideration to the characteristics o the image in the intra rame bit allocation, and assign the redundant code bit to the subsequent coding rame while guarantee the quality o the intra rame. This strategy make the ollow-up coding rames have enough bit to code. In the coding process o P/B rame, the bit allocation or LCU has reerred to the gradient o residual belonging to collocated LCU. It will make more rational bit allocation and improve the quality o the encoding. The deviation o PSNR in [7] is averagely.52, but our proposed method is averagely.40. Fig.6 shows the luctuation o PSNR belonging to two sequences. From the Fig.6, we can observe that the deviation o PSNR has made an improvement, this is because the proposed algorithm can better allocate reasonable bit rate to rames according to the complexity o each rame. Sequence type class A class B class C class D Table 2 The results o PSNR and the deviation PSNR Sequence The deviation o Target bit PSNR(dB Delta PSNR rate(kb/s PSNR(dB [7] Proposed [7] Proposed Traic PeopleOnStreet Cactus BasketballDrive Kimono ParkScene BasketballDrill BQMall PartyScene RaceHorsesC BasketballPass BlowingBubbles BQSquare E-ISSN: Volume 2, 206

8 RaceHorses FourPeople class E Johnny Average [7] Proposed (a PSNR(dB Frame index 40 (a [7] Proposed (b 35 PSNR(dB Frame index (b Fig.6 The deviation o PSNR. (a PartyScene, (b RaceHorsesC. Fig.7 shows the quality o PartyScene at dierent rate control algorithm. 7(a and 7(b are the result o our proposed method and reerence method, respectively. From the igure 7(c and 7(d, we can obviously observe that our method has obtained a avorable visual quality. The obvious reason is that our proposed algorithm has considered image complexity in the process o bit allocation, i.e., the area which has ully image complexity will be assigned more bits than the lat region. (c (d Fig.7 The quality o 39th rame in PartyScene under dierent rate control. (a Reerence [7], (b Proposed method, (c The enlarged image o reerence [7], (d The enlarged image o proposed method 6 Conclusion Rate control plays a key role in video coding and communication systems. In this paper, we presented a novel bit allocation strategy on ame layer and LCU layer based on image complexity. In the rame layer, we established the bit allocation model or intra rame, and assigned a rational bit according the image complexity o ones in each GOP. In the LCU layer, we established quantization calculation model or LCU which is belong to intra rame, and assigned bit or LCU in P/B rame with a new strategy. Experimental results show that our proposed method has good perormance in the PSNR, the deviation o PSNR and the quality o coding. Acknowledgment: This work was supported by the national natural science oundation o China under Grant 6763, Grant , and Grant U30257, National Key Technology Research and E-ISSN: Volume 2, 206

9 Development Program o the Ministry o Science and Technology o China Grant 202BAH67F0. Reerences [] Sullivan, G. J., Ohm, J. R., Han, W. J., and Wiegand, T. Overview o the high eiciency video coding (HEVC standard. Circuits and Systems or Video Technology, 22(2, , 202. [2] Ohm, J. R., Sullivan, G. J., Schwarz, H., Tan, T. K., and Wiegand, T. Comparison o the coding eiciency o video coding standards including high eiciency video coding (HEVC. Circuits and Systems or Video Technology, 22(2, , 202. [3] Chen, Z., and Ngan, K. N. Recent advances in rate control or video coding. Signal Processing: Image Communication, 22(, 9-38, [4] Choi, H., Yoo, J., Nam, J., Sim, D., and Bajic, I. V. Pixel-wise uniied rate-quantization model or multi-level rate control. IEEE Journal o Selected Topics in Signal Processing, 7(6, 2-23, 203. [5] Liang, X., Wang, Q., Zhou, Y., Luo, B., and Men, A. A novel rq model based rate control scheme in HEVC. In Visual Communications and Image Processing (VCIP (pp. -6, 203. [6] Li, B., Li, H., Li, L., and Zhang, J. (202. Rate control by R-lambda model or HEVC. Jt. Collab. Team Video Coding (JCT-VC o ITU-T SG, 6, -. [7] Li, B., Li, H., Li, L., and Zhang, J. λ Domain Rate Control Algorithm or High Eiciency Video Coding. IEEE transactions on image processing: a publication o the IEEE Signal Processing Society, 23(9, , 204. [8] Wang, S., Ma, S., Wang, S., Zhao, D., and Gao, W. Rate-GOP based rate control or high eiciency video coding. IEEE Journal o Selected Topics in Signal Processing, 7(6, 0-, 203. [9] Meddeb, M., Cagnazzo, M., and Pesquet- Popescu, B. Region-o-interest-based rate control scheme or high-eiciency video coding. APSIPA Transactions on Signal and Inormation Processing, 3, e6, 204. [0] Wang, M., Ngan, K. N., and Li, H. An Eicient Frame-Content Based Intra Frame Rate Control or High Eiciency Video Coding. Signal Processing Letters, 22(7, , 205. [] He, Z., and Mitra, S. K. Optimum bit allocation and accurate rate control or video coding via ρ-domain source modeling. Circuits and Systems or Video Technology, 2(0, , [2] Lainema, J., Bossen, F., Han, W. J., Min, J., and Ugur, K. Intra coding o the HEVC standard. Circuits and Systems or Video Technology, 22(2, , 202. [3] BenHajyousse, A., and Ezzedine, T. Analysis o Residual data or High Eiciency Video Coding (HEVC. IJCSNS International Journal o Computer Science and Network Security, 2(8, 90-93, 202. [4] Baroncini, V., Ohm, J. R., and Sullivan, G. (200. Report o subjective test results o responses to the joint call or proposals (cp on video coding technology or high eiciency video coding (HEVC. ISO/IEC JTC/SC29/WG MPEG200 N, 275. [5] Bossen, F. (203. Common test conditions and sotware reerence conigurations. Doc. JCTVC-K00. In th Meeting: Joint Collaborative Team on Video Coding (JCT-VC o ITU-T SG (Vol. 6. E-ISSN: Volume 2, 206