MULTIMEDIA PROCESSING PROJECT REPORT

Transcription

1 EE 5359 FALL 2009 MULTIMEDIA PROCESSING PROJECT REPORT RATE-DISTORTION OPTIMIZATION USING SSIM IN H.264 I-FRAME ENCODER INSTRUCTOR: DR. K. R. RAO Babu Hemanth Kumar Aswathappa Department of Electrical Engineering University of Texas at Arlington 1 P a g e

2 List of acronyms AVC CABAC CALVC D HDTV HVS I ITU JPEG JVT MPEG MSE MSSIM PSNR QP RD SDTV SSD SSIM Advanced Video Coding Context Adaptive Binary Arithmetic Coding context Adaptive Variable Length Coding Distortion High Definition Television Human Visual System Intra-frame International Telecommunication Union Joint Photographic Experts Group Joint Video Team Moving Picture Experts Group Mean Squared Error Mean Structural Similarity Index Measurement Peak to peak signal to noise Ratio Quantization Parameter Rate Distortion Standard Definition Television Sum of Squared Differences Structural Similarity Index Measurement 2 P a g e

3 ABSTRACT In the rate-distortion optimization for H.264 I-frame encoder [1], the distortion (D) is measured as the sum of the squared differences between the reconstructed and the original images, which is same as MSE. Although peak-to-peak signal-to-noise ratio (PSNR) and MSE are currently the most widely used objective metrics due to their low complexity and clear physical meaning, they are also widely criticized for not correlating well with human visual system (HVS) for a long time [2]. During past several decades a great deal of effort has been made to develop new image quality assessment based on error sensitivity theory of HVS, but only limited success has been achieved by the reason that the HVS has not been well comprehended.[2] Recently a new philosophy for image quality measurement was proposed, based on the assumption that the human visual system is highly adapted to extract structural information from the viewing field. It follows that a measure of structural information change can provide a good approximation to perceived image distortion. In this new theory, an item called structural similarity index (SSIM) including three comparisons is introduced to measure the structural information change. Experiments have shown that the SSIM index method is easy to implement and can better correspond with human perceived measurement than PSNR (or MSE)[4][8][9] The main idea of this project is to employ SSIM in the rate-distortion optimizations of H.264 I- frame encoder to choose the best prediction mode(s). The required modifications will be done on the JVT reference software JM92 program [3]. Results in terms of size of the compressed image, SSIM of the whole reconstructed image for H.264-JM92 software and the new method will be compared. 3 P a g e

4 INTRODUCTION MSE- MEAN SQUARED ERROR MSE is a signal fidelity measure. The goal of a signal fidelity measure is to compare two signals by providing a quantitative score that describes the degree of similarity/ fidelity or, conversely, the level of error/distortion between them. Usually, it is assumed that one of the signals is a pristine original, while the other is distorted or contaminated by errors. Suppose that x = { xi i = 1, 2,, N} and y = { yi i = 1, 2,, N} are two finite-length, discrete signals (e.g., visual images), where N is the number of signal samples (pixels, if the signals are images) and xi and yi are the values of the i th samples in x and y, respectively. The MSE between the signals x and y is 1 MSE(, ) ( x y ) N N 2 xy i i (1) i 1 In the MSE, we will often refer to the error signal ei,= xi yi, which is the difference between the original and distorted signals. If one of the signals is an original signal of acceptable (or perhaps pristine) quality, and the other is a distorted version of it whose quality is being evaluated, then the MSE may also be regarded as a measure of signal quality. A more general form is the lp norm is MSE is often converted into a peak-to-peak signal-to-noise ratio (PSNR) measure (2) 2 L PSNR 10log 10 (3) MSE where L is the dynamic range of allowable image pixel intensities. For example, for images that 8 have allocations of 8 bits/pixel of gray-scale, L = 2 1 = 255. The PSNR is useful if images having different dynamic ranges are being compared, but otherwise contains no new information relative to the MSE. WHY MSE [2]? The MSE has many attractive features: 1. It is simple. It is parameter free and inexpensive to compute, with a complexity of only one multiply and two additions per sample. It is also memoryless the squared error can be evaluated at each sample, independent of other samples. 2. It has a clear physical meaning it is the natural way to define the energy of the error signal. Such an energy measure is preserved after any orthogonal (or unitary) linear transformation, such as the Fourier transform (Parseval s theorem). The energy preserving property guarantees that the energy of a signal distortion in the transform domain is the same as in the signal domain. 4 P a g e

5 3. The MSE is an excellent metric in the context of optimization. Minimum-MSE (MMSE) optimization problems often have closed-form analytical solutions, and when they do not, iterative numerical optimization procedures are often easy to formulate, since the gradient and the Hessian matrix [2] of the MSE are easy to compute. 4. MSE is widely used simply because it is a convention. Historically, it has been employed extensively for optimizing and assessing a wide variety of signal processing applications, including filter design, signal compression, restoration, denoising, reconstruction, and classification. Moreover, throughout the literature, competing algorithms have most often been compared using the MSE/PSNR. It therefore provides a convenient and extensive standard against which the MSE/PSNR results of new algorithms may be compared. This saves time and effort but further propagates the use of the MSE. WHAT IS WRONG WITH MSE [2]? It is apparent that the MSE possesses many favorable properties for application and analysis, but the reader might point out that a more fundamental issue has been missing. That is, does the MSE really measure signal fidelity? Given all of its attractive features, a signal processing practitioner might opt for the MSE if it proved to be a reasonable signal fidelity measure. But is that the case? Unfortunately, the converse appears true when the MSE is used to predict human perception of image fidelity and quality. An illustrative example is shown in Figure 1[2], where an original Einstein image is altered by different types of distortion: a contrast stretch, mean luminance shift, contamination by additive white Gaussian noise, impulsive noise distortion, JPEG compression [16], blur, spatial scaling, spatial shift, and rotation. [FIG1] Comparison of image fidelity measures for Einstein image altered with different types of distortion. (a)reference image. (b) Mean contrast stretch. (c) Luminance shift. (d) Gaussian noise contamination. (e) Impulsive noise contamination. (f) JPEG compression [16] (g) Blurring. (h) Spatial scaling (zooming out). (i) Spatial shift (to the right). (j) Spatial shift (to the left). (k) Rotation (counter-clockwise). (l) Rotation (clockwise). [2] 5 P a g e

6 In figure 1, both MSE values and values of another quality index, the structural similarity (SSIM) index, are given. The SSIM index is described in detail later. Note that the MSE values [relative to the original image (a)] of several of the distorted images are nearly identical [images (b) (g)], even though the same images present dramatically (and obviously) different visual quality. Also notice that images that undergo small geometrical modifications [images (h) (i)] may have very large MSE values relative to the original, yet show a negligible loss of perceived quality. So a natural question is: What is the problem with the MSE? [2] IMPLICIT ASSUMPTIONS WHEN USING THE MSE[2] 1. Signal fidelity is independent of temporal or spatial relationships between the samples of the original signal. In other words, if the original and distorted signals are randomly reordered in the same way, then the MSE between them will be unchanged. 2. Signal fidelity is independent of any relationship between the original signal and the error signal. For a given error signal, the MSE remains unchanged, regardless of which original signal it is added to. 3. Signal fidelity is independent of the signs of the error signal samples. 4. All signal samples are equally important to signal fidelity. Unfortunately, not one of them holds (even roughly) in the context of measuring the visual perception of image fidelity. Dramatic visual examples of the failure of the MSE with respect to the veracity of these assumptions is demonstrated in Figure 2. 6 P a g e

7 [FIG2] Failure of the MSE metric. (a) An original image (top left) is distorted by adding independent white Gaussian noise (bottom left). In the top-right image, the pixels are reordered by sorting pixel intensity values. The same reordering process is applied to the bottom-left image to create the bottom-right image.. (b) Two original images (top left and top right) are distorted by adding the same error image (middle), which is fully correlated with the top-left image. The MSE between the two left images and between the two right images are the same, but the perceived distortion of the bottomright image is much stronger than that of the bottom-left image. (c) An original image (left) is distorted by adding a positive constant (top right) and by adding the same constant, but with random signs (bottom right). The MSE between the original and any of the right images are the same, but the right images exhibit drastically different visual distortions. (d) An original image (top left) is distorted by adding independent white Gaussian noise (top right). The energy distribution of the absolute difference signal (bottom left, enhanced for visibility), is uniform. However, the perceived noise level is space variant, which is reflected in the SSIM map (bottom right, enhanced for visibility).[2] In Figure 2(a), the bottom-left image was created by adding independent white Gaussian noise to the original image (topleft). In the top-right image, the spatial ordering of the pixels was changed (through a sorting procedure), but without changing any of the pixel values from those in the original. The bottom- right image was obtained by applying the same reordering procedure to the bottom-left image. Of course, the MSE between the two left images, and between the two right images, are identical. Yet, the bottom-right image appears significantly noisier than the bottomleft image the perceived visual fidelity of the bottom-right image is much poorer than that of the bottom-left image. Apparently, MSE Assumption 1 is not a good one when measuring the fidelity of images. This is an excellent example of the failure of the MSE (and all other lp metrics) to take into account the dependencies (textures, orderings, patterns, etc.) that occur 7 P a g e

8 between signal samples. Since image signals are highly structured the ordering of the signal samples carries important perceptual structural information about the contents of the visual scene this is a severe shortcoming of the MSE for image fidelity measurement. Figure 2(b) conveys a dramatic example of the failure of MSE Assumption 2. In this figure, the same error signal was added to both original images (top left) and (top right). The error signal was created to be fully correlated with the top-left image. Both distorted images have exactly the same MSE with respect to their originals, but the visual distortion of the bottom-right image is much stronger than that of the bottom-left image. Clearly, the correlation (and dependency) between the error signal and the underlying image signal significantly affects perceptual image distortion Figure 2(c) depicts the failure of the underlying MSE Assumption 3. In this figure, the first distorted image was obtained by adding a constant value to all pixels in the original image, while the second distorted image was generated by the same method, except that the signs of the constant were randomly chosen to be positive or negative. The visual fidelity of the two distorted images is drastically different. Yet, the MSE ignores the effect of signs and reports the same fidelity measure for both distorted images. Figure 2(d) demonstrates a particularly instructive example of both MSE Assumptions 1 and 4. Distorted image (top right) was created by adding independent white Gaussian noise to the original image (top left). Clearly, the degree of noise-induced visual distortion varies significantly across the spatial coordinates of the image. In particular, the noise in the facial (and other smooth-intensity) regions appears rather severe, yet is visually negligible in other regions containing patterns and textures. The perceived fidelity of the distorted image varies over space, although the error signal (bottom left) has a uniform energy distribution across space. Since all image pixels are treated equally in the formulation of the MSE, such image content-dependent variations in image fidelity cannot be accounted for. SSIM ALTERNATIVE APPROACH One recently proposed approach to image fidelity measurement, which may also prove highly effective for measuring the fidelity of other signals, is the SSIM index. The principal philosophy underlying the original SSIM approach is that the human visual system [3] is highly adapted to extract structural information from visual scenes. Therefore, at least for image fidelity measurement, the retention of signal structure should be an important ingredient. Equivalently, an algorithm may seek to measure structural distortion to achieve image fidelity measurement. Figure 3 [2] helps illustrate the distinction between structural and nonstructural distortions. In the figure, the nonstructural distortions (a change of luminance or brightness, a change of contrast, Gamma distortion, and a spatial shift) are caused by ambient environmental or instrumental conditions occurring during image acquisition and display. These distortions do not change the structures of images of the objects in the visual scene. However, other distortions (additive noise and blur and lossy compression) significantly distort the structures of images of the objects. If we view the human visual system as an ideal information extractor that seeks to identify and recognize objects in the visual scene, then it must be highly sensitive to the structural distortions 8 P a g e

9 and automatically compensates for the nonstructural distortions. Consequently, an effective objective signal fidelity measure should simulate this functionality. [FIG3] Examples of structural versus nonstructural distortions.[2] [FIG4] Block diagram of structural similarity measurement system [4] 9 P a g e

10 The system diagram of structural similarity measurement system is shown in Fig. 4. Suppose that x = { xi i = 1, 2,, N} and y = { yi i = 1, 2,, N} are two finite-length image signals, which have been aligned with each other (e.g., spatial patches extracted from each image). where N is the number of signal samples (pixels, if the signals are images) and xi and yi are the values of the i th samples in x and y, respectively If we consider one of the signals to have perfect quality, then the similarity measure can serve as a quantitative measurement of the quality of the second signal. The system separates the task of similarity measurement into three comparisons: luminance, contrast and structure. First, the luminance of each signal is compared. Assuming discrete signals, this is estimated as the mean intensity x 1 N N The luminance comparison function l( xy, ) is then a function of x and. y i 1 x i Second, we remove the mean intensity from the signal. In discrete form, the resulting signal ( x x ) corresponds to the projection of vector onto the hyperplane defined by (4) N i 1 X 0 (5) i where X= { X i i = 1, 2,, N} is a finite-length image signal. We use the standard deviation (the square root of variance) as an estimate of the signal contrast. An unbiased estimate in discrete form is given by 1 N ( x ) x i X N i 1 2 The contrast comparison c( xy, ) is then the comparison of σ and σy Third, the signal is normalized (divided) by its own standard deviation, so that the two signals being compared have unit standard deviation. The structure comparison s( xy, ) is implemented on these normalized signals [( x ) / ] and [( y ) / ] x x Finally, the three components are combined to yield an overall similarity measure where l( xy, ) f(, ) y S( x, y) f ( l( x, y), c( x, y), s( x, y )) (7) x y c( xy, ) f( x, y) s( x, y) f([( x ) / ],[( y ) / ]) x x y y An important point is that the three components are relatively independent. For example, the change of luminance and/or contrast will not affect the structures of images. In order to complete the definition of the similarity measure in (6), we need to define the three functions l( xy, ) c( xy, ) and s( xy, ) as well as the combination function f(.). We also would like the similarity measure to satisfy the following properties. y (6) 10 P a g e

11 2 C1 l( xy, ) (8) C1 x y 2 2 x y 2 2 C1 L K 1 (9) where L is the dynamic range of the pixel values (255 for 8-bit grayscale images), and K1 1 is a small constant. Similar considerations also apply to contrast comparison and structure comparison described later. Equation (7) is easily seen to obey the three properties of SSIM. Equation (7) is also qualitatively consistent with Weber s law, which has been widely used to model light adaptation (also called luminance masking) in the HVS. According to Weber s law [18],the magnitude of a just-noticeable luminance change I is approximately proportional to the background luminance for a wide range of luminance values. In other words, the HVS is sensitive to the relative luminance change, and not the absolute luminance change. Letting R represent the size of luminance change relative to background luminance, we can write R as Substituting R into (7) gives y R 1 x 2 x (10) 2(1 R) l( xy, ) 2 C (11) 1 1 (1 R) If we assume is small enough (relative to ) to be ignored, then is a function only of R, qualitatively consistent with Weber s law. The contrast comparison function takes a similar form 2 x y C 2 c( xy, ) (12) 2 2 C 2 x y 2 where C2 ( K2L ), and K 2 1. This definition again satisfies the three properties. An important feature of this function is that with the same amount of contrast change, this measure is less sensitive to the case of high base contrast than low base contrast. This is consistent with the contrast-masking feature of the HVS [2]. Structure comparison is conducted after luminance subtraction and variance normalization. Specifically, we associate the two unit vectors [( x ) / ] and [( y ) / ], each lying in the hyperplane defined by (3), with the structure of the two images. The correlation (inner product) between these is a simple and effective measure to quantify the structural similarity. x x y y 11 P a g e

12 Notice that the correlation between [( x ) / ] and [( y ) / ] is equivalent to the x correlation coefficient between x and y. Thus, we define the structure comparison function as follows: 2 xy C3 s( xy, ) (13) x y C3 As in the luminance and contrast measures, we have introduced a small constant C3 in both denominator and numerator. In discrete form, can be estimated as 1 N ( x )( y ) xy i x i y N i 1 Note also that s( xy, ) can take on negative values. Finally, we combine the three comparisons of (7), (10) and (11) and name the resulting similarity measure the SSIM index between signals and SSIM ( x, y) [ l( x, y)] [ c( x, y)] [ s( x, y )] (15) where α >1, β>1, and γ>1 are parameters used to adjust the relative importance of the three components. It is easy to verify that this definition satisfies the three conditions given above. In order to simplify the expression, we set α = β= γ=1 and C3 C 2/2. This results in a specific form of the SSIM index SSIM (, ) (2 x y C1)(2 xy C2) ( C1)( C2) (16) x y x y In practice, one usually requires a single overall quality measure of the entire image. We use a mean SSIM (MSSIM) index to evaluate the overall image quality: x y y (14) (17) where X and Y are the reference and the distorted images, respectively; x j and y j are the image contents at the j th local window; and M is the number of local windows of the image. To apply the SSIM index for image quality measurement it is preferable to apply it locally (on image blocks or patches) than globally (over the entire image). SSIM index is most commonly computed within the local window which moves pixel by pixel across the entire image. Such a sliding window approach is shown in figure 5 12 P a g e

13 H.264 Overview [FIG5] Sliding window approach for image quality assessment H.264/MPEG-4 AVC [5] is the newest video compression standard, which promises significant improvements over all previous video compression standards. The performance comparisons show that H.264 can achieve a coding efficiency improvement of about 1.5 times or greater for each test sequence related to multimedia, SDTV and HDTV compared to MPEG-2 [5]. The ITU-T name for the standard is H.264 while the ISO/IEC name is MPEG-4 advanced video coding (AVC), which is Part 10 of the MPEG-4 standard [5]. H.264/AVC CODING PROCESS The block diagram for H.264 coding is shown in Figure 7. Encoder may select between intra and inter coding for block-shaped regions of each picture. Intra coding can provide access points to the coded sequence where decoding can begin and continue correctly. Intra coding uses various 13 P a g e

14 spatial prediction modes to reduce spatial redundancy in the source signal for a single picture. Inter coding (predictive or bi-predictive) is more efficient using inter prediction of each block of sample values from some previously decoded pictures. Inter coding uses motion vectors for block-based inter prediction to reduce temporal redundancy among different pictures. During motion estimation, traditional codecs commonly process frames at the macroblock level (16 pixels by 16 pixels). H.264 can process on segments within a macroblock, ranging in size from the commonly used 16x16 to as small as 4x4 as shown in fig 6, which helps to code complex motion in areas of high detail. Prediction is obtained from deblocking filtered signal of previous reconstructed pictures. The deblocking filter is to reduce the blocking artifacts at the block boundaries. Motion vectors and intra prediction modes may be specified for a variety of block sizes in the picture. The prediction residual is then further compressed using a transform to remove spatial correlation in the block before it is quantized. Finally, the motion vectors or intra prediction modes are combined with the quantized transform coefficient information and encoded using entropy code such as context-adaptive variable length codes (CAVLC) or context adaptive binary arithmetic coding (CABAC). [FIG 6a] Macroblock partitions for motion estimation/motion compensation 16x16, 16x8, 8x16 and 8x8 [FIG6b] Macroblock sub-partitions for motion estimation/ motion compensation 8x8, 8x4, 4x8 and 4x4 14 P a g e

15 [FIG 7a] Block diagram of H.264 encoder [5] [FIG 7b] Block diagram of H.264 decoder [5] INTRA-PREDICTION H.264 uses the methods of predicting intra-coded macroblocks to reduce the high amount of bits coded by original input signal itself. For encoding a block or macroblock in Intra-coded mode, a prediction block is formed based on previously reconstructed (but, unfiltered for deblocking) blocks. The residual signal between the current block and the prediction is finally encoded. For 15 P a g e

16 the luma samples, the prediction block may be formed for each 4 x 4 subblock, each 8 x 8 block, or for a 16 x 16 macroblock. One case is selected from a total of 9 prediction modes for each 4 x 4 and 8 x 8 luma blocks; 4 modes for a 16 x 16 luma block; and 4 modes for each chroma blocks. Figure 8a shows a 4 x 4 luma block that is to be predicted. For the predicted samples [a, b,, p] for the current block, the above and left previously reconstructed samples [A, B,, M] are used according to direction modes. The arrows in Figure 7 indicate the direction of prediction in each mode. For mode 0 (vertical) and mode 1 (horizontal), the predicted samples are formed by extrapolation from upper samples [A, B, C, D] and from left samples [I, J, K, L], respectively. For mode 2 (DC), all of the predicted samples are formed by mean of upper and left samples [A, B, C, D, I, J, K, L]. For mode 3 (diagonal down left), mode 4 (diagonal down right), mode 5 (vertical right), mode 6 (horizontal down), mode 7 (vertical left), and mode 8 (horizontal up), the predicted samples are formed from a weighted average of the prediction samples A-M. For example, samples a and d are respectively predicted by round(i/4 + M/2 + A/4) and round(b/4 + C/2 + D/4) in mode 4, also by round(i/2 + J/2) and round(j/4 + K/2 + L/4) in mode 8. The encoder may select the prediction mode for each block that minimizes the residual between the block to be encoded and its prediction. [FIG 8a]. Intra 4 x 4 prediction mode directions (vertical : 0, horizontal : 1, DC : 2, diagonal down left : 3, diagonal down right : 4, vertical right : 5, horizontal down : 6, vertical left : 7, horizontal up : 8) [5] [FIG8b] Intra 16 X 16 prediction modes ( vertical:0 ; horizontal : 1, DC : 2; plane:3) [19] Mode 1 (horizontal): extrapolation from left samples (V). Mode 2 (DC): mean of upper and left-hand samples (H+V). Mode 4 (Plane): a linear plane function is fitted to the upper and left-hand samples H and V. This works well in areas of smoothly-varying luminance. 16 P a g e

17 H.264 AND I-FRAME ENCODER In H.264 I-frame encoder, each picture is partitioned into fixed-size macro blocks (MB) that cover a rectangular area of samples of the luma component and 8 8 samples of each chroma component(4:2:0 format). Figure 9, shows 3 formats known as 4:4:4, 4:2:2 and 4:2:0 video. 4:4:4 is full bandwidth YUV video, and each macroblock consists of 4 Y blocks, and 4 U/V blocks. Being full bandwidth, this format contains as much as data would if it were in the RGB color space. 4:2:2 contains half as much chrominance information as 4:4:4 and 4:2:0 contains one quarter of the chrominance information. The focus of this project is to use 4:2:0 format since it is the format typically used in video streaming application. [FIG 9] Different YUV formats[19] Each macroblock is spatially predicted using its neighbouring samples of previously coded blocks which are to the left and/or above the block, and the prediction residual is integer transformed, quantized and transmitted using entropy coding. The latest JVT reference software version (JM92) of H.264 [3] provides three types of intra predictions denoted as intra_16x16, intra_8x8 and intra_4x4. The intra_16x16 which supports 4 prediction modes performs prediction of the whole macroblock and is suited for smooth areas, while the intra_8x8 and intra_4x4 which performs 8 8 and 4 4 block respectively support 9 prediction modes and are suited for detailed parts of the picture. The best prediction mode(s) are chosen utilizing the R-D optimization which is described as: J ( s, c, MODE QP) D( s, c, MODE QP) r( s, c, MODE QP) (18) MODE 17 P a g e

18 In (18),the distortion D(s,c,MODE QP) is measured as sum of squared differences(ssd) between the original block s and the reconstructed block c, and QP is the quantization parameter, MODE is the prediction mode. r(s,c,mode QP) is number of bits for coding the block. The modes(s) with the minimum J(s,c,MODE QP) are chosen as the prediction mode(s) of the macroblock. R-D OPTIMIZATION USING SSIM As the SSIM index method performs better as image quality measurement than MSE (SSD), the main purpose is to replace the SSD with the SSIM index in the R-D optimization of H.264 I- frame encoder. The quality of the reconstructed picture is higher when its SSIM index is greater while the SSD performs the other way. Therefore the distortion is measured as: D(s, c,mode QP)= 1 SSIM(s, c) (19) where s and c are the original and reconstructed image blocks respectively. Due to the change in the distortion measure, the Lagrangian multiplier should be modified correspondingly. The new Lagrangian multiplier [6] in this algorithm is where QP denotes the quantization parameter. Consequently, the new R-D cost function can be written as: J ( s, c, MODE QP) 1 SSIM ( s, c) r( s, c, MODE QP) (21) The new algorithm is using SSIM index instead of SSD as the distortion measure in RDCost_for_4x4IntraBlock, RDCost_for_8x8IntraBlock and RDCost_for_macroblocks. The SSIM indexes of all types of prediction blocks are computed within 4 4 nonoverlapping square windows, while slide window, which is of 16 16, is used to compute the entire reconstructed image quality MSSIM (mean SSIM). Furthermore, the parameter setting here is chosen as follows: K1=0.01, K2=0.03, L=255. MODE (20) ALGORITHM The major steps for each macroblock selecting the best prediction mode(s) can be summarized as follows: Step 1: Choose one Intra_Chroma prediction mode and generate the intra prediction blocks for U and V component respectively. 18 P a g e

19 Step 2: Find the best Intra_16x16 prediction mode. a. Generate the four prediction blocks respectively for the Luma component ac-cording to the four Intra_16x16 prediction modes. b. Perform Hadamard transform[1], for the residual blocks and then sum up the absolute values of all the Hadamard transform coefficients as the cost. c. The mode that has the lowest cost is chosen as the best Intra_16x16 prediction mode. Step 3: Find the best Intra_4x4 prediction modes Divide the luma component of that macroblock into sixteen 4 4 non-overlapped blocks. For each 4 4 block do the following sub-steps: a. Generate nine prediction blocks based on the nine Intra_4x4 prediction modes as in [FIG 7] b. Compute the SSIM of the 4 4 reconstructed block and the original one. c. Calculate the cost of the 4 4 block according to equation (21). d. The mode that has the minimum cost is chosen as the best mode. Step 4: Find the best Intra_8x8 prediction modes Divide the luma component of that macroblock into four 8 8 non-overlapped blocks. For each 8 8 block do the following sub-steps: a. Generate nine prediction blocks based on the nine Intra_8 8 prediction modes. b. Compute the SSIM of the 8 8 reconstructed block and the original one. c. Calculate the cost of the 8 8 block according to equation (21). d. The mode that has the minimum cost is chosen as the best mode. Step 5: Find the best prediction mode for the whole macroblock a. Figure out the SSIM of the reconstructed and the original macroblock in best Intra_16x16 mode, the best Intra_4x4 modes and the best Intra_8x8 modes respectively. As each macroblock includes 16x16 pixels of Y component, 8x8 pixels of U component and 8x8 pixels of V component, we first count the SSIM of the reconstructed and the original blocks for each component and then combine them to a weighted averaged SSIM. Following weighted summation is used to generate the quality index for each macroblock by the reason that HVS is more sensitive to luma than chroma component. SSIMmb = 0.5 * SSIMy *(SSIMu + SSIMv) (22) where SSIMmb SSIM of the entire macro block SSIMy SSIM of Y component SSIMu SSIM of U component SSIMv SSIM of V component b. Calculate the rate-distortion cost using equation (21) for the best Intra_16x16 mode, the best Intra_4x4 modes and the best Intra_8x8 modes of the whole macrob-lock respectively. c. The mode having the minimum cost is chosen as the best prediction mode of that macroblock. Step 6: Repeat Step 1 to Ste 5 until all the Intra_Chroma prediction modes are used. 19 P a g e

20 SIMULATION RESULTS Simulations are carried out using several color video pictures of various sizes (as Table 1) in YUV format (4:2:0). All the simulations are based on the JVT reference software JM92 program [3]and conducted on a P4/2.0GHz personal computer with 256MB RAM and Microsoft Windows-XP as the operation system. MSSIM is used to indicate the quality of the entire reconstructed image. First we calculate the local SSIMs for the Y component by using slide window, which moves rightwards and downwards pixel by pixel. Then these local SSIMs are averaged into MSSIM Y. MSSIM U for U component and MSSIM V for V component are generated in the similar way while a 8 8 slide window is used instead. Finally, the MSSIM Y, MSSIM U and MSSIM V are combined into an overall image quality measurement MSSIM as equation (16) Table 1 : Test pictures [18] Akiyo QCIF Format Bridge(Close) Bridge(Far) 20 P a g e

21 Carphone Claire Coastgaurd Container Foreman 21 P a g e

22 Grandma Hall Monitor High Way Miss America Mobile 22 P a g e

23 Mother and Daughter News Salesman Silent Suzie Results in terms of total number of bits of the compressed image, MSSIM (a weighted average of Y, U, V component as formula (16)) of the whole reconstructed image and the comparison between the two methods are listed in Table 2~4 with the quantization parameter (QP) equal to 30, 20 and 10 respectively. 23 P a g e

24 Table 5 shows the original and reconstructed images produced by H.264 JM92 and H.264 JM92- SSIM methods respectively. Image Encoded Image size (# of Bits) H.264 JM92 H.264 JM92-SSIM Comparison(%) Encoded MSSIM Time Image Time #Bits MSSIM MSSIM (ms) Size (ms) Decrease Decrease (# of Bits) Time Increase coastguard Akiyo bridge-close Carphone Claire Container grandma Hall Highway miss-america mother-daughter News Salesman silent suzie Table 2. Results of comparison between H.264 JM92 and H.264 JM92-SSIM method for QP=30 24 P a g e

25 Image Encoded Image size (# of Bits) H.264 JM92 H.264 JM92-SSIM Comparison(%) Encoded MSSIM Time Image Time Bit MSSIM MSSIM (ms) Size (ms) Decrease Decrease (# of Bits) coastguard Akiyo bridge-close Carphone Claire Container grandma Hall Highway miss-america mother-daughter News Salesman silent suzie Table 3. Results of comparison between H.264 JM92 and H.264 JM92-SSIM method for QP=20 Time Increase Image Encoded Image size (# of Bits) H.264 JM92 H.264 JM92-SSIM Comparison(%) Encoded MSSIM Time Image Time Bit MSSIM MSSIM (ms) Size (ms) Decrease Decrease (# of Bits) Time Increase coastguard Akiyo bridge-close Carphone Claire Container grandma Hall Highway miss-america mother-daughter News Salesman silent suzie Table 4. Results of comparison between H.264 JM92 and H.264 JM92-SSIM method for QP=10 25 P a g e

26 Original Image Reconstructed Image JM92 Original QP=30 Reconstructed Image JM92-SSIM QP=30 MSSIM = MSSIM = MSSIM= MSSIM= MSSIM = MSSIM = MSSIM = MSSIM = P a g e

27 MSSIM = MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= MSSIM = MSSIM = P a g e

28 MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= P a g e

29 MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= MSSIM= Table 5: Original and reconstructed images produced by JM92 and JM92-SSIM methods respectively 29 P a g e

30 CONCLUSIONS Simulations show that the proposed method can reduce approximately 2~9% bit rate while maintaining almost the same perceptual quality and costing almost the same encoding time for QP=30, 4-20% bit rate reduction for QP=20, 18-35% bit rate reduction for QP=10. REFERENCES [1] Zhi-Yi Mai, et al A new-rate distortion optimization using structural information in H.264 I-frame encoder ACIVS 2005, LNCS 3708, pp , [2] Z. Wang and A. C. Bovik, Mean squared error: love it or leave it? - A new look at signal fidelity measures, IEEE Signal Processing Magazine, vol. 26, no. 1, pp , Jan [3] JM Software website: [4] Z. Wang, et al., Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Processing, vol. 13, no. 4, pp , Apr [Online] Available: [5] S.K. Kwon, A. Tamhankar and K.R. Rao Overview of H.264 / MPEG-4 Part 10 J. VCIR, Vol. 17, pp , April 2006, Special Issue on "Emerging H.264/AVC Video Coding Standard," [6] T. Wiegand and B. Girod, Lagrange multiplier selection in hybrid video coder control, in IEEE Int. Conf. on Image Processing, vol.3, pp , 2001 [7] T. Wiegand, et al Overview of the H.264/AVC video coding standard, IEEE Trans. on CAS for Video Technology, no.7, Vol. 13, pp , July [8] Z. Wang, A. C. Bovik and L. Lu, Why is image quality assessment so difficult? IEEE International Conference on Acoustics, Speech, & Signal Processing, pp , May [9] Z. Wang, L. Lu and A. C. Bovik, "Video quality assessment using structural distortion measurement", IEEE International Conference on Image Processing,, Vol: 3, Page(s): 65-68, Rochester, NY, September 22-25, 2002 [10] G. J. Sullivan and T. Wiegand., Rate-distortion optimization for video compression, IEEE Signal Processing Magazine, vol. 15, no. 6, pp , Nov [11] The SSIM Index for Image Quality Assessment [12] Z. Wang, and A. C. Bovik, A universal image quality index, IEEE Signal Processing Letters, vol. 9, no. 3, pp , March P a g e

31 [13] Z. Wang, L. Lu, and A. C. Bovik, Video quality assessment based on structural distortion measurement, Signal Processing: Image Comm., special issue on Objective video quality metrics, vol. 19, no. 2, pp , Feb [14]JPEG Reference website: [15] Jie Chen, et al., "WLD: A robust local image descriptor," IEEE Transactions on Pattern Analysis and Machine Intelligence, 06 Aug IEEE computer Society Digital Library. IEEE Computer Society [16] Z. Wang, and A. C. Bovik Modern image quality assessment (Synthesis Lectures on Image, Video, & Multimedia Processing) Morgan and Claypool Publishers, page 52 [17] I. Richardson, V-Codex, White Paper An overview of H.264 advanced video coding, 07 [18] Video test sequences link: [19] JVT Draft ITU-T recommendation and final draft international standard of joint video specification (ITU-T rec. H.264 ISO/IEC AVC), March 2003, JVT-G050 available on [20] Z. Wang, et al., "Objective video quality assessment," The Handbook of Video Databases: Design and Applications (B. Furht and O. Marqure, eds.), CRC Press, pp , Sept [21] The VC-1 and H.264 Video Compression Standards for Broadband Video Services By Hari Kalva, Jae-Beom Lee 31 P a g e