EVALUATION OF 60 FULL-REFERENCE IMAGE QUALITY METRICS ON THE CID:IQ. Marius Pedersen. Gjøvik University College, Gjøvik, Norway

Transcription

1 EVALUATION OF 60 FULL-REFERENCE IMAGE QUALITY METRICS ON THE CID:IQ Marius Pedersen Gjøvik University College, Gjøvik, Norway ABSTRACT Image quality metrics have become very popular and new metrics are proposed continuously. They have usually been developed with the goal of correlating with subjective image quality assessment. We perform an extensive evaluation of 60 state-of-the-art image quality metrics, including well-known metrics, such as SSIM, multiscale SSIM, VIF, MSE, S-DEE, CID, MAD, S-CIELAB, SHAME, VSNR, and PSNR. Evaluation is performed on the on the Colourlab Image Database: Image Quality (CID:IQ), a database consisting of 690 images where the subjective data has been collected at two different viewing distances. The performance of the image quality metrics is assessed in terms of correlation to subjective data. Index Terms image quality, metrics, full-reference 1. INTRODUCTION Many image quality metrics have been proposed in the literature; a survey and classification of more than 100 metrics was given by Pedersen and Hardeberg [1]. In order to assess the performance of these metrics they need to be evaluated against the percept. Evaluation of metrics have been carried out in the literature, but new metrics are often only compared against standard metrics as MSE and PSNR, or against a small set of metrics. A thorough benchmarking is necessary to determine the performance of of existing metrics. The viewing distance is very important for the impression of quality, noise in an image can, for example, be imperceptible at a certain viewing distance, but when viewed at a shorter distance the noise becomes perceptible. It is important that the metrics are capable of incorporating this feature of the human visual system. In order to achieve an extensive evaluation of the metrics, we have used the Colourlab Image Database:Image Quality (CID:IQ) [2], where subjective scores have been collected from two different viewing distances. This enables us to evaluate the robustness and stability of the metrics when the viewing distance is changed. Our goal is to carry out an extensive benchmarking of 60 full-reference image quality metrics (Table 1). To the best of our knowledge, this is the largest evaluation of image quality metrics carried out to date. The paper is organized as follows: first we introduce the experimental setup, then results and discussion, before we conclude and propose future work. Table 1. List of the 60 metrics calculated on the CID:IQ. HVS indicates if the metric has any simulation of the human visual system, c/g indicates a color or grayscale metric. Year Name Type HVS C/G - PSNR Image difference No Gray 1976 ΔEab Color difference No Color 1992 Linlab [3] Image fidelity Yes Color 1995 ΔE 94 Color difference No Color 1995 YCxCz [4] Image difference Yes Color 1996 S-CIELAB [5] Image difference Yes Color 1997 HVSREAL [6] Image Similarity Yes Color 1999 PDM [7] Image fidelity Yes Color 2000 NQM [8] Image quality No Gray 2000 WSNR [8] Image quality No Gray 2001 ΔE 00 Color difference No Color 2002 UIQI [9] Image quality No Gray 2002 Hue angle [10] Image difference No Color 2003 Q color [11] Image fidelity No Color 2003 MSSIM [12] Image quality No Gray 2004 SSIM [13] Image quality No Gray 2004 IFC [14] Image fidelity No Gray 2005 CWSSIM [15] Image similarity No Gray 2006 VIF [16] Image fidelity Yes Gray 2006 VIFP [16] Image fidelity Yes Gray 2006 PHVS [17] Image quality Yes Gray 2006 MSVD [18] Image quality No Gray 2006 SSIMipt [19] Image difference No Color 2006 QILV [20] Image quality No Gray 2007 PHVSM [21] Image quality Yes Gray 2007 VSNR [22] Image fidelity Yes Gray 2009 ΔE E [23] Color difference No Color 2009 S-DEE [24] Image difference Yes Color 2009 SHAME [25] Image quality Yes Color 2009 SHAME-II [25] Image quality Yes Color 2009 ABF [26] Image difference Yes Color 2009 WSSI [27] Image quality Yes Gray 2010 S DOG -CIELAB [28] Image quality Yes Color 2010 S DOG -DEE [28] Image quality Yes Color 2010 M DOG -DEE [29] Image difference Yes Color 2010 RFSIM [30] Image quality Yes Gray 2010 IWSSIM [31] Image quality No Gray 2010 IWMSE [31] Image quality No Gray 2010 IWPSNR [31] Image quality No Gray 2010 MAD [32] Image quality Yes Color 2010 DNIQM [33] Image quality Yes Color 2011 TVDSimple [34] Image quality Yes Color 2011 FSIM [35] Image quality Yes Gray 2011 FSIMc [35] Image quality Yes Color 2011 PSNRHA [36] Image quality Yes Gray 2011 PSNRHMA [36] Image quality Yes Gray 2011 colorpsnrha [36] Image quality Yes Color 2011 colorpsnrhma [36] Image quality Yes Color 2011 DCTex [37] Image quality Yes Color 2012 SRSIM [38] Image quality No Gray 2011 ADM [39] Image quality Yes Gray 2012 GSM [40] Image quality Yes Gray 2013 CID [41] Image difference Yes Color 2013 ESSIM [42] Image quality Yes Color 2013 SFF [43] Image quality Yes Color 2014 VSI [44] Image quality No Color 2014 WASH [45] Image quality No Gray 2014 GMSD [46] Image quality No Gray 2014 icid [47] Image difference Yes Color 2014 TVDShearlet [48] Image difference Yes Color

2 2. EXPERIMENTAL SETUP 2.1. Metrics and database Table 1 shows the list of metrics evaluated in this work. The metrics include pixel-wise metrics, spatial metrics, grayscale metrics, color metrics, etc. Color images have been converted for grayscale metrics using a weighted sum of R, G, and B components: R G B. All60 metrics in Table 1 have been calculated on the CID:IQ [2]. CID:IQ contains 23 original images, which have been modified with six distortions; JPEG2000 compression, JPEG compression, blur, Poisson noise, ΔE gamut mapping, and SGCK gamut mapping. All original images are applied these distortions in five levels from low degree of quality to high degree of quality degradation. 17 observers participated in the experiment, which was carried out at two viewing distances; 50 cm and 100 cm. The level of ambient illumination is approximately 4 lux. The chromaticity of the white displayed on the color monitor was D65 and luminance level of the monitor was 80 cd/m 2. All settings are suited for srgb color space Performance measures The performance of each metric is calculated as the correlation between subjective scores and the values calculated by the metric. We opted for three standard types of correlation. The Pearson s correlation coefficient, which assumes a normal distribution in the uncertainty of the data values and that the variables are ordinal. The Spearman s rank-correlation coefficient, which is a non-parametric measure of association based on the ranks of the data values, that describes the relationship between the variables without making any assumptions about the frequency distribution. Kendall s tau rank correlation coefficient, which is a non-parametric measure of correlation between two ranked variables. We calculate the linear correlation between the observer scores and the metric scores. The relationship between the metrics and subjective scores are not necessarily linear. Therefore, we also investigate the correlation using non-linear regression by applying a mapping function [16]: ( ) 1 f (x)=θ e θ + θ 2(x θ 3 ) 4 X + θ 5, (1) where θ i, i = 1,2,3,4, and 5 are the parameters to be be fitted. Initial parameters are max(sub. scores), min(sub. scores), median(metric scores), 0.1, and % confidence intervals are calculated using Fisher s z transformation [49]. 3. RESULTS AND DISCUSSION 3.1. Linear correlation We start by evaluating the linear correlation between the metric scores and observer scores. Figure 1 shows the linear Pearson correlation between the metrics and the observer scores for 50cm viewing distance. The highest correlation coefficient is obtained by the CID metric, but it is not statistically significantly different from other metrics, as WSSI, MAD, and colorpsnrha. Similar results are found for the linear Spearman correlation for 50cm, where CID still has a high correlation coefficient. It is worth noting that most of the metrics, such as multiscale SSIM (MSSIM) and IWS- SIM, have a higher Spearman correlation compared to Pearson. SDOGDEE has the highest increase in correlation from Pearson to Spearman, but it is fairly low (0.445). Kendall correlation coefficients are similar to those of Pearson and Spearman; CID has the highest correlation coefficient. For 100 cm viewing distance MAD has the highest Pearson correlation. Many other metrics also perform well, as ESSIM, GSM, CID, IWSSIM and more. Compared to 50 cm, NQM is the metric highest the highest increase in Pearson performance (0.485 for 50 cm and for 100 cm), while Qcolor has the largest decrease. colorpsnrha has the smallest difference in Pearson correlation between the two distances (0.736 for 50 cm and for 100 cm), which might indicate that it is adapting to the viewing distance. IWSSIM has the highest Spearman correlation, though not statistically significantly different from MAD, FSIM, FSIMc, and others. SDOGDEE is the metric with the highest increase in performance from Pearson to Spearman correlation (0.201 to 0.498). Kendall correlation coefficients are similar to Spearman, and IWSSIM has the highest Kendall coefficient Non-linear correlation Figure 2 shows that WSSI has the highest correlation coefficient with the non-linear fitting for 50 cm viewing distance. However, it cannot be distinguished from MSSIM, but it is significantly different from the other metrics. WSSI is significantly different from MAD, which was not the case for the linear analysis (Figure 1). Many of the metrics evaluated are based on each other, and it is interesting to notice that the color difference formula DEE has a statistically significantly better Pearson performance over its extension SDOGDEE. The same analysis can also be done for the TVDsimple and TVDShearlet. The metrics with the highest Pearson correlation are grayscale metrics. The best color metric is CID. IWSSIM has the highest non-linear Pearson correlation for 100cm. However, it is not significantly different from many other metrics, such as MAD, FSIMc, ADM, and VSI. WSSI, having highest Pearson correlation, is statistically significantly different compared to IWSSIM and MAD. Overall, some metrics (WSNR, IFC, and others) have problems with scale differences, where different images have been rated similar by observers but the metrics rate they clearly differently. Therefore they have an overall lower correlation coefficient compared to the best metrics. Also, some metrics (for example WSNR, UIQI, and IFC) give a high correlation within a single distortion, but scale differences between distortions, which results in an overall low correlation for all images and distortions.

3 1 Linear Pearson correlation values with a 95% confidence interval. Viewing distance 50 cm 0.8 Pearson Correlation PSNR SSIM FSIM FSIMc RFSIM SRSIM MSSSIM IWSSIM IWMSE IWPSNR UIQI VIF VSNR IFC NQM WSNR PHVSM PHVS PSNRHMA PSNRHA colorpsnrhma colorpsnrha CWSSIM DCTex CID icid LINLAB YCxCz WSSI MAD ABF Metrics Fig. 1. Linear Pearson correlation 50 cm. CID has the highest correlation coefficient, but it not statistically significantly different from many other metrics, such as MAD, WSSI, colorpsnrha, and VIF. PDM DEE SDEE SDOGDEE SDOGCIELAB MDOGDEE DNIQM HueAngle HVSREAL SCIELAB Qcolor SHAME SHAMEII SSIMipt MSVD DeltaEab DeltaE94 DeltaE00 WASH ADM GSM SFF GMSD ESSIM QILV VSI TVDSimple TVDShearlet VIFP 4. CONCLUSION We have calculated 60 full-reference image quality metrics on the Colourlab Image Database:Image Quality (CID:IQ), and evaluated the performance these metrics based on correlation with subjective data. Analysis of the correlation shows that many metrics perform well, among them the CID, WSSI, IWSSIM, MAD. The results show some differences between the linear and non-linear correlation. We also notice that the performance changes for the two different viewing distances in the CID:IQ. There is an indication that the grayscale metrics have higher correlation coefficients than color metrics. Future work include evaluation of the metrics on individual distortions, as well as evaluation on other databases. References [1] M. Pedersen and J. Y. Hardeberg, Full-reference image quality metrics: Classification and evaluation, Found. Trends. Comp. Graphics and Vision, vol. 7, no. 1, pp. 1 80, [2] X. Liu, M. Pedersen, and J.Y. Hardeberg, CID:IQ - a new image quality database, in Image and Signal Processing, A. Elmoataz, O. Lezoray, F. Nouboud, and D. Mammass, Eds., vol of Lecture Notes in Computer Science, pp Springer, Cherbourg, France, Jul [3] B. Kolpatzik and C. Bouman, Optimized error diffusion for image display, Journal of Electronic Imaging, vol. 1, no. 3, pp , [4] B. Kolpatzik and C. A. Bouman, Optimal universal color palette design for error diffusion, Journal of Electronic Imaging, vol. 4, no. 2, pp , Apr [5] X. Zhang and B. A. Wandell, A spatial extension of CIELAB for digital color image reproduction, in Soc. Inform. Display 96 Digest, San Diego,CA, 1996, pp [6] T. Frese, C. A. Bouman, and J. P. Allebach, A methodology for designing image similarity metrics based on human visual system models, in Human Vision and Electronic Imaging II, San Jose, CA, Feb 1997, vol. 3016, pp [7] N. Avadhanam and V. R. Algazi, Evaluation of a human-vision-system-based image fidelity metric for image compression, in Applications of Digital Image Processing XXII, A. G. Tescher, Ed., San Jose, CA, USA, 1999, vol of Proc. SPIE, pp [8] N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik, Image quality assessment based on a degradation model, IEEE Trans. Image Processing, vol. 9, pp , [9] Z. Wang and A. C. Bovik, A universal image quality index, IEEE Signal Processing Lett,, vol. 9, pp , [10] G. Hong and M. R. Luo, Perceptually based colour difference for complex images, in 9th Congress of the International Colour Association, R. Chung and A. Rodrigues, Eds., Rochester, NY Jun 2002, vol. 4421, pp [11] A. Toet and M. P. Lucassen, A new universal colour image fidelity metric, Displays, vol. 24, pp , [12] Z. Wang, E. P. Simoncelli, and A. C. Bovik, Multiscale structural similarity for image quality assessment, in Asilomar Conference on Signals, Systems and Computers, Pacific Grove, CA, Nov 2003, vol. 2, pp , IEEE.

4 1 Non linear Pearson correlation values with a 95% confidence interval. Viewing distance 50 cm 0.8 Pearson Correlation PSNR SSIM FSIM FSIMc RFSIM SRSIM MSSSIM IWSSIM IWMSE IWPSNR UIQI VIF VSNR IFC NQM WSNR PHVSM PHVS PSNRHMA PSNRHA colorpsnrhma colorpsnrha CWSSIM DCTex CID icid LINLAB YCxCz Metrics WSSI MAD Fig. 2. Non-linear Pearson correlation 50 cm. WSSI has the highest Pearson correlation coefficient, but it is not statistically significantly different from MSSIM. The highest performing color metric is CID. ABF PDM DEE SDEE SDOGDEE SDOGCIELAB MDOGDEE DNIQM HueAngle HVSREAL SCIELAB Qcolor SHAME SHAMEII SSIMipt MSVD DeltaEab DeltaE94 DeltaE00 WASH ADM GSM SFF GMSD ESSIM QILV VSI TVDSimple TVDShearlet VIFP [13] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE Trans. Image Processing, vol. 13, no. 4, pp , [14] H. R. Sheikh, A. C. Bovik, and G. de Veciana, An information fidelity criterion for image quality assessment using natural scene statistics, IEEE Trans. Image Processing, vol. 14, no. 12, pp , Dec [15] Z. Wang and E. P. Simoncelli, Translation insensitive image similarity in complex wavelet domain, in Int. conference on acoustics, speech and signal processing, Philadelphia, PA, Mar 2005, vol. 2, pp , IEEE. [16] H. R. Sheikh and A. C. Bovik, Image information and visual quality, IEEE Trans. Image Processing, vol. 15, no. 2, pp , [17] K. Egiazarian, J. Astola, N. Ponomarenko, V. Lukin, F. Battisti, and M. Carli, Two new full-reference quality metrics based on HVS, in Proceedings of the Second International Workshop on Video Processing and Quality Metrics, Scottsdale, USA, Jan 2006, pp [18] A. Shnayderman, A. Gusev, and A. M. Eskicioglu, An SVD-based grayscale image quality measure for local and global assessment, IEEE Trans. Image Processing, vol. 15, no. 2, pp , [19] N. Bonnier, F. Schmitt, H. Brettel, and S. Berche, Evaluation of spatial gamut mapping algorithms, in Color Imaging Conference, Scottsdale, AZ, Nov 2006, pp , IS&T/SID. [20] S. Aja-Fernandez, R.S.J. Estepar, C. Alberola-Lopez, and C.-F. Westin, Image quality assessment based on local variance, in Int. Conf. Engineering in Medicine and Biology Society, Aug 2006, pp [21] N. Ponomarenko, F. Silvestri, K. Egiazarian, M. Carli, J. Astola, and V. Lukin, On between-coefficient contrast masking of DCT basis functions, in Third International Workshop on Video Processing and Quality Metrics for Consumer Electronics VPQM-07, Scottsdale, AZ, Jan 2007, pp [22] D. M. Chandler and S. S. Hemami, VSNR: A waveletbased visual signal-to-noise ratio for natural images, IEEE Trans. Image Processing, vol. 16, no. 9, pp , Sep [23] C. Oleari, M. Melgosa, and R. Huertas, Euclidean color-difference formula for small-medium color differences in log-compressed OSA-UCS space, J. Opt. Soc. Am. A, vol. 26, no. 1, pp , [24] G. Simone, C. Oleari, and I. Farup, Performance of the euclidean color-difference formula in log-compressed OSA-UCS space applied to modified-image-difference metrics, in 11th Congress of the International Colour Association (AIC), Sydney, Australia, Sep/Oct [25] M. Pedersen and J. Y. Hardeberg, A new spatial hue angle metric for perceptual image difference, in Computational Color Imaging, Saint Etienne, France, Mar 2009, vol of Lecture Notes in Computer Science, pp , Springer Berlin / Heidelberg. [26] Z. Wang and J. Y. Hardeberg, An adaptive bilateral filter for predicting color image difference, in Color Imaging Conference, Albuquerque, NM, USA, Nov 2009, pp , IS&T/SID. [27] S. Rezazadeh and S. Coulombe, A novel approach for computing and pooling structural similarity index in the discrete wavelet domain, in International Conference on Image Processing, Nov 2009, pp

5 [28] S. A. Ajagamelle, M. Pedersen, and G. Simone, Analysis of the difference of gaussians model in image difference metrics, in 5th European Conference on Colour in Graphics, Imaging, and Vision (CGIV), Joensuu, Finland, Jun 2010, pp , IS&T. [29] G. Simone, V. Caracciolo, M. Pedersen, and F. A. Cheikh, Evaluation of a difference of gaussians based image difference metric in relation to perceived compression artifacts, in Int. Symposium Advances in Visual Computing, Las Vegas, NV, Nov 2010, Lecture Notes in Computer Science, pp , Springer. [30] L. Zhang, L. Zhang, and X. Mou, RFSIM: A feature based image quality assessment metric using riesz transforms, in Internatonal Conference on Image Processing, Hong Kong, Sep 2010, pp , IEEE. [31] F. Zhang, S. Li, L. Ma, and K. N. Ngan, Limitation and challenges of image quality measurement, in Visual Communications and Image Processing conference, P. Frossard, H. Li, F. Wu, B. Girod, S. Li, and G. Wei, Eds., Huang Shan, China, Jul 2010, vol of Proceedings of SPIE, pp [32] E. C. Larson and D. M. Chandler, Most apparent distortion: full-reference image quality assessment and the role of strategy, Journal of Electronic Imaging, vol. 19, no. 1, pp , Mar [33] V. Laparra, J. Mu noz Marí, and J. Malo, Divisive normalization image quality metric revisited, J. Opt. Soc. Am. A, vol. 27, no. 4, pp , Apr [34] M. Pedersen, G. Simone, M. Gong, and I. Farup, A total variation based color image quality metric with perceptual contrast filtering, in International conference on Pervasive Computing, Signal Processing and Applications, Gjøvik, Norway, Sep [35] L. Zhang, L. Zhang, X. Mou, and D. Zhang, FSIM: A feature similarity index for image quality assessment, IEEE Trans. Image Processing, [36] N. Ponomarenko, O. Ieremeiev, V. Lukin, K. Egiazarian, and M. Carli, Modified image visual quality metrics for contrast change and mean shift accounting, in 11th International Conference The Experience of Designing and Application of CAD Systems in Microelectronics, 2011, pp [37] F. Zhang, L. Ma, S. Li, and K. N. Ngan, Practical image quality metric applied to image coding, IEEE Trans. on Multimedia, vol. 13, no. 4, pp , Aug [38] L. Zhang and H. Li, SR-SIM: A fast and high performance iqa index based on spectral residual, in International Conference on Image Processing, Sep 2012, pp [39] S. Li, F. Zhang, L. Ma, and K. Ngi Ngan, Image quality assessment by separately evaluating detail losses and additive impairments, IEEE Trans. on Multimedia, vol. 13, no. 5, pp , Oct [40] A. Liu, W. Lin, and M. Narwaria, Image quality assessment based on gradient similarity, IEEE Trans. on Image Processing, vol. 21, no. 4, pp , [41] I. Lissner, J. Preiss, P. Urban, M. S. Lichtenauer, and P. Zolliker, Image-difference prediction: From grayscale to color, IEEE Trans. on Image Processing, vol. 22, no. 2, pp , Feb [42] X. Zhang, X. Feng, W. Wang, and W. Xue, Edge strength similarity for image quality assessment, IEEE Signal Processing Lett,, vol. 20, no. 4, pp , Apr [43] H-W. Chang, H. Yang, Y. Gan, and M-H. Wang, Sparse feature fidelity for perceptual image quality assessment, IEEE Trans. on Image Processing, vol. 22, no. 10, pp , Oct [44] L. Zhang, Y. Shen, and H. Li, Vsi: A visual saliencyinduced index for perceptual image quality assessment, IEEE Trans. on Image Processing, vol. 23, no. 10, pp , Oct [45] M. Reenu, D. David, S.S.A Raj, and M.S. Nair, Wavelet based sharp features (WASH): An image quality assessment metric based on hvs, in International Conference on Advanced Computing, Networking and Security (ADCONS), Dec 2013, pp [46] W. Xue, L. Zhang, X. Mou, and A. C. Bovik, Gradient magnitude similarity deviation: A highly efficient perceptual image quality index, IEEE Trans. on Image Processing, vol. 23, no. 2, pp , Feb [47] J. Preiss, F. Fernandes, and P. Urban, Color-image quality assessment: From prediction to optimization, IEEE Trans. on Image Processing, vol. 23, no. 3, pp , Mar [48] M. Pedersen, An image difference metric based on simulation of image detail visibility and total variation, in Color and Imaging Conference, Boston, Ma, Nov 2014, pp [49] Video Quality Experts Group, Final report from the video quality experts group: Validation of reducedreference and no-reference objective models for standard definition television, phase I, Tech. Rep., International Telecommunication Union, 2009.