Objective Image Quality Assessment Current Status and What s Beyond

Transcription

1 Objective Image Quality Assessment Current Status and What s Beyond Zhou Wang Department of Electrical and Computer Engineering University of Waterloo 2015

2 Collaborators Past/Current Collaborators Prof. Alan C. Bovik, UT-Austin Prof. Eero P. Simoncelli, NYU Prof. David Zhang, HK PolyU Prof. Lei Zhang, HK PolyU Prof. Edward Vrscay, U Waterloo Prof. Guangzhe Fan, U Waterloo Prof. David Koff, McMaster U Prof. Jiying Zhao, U Ottawa Prof. Wen Gao, Peking U Prof. Siwei Ma, Peking U Prof. Weisi Lin, NTU Dr. Hamid R. Sheikh, TI Dr. Ligang Lu, IBM Dr. Mehul Sampat, UCSF Dr. Umesh Rajashekar, NYU Past/Current Students Qiang Li Xinli Shang Basavaraj Hiremath Gijesh Varghese Dominique Brunet Rania Hassen Kai Zeng Abdul Rehman Hojatollah Yaganeh Ilona Kowalik- Urbaniak Ehsan Nadernejad Rasoul Nasiri Shahrukh Athar Nima Nikvand Shiqi Wang Yang Gao Jiheng Wang Mohammad Rostami Ibrahim Al-Solami Tiesong Zhao Kede Ma Chris Redekop Yuming Fang Wentao Liu Ke Gu Qingbo Wu

3 Motivation original Image MSE=0, MSSIM=1 MSE=144, MSSIM=0.913 MSE=144, MSSIM=0.988 MSE=142, MSSIM=0.662 MSE=144, MSSIM=0.694 MSE=144, MSSIM=0.840

4 Objective Image Quality Assessment (IQA) Goal Automatically predict perceived image quality Classification Based on Availability of Reference Full-Reference (FR): reference (original) image available No-Reference (NR): reference image not available Reduced-Reference (RR): reference image partially available Classification Based on Application Scope General-purpose Application-specific

5 Perceptual Image Processing Overview IP Application... images Optimize IP Systems and Algorithms Define Perceptual Image Quality Measures compression enhancement rendering printing communication restoration Application Scope displaying watermarking data hiding denoising

6 Error Visibility Method: Idea distorted signal = reference signal + error signal Quantify error signal perceptually Representative work Frequency weighting (pioneering work) [Mannos & Sakrison 74] Sarnoff model [Lubin 93] Visible difference predictor [Daly 93] Perceptual image distortion [Teo & Heeger 94] DCT-based method [Watson 93] Wavelet-based method [Safranek 89, Watson et al. 97]

7 Error Visibility Method: Framework Reference signal Distorted signal Preprocessing Channel Decomposition... Error Normalization... Error Pooling Quality/ Distortion Measure Goal: simulate relevant early HVS components Structures motivated by physiology Parameters determined by psychophysics Problems Suprathreshold problem Natural image complexity problem

8 Structural Similarity Methods: Idea Purpose of vision: extract structural information Quantify structural distortion Questions: How to define structural/nonstructural distortions? How to separate structural/nonstructural distortions? [Wang & Bovik, IEEE Signal Proc. Letters, 02] [Wang, Bovik, Sheikh & Simoncelli, IEEE Trans. Image Processing, 04]

9 What are Structural/Non-Structural Distortions? original image non-structural distortions structural distortions luminance change contrast change blurring noise contamination Gamma distortion spatial shift JPEG blocking wavelet ringing

10 Separation of Structural/Non-Structural Distortions original image distorted image [Wang & Simoncelli, ICIP 05]

11 Separation of Structural/Non-Structural Distortions original image nonstructural distortion distorted image structural distortion [Wang & Simoncelli, ICIP 05]

12 Separation of Structural/Non-Structural Distortions original image + nonstructural distortion distorted image structural distortion [Wang & Simoncelli, ICIP 05]

13 Separation of Structural/Non-Structural Distortions original image + nonstructural distortion + distorted image structural distortion [Wang & Simoncelli, ICIP 05]

14 Adaptive Linear System nonstructural distortion original image structural distortion distorted image [Wang & Simoncelli, ICIP 05]

15 Adaptive Linear System nonstructural distortion nonstructural distortion components original image = c 1. + c c K. structural distortion structural distortion components distorted image = c. K c. K c M. [Wang & Simoncelli, ICIP 05]

16 Adaptive Linear System nonstructural distortion components original image c 1. + c c K. = structural distortion components distorted image + c. K c. K c M. overcomplete, adaptive basis in the space of all images [Wang & Simoncelli, ICIP 05]

17 Adaptive Linear System c 1. + c c K. = + c. K c. K c M. [Wang & Simoncelli, ICIP 05]

18 Adaptive Linear System For each, define a weighting factor [Wang & Simoncelli, ICIP 05]

19 Adaptive Linear System For each, define a weighting factor Find such that [Wang & Simoncelli, ICIP 05]

20 Direct solution: Adaptive Linear System Simplified solution: Split into adaptive and non-adaptive parts: New solution: [Wang & Simoncelli, ICIP 05]

21 Structural Similarity Index [Wang & Bovik, IEEE Signal Proc. Letters, 02] [Wang, Bovik, Sheikh & Simoncelli, IEEE Trans. Image Processing, 04] quality map original image distortion/similarity measure within sliding window distorted image pooling quality score

22 Gaussian noise corrupted image original image SSIM index map absolute error map

23 JPEG compressed image original image SSIM index map absolute error map

24 original Image Comparison MSE=0, MSSIM=1 MSE=144, MSSIM=0.913 MSE=144, MSSIM=0.988 MSE=142, MSSIM=0.662 MSE=144, MSSIM=0.694 MSE=144, MSSIM=0.840

25 Objective IQA What have been Achieved Successful General Approaches Psychological and physiological visibility models Structural similarity models Natural scene statistics (NSS) and information theoretic models Visual attention models Machine learning approaches Performance Evaluation Multiple public subject-rated image database available LIVE, IVC, Toyama, TID2008, TID2013, CSIQ, Correlation between subjective and objective scores FR, RR, and NR methods all achieved remarkable performance (high correlation coefficient numbers)

26 What s Next? Further improvement (FR, RR, NR) Accuracy, speed, pooling, geometric distortions, textures, Extended dimensionality Color, hyperspectral, 3D, video High dynamic range, high frame rate, Extended applications Biomedical, geoscience, Pattern recognition, object tracking Visual communications, video streaming* Perceptual optimization* Image restoration, enhancement, compression

27 Perceptual Optimization

28 Perceptual Optimization: Great Potentials The real value of objective IQA models Quality monitoring and algorithm/system comparisons Guiding algorithm/system design and optimization Great opportunity Few image processing algorithms/systems are carefully designed for optimal perceptual quality Great challenge Current IQA models are not developed for perceptual optimization Current work on how to optimize IQA measures is very limited

29 Wish List of a Nice Quality Metric Simple and low cost Valid distance metric Fixed point theories and methods, convergence properties Invariance under orthogonal transforms Pass between spatial and transform domain processes Differentiability Basis of many iterative algorithms; avoid singular points Convexity Uniqueness of solutions; allow for convex optimization methods L2 norm (MSE) satisfies all! SSIM the closest [Brunet, Vrscay & Wang, IEEE Trans Image Processing 12]

30 Example: MAD Competition MSE vs. SSIM [Wang & Simoncelli, Journal of Vision, 09]

31 Example: MAD Competition MSE vs. SSIM original image initial distortion initial image [Wang & Simoncelli, Journal of Vision, 09]

32 Example: MAD Competition MSE vs. SSIM original image best SSIM for fixed MSE worst SSIM for fixed MSE [Wang & Simoncelli, Journal of Vision, 09]

33 Example: MAD Competition MSE vs. SSIM original image best MSE for fixed SSIM best SSIM for fixed MSE worst MSE for fixed SSIM worst SSIM for fixed MSE [Wang & Simoncelli, Journal of Vision, 09]

34 original image initial distorted image best SSIM for fixed MSE worst SSIM for fixed MSE best MSE for fixed SSIM worst MSE for fixed SSIM

35 original image initial distorted image best SSIM for fixed MSE worst SSIM for fixed MSE best MSE for fixed SSIM worst MSE for fixed SSIM

36 HDR Tone Mapping vs. MEF HDR image Construction TMO MEF

37 IQA Problem: Which TMO is the Best? Reference image: HDR, Test image: LDR No effective solution in the literature

38 Tone Mapped Image Quality Index (TMQI) Tone mapped images and multi-scale quality maps indicating local structural detail loss and distortions [Yaganeh & Wang, IEEE Trans Image Proc. 13]

39 Application of TMQI: Optimal Design of TMOs Given HDR image X, find the Maximum Structural Fidelity (MSF) tone-mapped image: Assuming smooth behavior of the SF function, use gradient ascent method [Yaganeh & Wang, ICASSP 13, TIP 15]

40 [Yaganeh & Wang, ICASSP 13, TIP 15]

41 Application of TMQI: Parameter Tuning in TMO [Drago 03] b = 0.1, TMQI = 0.89 b = 0.8, TMQI = 0.90 b = 1, TMQI = 0.84 [Yaganeh & Wang, IEEE Trans Image Proc. 13]

42 Application of TMQI: Fusion of Tone Mapped Images TMO1, TMQI = 0.80 Local quality weighted averaging Fused image, TMQI = 0.94 TMO2, TMQI = 0.87 [Yaganeh & Wang, IEEE Trans Image Proc. 13]

43 IQA Problem: Which MEF is the Best?

44 MEF-IQA Multi-exposure fused images and multi-scale quality maps indicating local structural detail loss and distortions [Ma, Zeng & Wang, 15]

45 initial image by Gamma (2.2) mapping 10 iterations 30 iterations 100 iterations 50 iterations [Ma, Yaganeh, Zeng & Wang, 14]

46 Application of MEF-IQA: Optimal Design of MEF Algorithms by [Song, TIP 12], one of the top performers in the literature by proposed method [Ma & Wang, TIP 15]

47 Applications to Visual Communications

48 From Quality to Quality-of-Experience (QoE) Video Hosting Server video stream Network Consumer Device video M M Video Quality Assessment Video QoE Assessment Limitations of VQA (PSNR, SSIM ) Network condition not considered Receiving device (speed, power, memory, ) not considered Display device not considered Display resolution not considered Viewing condition/environment not considered User expectation not considered

49 Current approach Adaptive Streaming (AS) At the hosting server, prepare multiple layers of the same video content (different resolutions, different bandwidths) At the player, optimally switch between layers Current strategy Switch to the best quality affordable layer (under constraints of network bandwidth, decoding speed, buffer size, power, etc.) What s wrong with the current AS systems? Use video bit rate as the key quality indicator But bit rate is not Quality, not to say QoE!

50 Smart Streaming (SS) Key difference from current AS: Bit rate-driven QoE-driven Why does SS make a difference from AS? Video QoE is content-dependent Video QoE is device and display resolution dependent

51 Main Challenge: A Meaningful QoE Model Desirable properties High accuracy High speed Meaningful assessment across display device/viewing condition Meaningful assessment across display resolution Meaningful assessment across content Localized quality indicator Objective models PSNR, SSIM, MS-SSIM, MOVIE, VQM don t work well A new model, named SSIMplus, satisfies all desirables

52 Concluding Remarks Objective IQA Exciting achievement, but more to do Desired Showcase the value of objective IQA research Great Opportunities and Challenges Perceptual optimization Extended applications

53 THE END Thank you!