Visual Quality Assessment using the IVQUEST software

Transcription

1 Visual Quality Assessment using the IVQUEST software I. Objective The objective of this project is to introduce students to automated visual quality assessment and how it is performed in practice by using the IVQUEST software. After finishing this assignment the student will be familiar with: Popular image quality metrics used in the IVQUEST software The image quality assessment process and available image quality databases Furthermore, students should be able to investigate and incorporate within IVQUEST more state-ofthe-art image quality metrics by themselves. II. Introduction to the IVQUEST Software The IVQUEST - Image and Video Quality Evaluation Software was developed at the Image, Video and Usability (IVU) Lab at Arizona State University and is available at: The IVQUET software includes 28 prominent quality metrics, including 12 full reference generic objective metrics, 1 reduced reference metric and 15 no reference metrics. Before using the software, please click on the Help button of the GUI interface and read the introduction of this software very carefully.

2 After clicking on the Help button, the help text, which includes the introduction of the software, the UI description, and the references of the supported metrics, will popup out. If you want to know how to run the software, please click on the Main UI Elements and How Tos. Please read these carefully. If you want to know more about the metrics, please click on the Supported metrics. The corresponding paper, code and developer s website are listed there. The IVQUEST software takes as input the MOS scores generated by the subjective test in addition to the reference and test images used for the subjective testing. It enables the user to select objective quality metrics to be applied to the selected input images. The software can then compute, in a batch processing

3 mode, the results for the selected objective metrics using the input images. The software can also perform correlation analysis on the obtained objective metric results using the input MOS in order to evaluate the performance of the chosen objective quality metrics. The IVQUEST software supports several performance evaluation tools including the Pearson linear correlation coefficient (PLCC), the Spearman rank order correlation coefficient (SROCC), root mean square error (RMSE), mean absolute error (MAE) and outlier ratio (OR). The Figure below shows the correlation analysis view of the IVQUEST software. III. Image Quality Databases 1. LIVE Image Database, Release 2 (available online at )

4 The source images used in this database include pictures of faces, people, animals, close-up shots, wide-angle shots, nature scenes, man-made objects, images with distinct foreground/background configurations, and images without any specific object of interest. The distortion types included: JPEG2000 compression: The reference images (in full color) were compressed using Kakadu reference software to have a bit rate from 0.28 bits per pixel (bpp) to 3.15 bpp. JPEG compression: The reference images (in full color) were compressed using Matlab command imwrite at bit rates 0.15 bpp to 3.34 bpp White noise: White Gaussian noise of standard deviation σ N is added to RGB components after scaling the components to between 0 and 1. The same σ N is used for R, G and B values. Value of σ N is varied from to 2.0. Gaussian Blur: R, G and B channels were filtered using circular-symmetric 2-D Gaussian kernel of standard deviation σ B pixels with the same kernel being used for each color component. σ B is varied from 0.42 to 15 pixels. Simulated Rayleigh Fading: Images were distorted by bit errors during transmission of compressed JPEG2000 bitstream over a simulated wireless channel. Receiver SNR was varied to generate bit streams corrupted with different proportion of bit errors. 2. Tampere Image Database (TID) 2008 (available online at ) In the Tampere Image Database, there are 25 reference images with 17 distortion types with 4 different levels of distortion for each type resulting in a total of 1700 images. The different types of distortions used in this database are as follows: Additive Gaussian noise Additive noise in color components is more intensive than additive noise in the luminance component Spatially correlated noise Masked noise High frequency noise Impulse noise Quantization noise Gaussian blur Image denoising JPEG compression

5 JPEG2000 compression JPEG transmission errors JPEG2000 transmission errors Non eccentricity pattern noise Local block-wise distortions of different intensity Mean shift (intensity shift) Contrast change IV. Requirements For this project, the students will be divided into groups consisting of two students, each. The tasks for each group are different from each other in terms of the objective metrics type, database and image distortion type. The table below shows the specific tasks for each group. Please consult the information under Project on Blackboard to see to which group you are assigned. If you are interested in working with a particular student in the class, please send an to Prof. Karam by date set by the instructor in class. If no is sent by that date, you will be assigned automatically to one of the groups. The requirements of this project include the following: 1. Investigating the papers and softwares for the image quality metrics used in the IVQUEST. 2. Investigating the image database to fully understand how the distorted images in the database are generated. 3. Using the IVQUEST software to assess the quality of distorted images in the given image databases; and finally generating the corresponding excel table, which lists the score of the chosen objective quality metrics for the tested images. 4. Using the IVOUEST software to evaluate the performance of the chosen objective quality metrics by performing correlation analysis on the obtained objective metric results and the subjective MOS scores given in the image database. At least four performance measures should be used. V. Submission Instructions Submit by the due date via Blackboard under Project a zipped folder named LastNames_Project1.zip, where LastNames are the last names of the members in your group. The submitted zipped folder should contain: 1) Excel files (each excel file table corresponding to one quality metric type and one distortion type; include the metric name and distortion type in the corresponding file name) in a folder called Results; 2) Report: Report should include title, authors names, introduction, description of adopted

6 metrics, results and discussion, conclusion, and references; report should be formatted as single-column, double-spaced, 12-point Times New Roman font. The following guidelines are suggested in preparing the final report: (a) Introduction: clearly identify the image quality assessment problem, the importance of it, and the areas of application, and report organization. (0.75 to 1 page) (b) Description of Adopted Metrics: Discuss the objective image quality metrics used in the IVQUET and compare them. For each group, students can focus on the objective quality metrics in the corresponding metric type. Each of the chosen quality metrics should be described briefly and compared by stating what are the main contributions of the considered metrics as compared to the others, what are the main ideas proposed and/or tools used (without details since the reader can refer to the referenced original paper for details), its advantages and disadvantages as compared to other metrics. The description should be concise. (3 to 5 pages) (c) Results: Show and discuss the results. More specifically, for each distortion type, the performances of all objective quality metrics are evaluated by at least four performance measures; and then show the results in a table and analyze and discuss these as part of the report. The following table shows an example; in your report, add the name of the used database in addition to the distortion type in the table caption. (2 to 5 pages). (d) Conclusion: discuss unsolved problems (things that still need to be solved/improved), impediments to further progress, and future directions for possible improvements (0.75 to 1 page).

7 Group Assignment Tasks Metric Type Database Distortion Type Group 1 Full Reference & Reduced Reference LIVE JPEG2000 compression & JPEG compression & Simulated Rayleigh Fading Group 2 Full Reference & Reduced Reference LIVE White noise & Gaussian Blur & Simulated Rayleigh Fading Group 3 No Reference & Reduced Reference LIVE JPEG2000 compression & JPEG compression & Simulated Rayleigh Fading Group 4 No Reference & Reduced Reference LIVE White noise & Gaussian blur & Simulated Rayleigh Fading Group 5 Full Reference & Reduced Reference TID Additive Gaussian noise & Additive noise in color component & Spatially correlated noise & Group 6 Full Reference & Reduced Reference TID Masked noise & High frequency noise & Impulse noise Group 7 Full Reference & Reduced Reference TID Quantization noise & Gaussian blur& Image denoising Group 8 Full Reference TID JPEG compression & JPEG2000 compression & JPEG transmission errors Group 9 Full Reference TID JPEG2000 transmission errors & Non eccentricity pattern noise & Local block-wise distortions of different intensity Group 10 Reduced Reference TID JPEG2000 transmission errors & Non eccentricity pattern noise & Local block-wise distortions of different intensity Group 11 Reduced Reference TID Additive Gaussian noise & Additive noise in color component & Spatially correlated noise Group 12 Reduced Reference TID Masked noise & High frequency noise & Impulse noise Group 13 Reduced Reference TID Quantization noise & Gaussian blur& Image denoising

8 Appendix 1: Instructions to Generate Input Excel Files for IVQUEST In order to use IVQUEST to test more than one image at a time, users need to generate an Excel file listing the names of test images and corresponding reference images. As an example, below are instructions for testing the mean-squared-error of six distorted images in IVQUEST. The Example Folder The example folder includes two subfolders: Test and Ref, which include the test images and the corresponding reference images, respectively, as shown below. Test Images Ref Images:

9 Instructions to generate an Excel file for Test Images follow: 1. Run IVQUEST. Choose Image, Full Reference and mean-squared error. Then Click on the Test files(s) button. 2. The File Browser window will pop-out. Browse the Test subfolder and all files in this folder will show in the Files in current directory. Then, add all the image files.

10 3. In the Save this File List window, select Save selected filenames in file for future testing and enter output file name. Note, users can choose the directory of the output file.

11 4. Click on Done. An Excel sheet, which lists all the files names, will be stored in the users selected directory.

12 Instructions to generate an Excel file for Reference Images 1. Repeat Steps 1 to 4 described above to generate the Excel sheet for the reference images. 2. Review and modify, if needed, the generated Excel file. in order to make sure that the number and order of the reference images are the same as the testing images as shown below. Final Test Excel Sheet Final Ref Excel Sheet

13 Instructions to Run 1. Run IVQUEST. Choose Image, Full Reference and mean-squared error. Then click on the Test files(s) button. 2. The File Browser will pop-out. Browse the Test subfolder and all files in this folder will show in the Files in current directory. Then, add all the test image files. Click on Done. 3. Click on the Test files(s) button again. Add the generated excel sheet for test images. Click on Done. 4. In IVQUEST, click on the Reference files(s) button. The File Browser will popout. Browse the Ref subfolder and all files in this folder will show in the Files in current directory. Then, add all the reference image files. Click on Done. 5. Click on the Reference files(s) button again. Add the generated excel sheet for reference images. Click on Done. 6. In IVQUEST, click on the Start button. The results will pop-out. You can also specify an Excel file to save the generated results.

14 Appendix 2: Tips for Running IVQUEST 1. If your MATLAB version is the student s use version, please include the wavelet toolbox in your Matlab library. 2. In the No-Reference correlation analysis, if the objective metric file (excel file) contains multiple metrics, you need to select all the same metric names as in the objective metric file in the GUI metric selection field. And then run the correlation analysis. For example, if the objective metric file contains 4 metric columns (nrjpeg, var, spectrum, cpbd), as shown in Fig. Appendix2_1, you need to select all four metric in the correlation analysis (box of selected metrics ), as illustrated in Fig. Appendix2_2. Fig. Appendix2_1. An example of objective metric file containing 4 metrics. Fig. Appendix2_2.In correlation analysis, select the same metrics as in the objective metric file.

15 3. In Reduced-Reference metric calculation, if the NSS (natural scene statistics) metric does not produce any output value, go to the folder IVQUESTR2\metrics\contributions\jp2knr_release\softwarerelease, change the extension of the file hist2d.dll to hist2d.dll_rename (make the.dll file inactive). Change the extension of hist2d.m_rename to hist2d.m. This will make the C version of the function hist2d(..) inactive, and we will calculate the NSS using the Matlab version of hist2d(..).