A Simple Second Derivative Based Blur Estimation Technique Thesis Presented in Partial Fulfillment of the Requirements for the Degree Master of Science in the Graduate School of The Ohio State University By Gourab Ghosh Roy, B.E. Graduate Program in Computer Science & Engineering The Ohio State University 2013 Thesis Committee: Brian Kulis, Advisor Mikhail Belkin
Copyright by Gourab Ghosh Roy 2013
Abstract Blur detection is a very important problem in image processing. Different sources can lead to blur in images, and much work has been done to have automated image quality assessment techniques consistent with human rating. In this work a no-reference second derivative based image metric for blur detection and estimation has been proposed. This method works by evaluating the magnitude of the second derivative at the edge points in an image, and calculating the proportion of edge points where the magnitude is greater than a certain threshold. Lower values of this proportion or the metric denote increased levels of blur in the image. Experiments show that this method can successfully differentiate between images with no blur and varying degrees of blur. Comparison with some other state-of-the-art quality assessment techniques on a standard dataset of Gaussian blur images shows that the proposed method gives moderately high performance values in terms of correspondence with human subjective scores. Coupled with the method s primary aspect of simplicity and subsequent ease of implementation, this makes it a probable choice for mobile applications. ii
Acknowledgements This work was motivated by involvement in the personal analytics research group with Dr. Mikhail Belkin, Dr. Simon Dennis, Dr. Jihun Hamm and other group members. iii
Vita 2004...Nava Nalanda High School 2006...South Point High School 2010...B.E. Electronics & Tele-Communication Engineering, Jadavpur University 2010-present... Fellow/Graduate Teaching Associate/ Graduate Research Associate, Computer Science and Engineering Department, The Ohio State University Publications G. G. Roy, S. Das, P. Chakraborty, and P.N. Suganthan. Design of Non-uniform Circular Antenna Arrays using a Modified Invasive Weed Optimization Algorithm. iv
IEEE Transactions on Antennas and Propagation, vol. 59, Issue 1, pp. 110-118, Jan. 2011. P. Chakraborty, G. G. Roy, S. Das, D. Jain and A. Abraham. An Improved Harmony Search Algorithm with Differential Mutation Operator. Fundamenta Informaticae Journal, vol. 95, Issue 4, pp. 401-426, Dec. 2009. G. G. Roy, P. Chakraborty, S. Das. Designing Fractional-order PIλDμ Controller Using Differential Harmony Search Algorithm. International Journal of Bio-Inspired Computation, vol. 2, No. 5, pp. 303-309, Oct. 2010. P. Chakraborty, G. G. Roy, B.K.Panigrahi, R.C.Bansal and A. Mohapatra. Dynamic Economic Dispatch Using Harmony Search algorithm with Modified Differential Mutation Operator. Electrical Engineering, vol. 94, Issue 4, pp. 197-205, Dec. 2012. T.K. Gandhi, P. Chakraborty, G. G. Roy and B.K.Panigrahi. Discrete harmony search based Expert model for epileptic seizure detection in electroencephalography. Expert Systems with Applications, vol. 39, Issue 4, pp. 4055-4062, March 2012. Fields of Study Major Field: Computer Science and Engineering v
Table of Contents Abstract... ii Acknowledgments... iii Vita... iv List of Tables... vii List of Figures... viii Introduction... 1 Previous Work... 3 Description of the method... 5 Results... 9 Discussion... 16 Conclusion... 18 References... 19 vi
List of Tables Table 1. Comparison of Spearman Rank Order Correlation Coefficient Values for LIVE database images with Gaussian blur...14 vii
List of Figures Figure 1. Plot of intensity along the vertical direction for the same edge point in the same image, without blur (red) and with blur (green)..7 Figure 2. (a) Original image (b) Image blurred with Gaussian filter of size 3*3, and sigma=6 (c) Image blurred with Gaussian filter of size 5*5, and sigma=10 (d) Image blurred with Gaussian filter of size 6*6, and sigma=12....10 Figure 3. (a) Original image (b) Image blurred with Gaussian filter of size 3*3, and sigma=6 (c) Image blurred with Gaussian filter of size 5*5, and sigma=10 (d) Image blurred with Gaussian filter of size 6*6, and sigma=12...12 Figure 4. Bar graph corresponding to Table 1, numbering order of methods same as in the table, red bar denotes the proposed method......15 viii
Introduction Image quality assessment is a key concept in the field of image and video processing. Many computer vision applications deal with automated assessment of the perceptual quality of an image. Probably the best possible way to obtain such an assessment is to have humans provide subjective rating of the images, but even for a slightly large-scale application this is not a practical option. There lies the significance of such quality assessment methods. The objective is to have such methods mimic human quality assessment. Out of the several types of image distortions, one of the most common types is image blur. Several factors can lead to an image being blurred during capture. One such case to consider is the associated motion while capturing images with a wearable camera like one in a cell phone. There has been a lot of work in the field of blur detection and estimation. The no-reference blur estimation techniques which do not use any knowledge of the original image are of primary importance. In this thesis work, one such no-reference blur detection technique is proposed. It uses the notion of second derivative at edge points in an image. The report is organized as follows: First some idea of previous work in blur detection is given, followed by the description of the proposed method. Then the results of this 1
proposed method are presented which includes a comparison with some other blur detection techniques on standard dataset images with Gaussian blur. Then there is the discussion section where the significance of the proposed method is highlighted, followed by conclusion and references. 2
Previous Work Much work has been done in the field of blur detection. Broadly, the blur detection algorithms can be divided into three categories. One is the class of full-reference algorithms, which use information from the original reference image in estimating the amount of blur in the blurry image. The other category is the class of no-reference algorithms, which do not use any information of the reference image and give an absolute metric value for the blurry image. From the practical point of view, no-reference algorithms are more significant because the original reference image is not always available in real situations. There is also a third category of reduced-reference algorithms which use only part of the information from the reference image. Another important point to mention in discussing about previous work is the fact that analysis of edges in an image has been widely used in blur estimation. Edges are the high frequency components in an image and as such, are most affected by the process of blur. Here a brief overview of the quality assessment algorithms to which the proposed method has been compared in performance is given. In [1], the authors propose a new spectral moment autofocus measure. An image quality measure is developed from the digital image power spectrum of normally acquired arbitrary scenes in [2]. An edge based blur detection method was proposed in [3] where the authors create an edge profile by 3
surrounding edge pixels with blocks and use kurtosis of those blocks for sharpness measure. A wavelet-based sharpness metric with immunity to noise in the image was presented in [4]. A gradient based image quality assessment method using blur and noise was proposed in [5]. The authors of [6] used a Harr wavelet transform based blur detection scheme. [7] obtains improved results in blur detection by edge analysis through multi-resolution decomposition. [8] proposed image comparison measures developed on ideas from fuzzy set theory. The metric of Blockwise Spectral distance measure for quality assessment was proposed in [9]. [10] was based on [11] where a visual discrimination model for image quality rating was presented. [12] developed a visual quality model based on the idea of Discrete Cosine Transform. A picture quality scale based on properties of visual perception for both global and local features was presented in [13]. The authors of [14] developed a distortion measure and noise quality measure for the effects of frequency distortion and additive noise respectively. [15] proposed a multiscale structural similarity method extracting structural information to measure perceived image quality. A new information fidelity criterion based on a model of natural scene statistics was used for image quality assessment in [16]. [17] proposed a visual information fidelity measure with improved performance by quantifying the loss of image information to the distortion process. 4
Description of the method The first step of the proposed method involves the application of the Canny edge detector [18] to an image. It is a standard edge detector method, which can be outlined as follows. a) First the image is smoothed with a Gaussian filter to remove noise. b) Horizontal and vertical components of the gradient are evaluated at each pixel location using a standard Sobel operator. Then those two are combined to calculate the pixel gradient amplitude and direction. c) Pixels with a local maximum of gradient amplitude in the direction of its gradient, considering the two neighboring pixels in that direction, are found. These are the edge pixels. d) Edges are traced in the image by performing thresholding with hysteresis which uses two thresholds. The method for blur detection proposed in this work consists of the following steps. 1) First the Canny edge detector is applied to an image to detect the edge pixels in that image. 2) For all locations in the image that are classified as edge points by step 1, the second derivatives along the horizontal and vertical directions are calculated. The 5
calculation is done based on how second derivatives are defined in discrete space, with little modification of the point of reference. Actually the second derivative in a particular direction is evaluated at the neighboring point of the edge point, which is evident from the following expressions. For the horizontal case denoted by the index x, the second derivative is calculated as G xx (x,y) = I(x,y)-2I(x-1,y)+I(x-2,y) whereas for the vertical case denoted by index y, it is calculated as G yy (x,y) = I(x,y)-2I(x,y-1)+I(x,y-2) where I(x,y) denotes the pixel value intensity at edge point location (x,y). One thing which should be mentioned here is that since step 1, the images have been converted to grayscale if they are in the RGB space. This conversion does not affect the performance of blur estimation; it is just that this makes the calculation of the metric simpler. 3) Then the second derivative values along two directions are combined to get the amplitude of the modified second derivative at the edge point. G 2 (x,y) = (G xx (x,y) 2 + G yy (x,y) 2 ) 1/2 This is done for all edge points. 4) The ratio of edge points that have these second derivative amplitudes greater than a particular threshold is calculated. This is the proposed image metric. The intuition behind this metric is that for a good quality image without blur, the second derivative amplitude defined as above is supposed to be high at most edge points. For images with blur where the affected edges have lost their sharpness and have been kind 6
of smeared, the second derivative amplitudes at those corresponding edge points will be decreased in comparison. An instance of an edge point where this difference in second derivative amplitudes can be visualized is given in Figure 1. Intensity of the same edge point and its two neighbors in the vertical direction has been plotted for a good quality image and a blurry version of the same image. Here the edge pixel is denoted by location 0 and its two neighbors by locations -1 and -2 as in step 2. The second derivative value is much larger in case of the non-blurry image compared to the blurry image. Figure 1. Plot of intensity along the vertical direction for the same edge point in the same image, without blur (red) and with blur (green) Based on this idea, the ratio of edge points that lie above a particular threshold, as mentioned in step 4, is chosen as the proposed metric. A higher value denotes an image without blur, whereas the value of this metric is low in images with blur. 7
A lot depends on how this threshold in step 4 is chosen. A well estimated guess is that it would be a function of the mean and/or the maximum of all the second derivative amplitude values. The threshold selected is max(a, max (b*max(vec), c*mean(vec))), where vec denotes the vector of second derivative amplitude values at all edge points. After experimenting with different sets of values, the optimal values of the parameters turn out to be a=30, b=0.225, c=2.25. This form of the threshold with these set of parameter values gives the best performance. 8
Results The performance of the proposed blur detection technique is first tested on some sample images. Gaussian blur has been introduced to an image, and the values of the proposed metric are observed. The results of those experiments are given in Figure 2. 9
(a) metric=0.0890 (b) metric=0.0472 (c) metric=0.0029 (d) metric=0.0019 Figure 2. (a) Original image (b) Image blurred with Gaussian filter of size 3*3, and sigma=6 (c) Image blurred with Gaussian filter of size 5*5, and sigma=10 (d) Image blurred with Gaussian filter of size 6*6, and sigma=12 As can be seen from the figures, for the original image, the metric value is the highest at 0.0890. On introducing a little amount of blurring, the metric value drops down to 0.0472. For increasingly larger amounts of blur, in (c) and (d), the metric value decreases to 0.0029 and 0.0019 respectively. So the proposed metric can successfully detect blur in images, as seen here by the clear differentiation in the metric values between the original 10
image and images with increasing levels of blur. One point to notice here is that the metric values can be scaled, but that does not affect the differentiation of blurry images from good quality images. A similar experiment is performed in Figure 3 on another image. Here too, the proposed metric gives clear distinction between the original image and the images with varying levels of Gaussian blur. As expected, the proposed metric value decreases with increasing extents of blur introduced in the image. 11
(a) metric=0.0759 (b) metric=0.0162 (c) metric=0.0052 (d) metric=0.0046 Figure 3. (a) Original image (b) Image blurred with Gaussian filter of size 3*3, and sigma=6 (c) Image blurred with Gaussian filter of size 5*5, and sigma=10 (d) Image blurred with Gaussian filter of size 6*6, and sigma=12 12
The performance of the proposed metric needed to be tested against some other state-ofthe-art blur detection algorithms. For that purpose of testing, the standard LIVE image quality assessment database has been used [19, 20 and 21]. The database consists of 29 original RGB images. The Gaussian blur distortion type is the focus of this experiment, so the 145 images of that particular distortion type are considered. A circular symmetric 2-D Gaussian kernel of standard deviation varying from 0.42 to 15 pixels is used for generation of these images with Gaussian blur. The database images have been assigned DMOS (Difference Mean Opinion Score) values after extensive subjective experiments, the details of which can be found in [20]. The metric of comparison is the Spearman rank order correlation coefficient (SROCC) between these DMOS values and the scores obtained from the image quality assessment algorithms after nonlinear regression as done in [20]. The performance of the other algorithms used for comparison is from [20] and [7]. The results are presented in Table 1. To have a better representation of how the proposed method stands in comparison to the other algorithms, a bar graph is also provided in Figure 4. The numbering order of methods in the graph is same as in Table 1. 13
No. Method SROCC Values 1 PSNR 0.7823 2 JND [10] 0.9389 3 DCTune [12] 0.6721 4 PQS [13] 0.9291 5 NQM [14] 0.8467 6 Fuzzy S7 [8] 0.6056 7 BSDM (S4) [9] 0.9600 8 SSIM(MS) [15] 0.9519 9 IFC [16] 0.9637 10 VIF [17] 0.9706 11 Frequency Threshold [1] 0.7305 12 IQM [2] 0.706 13 Kurtosis [3] 0.75 14 NIS [4] 0.754 15 Gradient based method [5] 0.8655 16 Wavelet based method [6] 0.7308 17 Edge analysis through MR decomposition [7] 0.8822 18 Proposed second-derivative based method 0.8846 Table 1. Comparison of Spearman Rank Order Correlation Coefficient Values for LIVE database images with Gaussian blur 14
Figure 4. Bar graph corresponding to Table 1, numbering order of methods same as in the table, red bar denotes the proposed method 15
Discussion From Figures 2 and 3, it is evident that the proposed metric can successfully differentiate between images with and without blur. On comparison with other image quality assessment algorithms in Table 1 and corresponding Figure 4, it is seen that that the method gives better performance than many existing methods, especially all those mentioned in [7]. There are some image quality assessment algorithms in [20] which give higher performance values on the standard LIVE dataset, with SROCC values as high as 0.97. But one very important point to note is the simplicity of the proposed method as outlined in its description, which leads to its ease of implementation. This simplicity and ease of use coupled with moderately high performance values contribute to the significance of this proposed blur estimation technique. So this holds promise for being successfully used in mobile applications which need automated detection of blur in captured images. The running time of the algorithm on an image depends on the number of edge points in the image. An estimate of the speed of the algorithm can be obtained by looking at the proposed method s run time averaged on all images in Figure 2 and all images in Figure 3 respectively. For all 698*515 images in Figure 2, the average time was 1.387 seconds in MATLAB running on an Intel i3 2.27 GHz machine with 4 GB DDR3 RAM having 64 16
bit Windows 7 installed. For all 512*512 images in Figure 3, in the same environment the average run time was 0.995 seconds. The use of second-derivative for blur estimation is not a concept which has been explored widely. However some recent work based on this concept has been done. In [22] profiles of second derivative in the direction of gradient are obtained on edge points and statistics associated with these profiles, for instance the distance between zero-crossing point and local extrema, are used for blur detection and estimation. But in the proposed method of this thesis, just the modified second derivative magnitudes at the edge points are used instead of looking at an entire profile, and this makes the overall method simpler in comparison. A suitable threshold based on these second derivative magnitudes for calculating the actual image metric for blur estimation has also been evaluated after experimentation. 17
Conclusion In this thesis work, a no-reference second derivative based blur estimation technique is presented. The simplicity of the method is its key factor. Comparison with some other quality assessment techniques on the Gaussian blur images of the LIVE database shows that the proposed method provides moderately high performance. These two factors combined make this method a good probable choice for mobile applications, something which can actually be tested in future work. 18
References [1] L. Firestone, Cook, N.Talsamia and K. Preston. Comparison of autofocus methods of automated microscopy. Citometry, vol. 12, pp. 195-206, 1991. [2] N. B. Nill, B. H Bouzas. Objective image quality measure derived from digital image power spectra. Opt. Eng. vol. 31, no. 4, pp. 813-825, Nov. 1992. [3] J. Caviedes, S. Gurbuz. No reference sharpness metric based on local edges Kurtosis. IEEE international conference on image processing, Rochester NY, 2002. [4] R. Fezli, L. J. Karem. No reference objective wavelet based noise immune image sharpness metric. IEEE International Conference on Image Processing. vol. 1, pp. 405-408, Sept. 2005. [5] M. G. Choi, J. H. Jung, and J. W. Jeon. No-Reference Image Quality Assessment using Blur and Noise, Word Academy of Science, Engineering and Technology, pp. 163-167, 50, 2009. [6] H. Hang, LI. Mingjing, H. J Zhang and C. Zhang. Blur Detection for images using Wavelet transform. Conference of multi-media and expositions, 2004. 19
[7] F. Kerouh, A. Serir. A No-Reference Blur Image Quality measure Based on Wavelet Transform. International Conference on Digital Information Processing and Communications, pp. 127-132, July 2012. [8] D. V. Weken, M. Nachtegael, and E. E. Kerre. Using similarity measures and homogeneity for the comparison of images. Image and Vision Computing, vol. 22, pp. 695 702, 2004. [9] I. Avcibas, B. Sankur, and K. Sayood, Statistical evaluation of image quality measures. Journal of Electronic Imaging, vol. 11, no. 2, pp. 206 23, Apr. 2002. [10] Sarnoff Corporation. JNDmetrix Technology. evaluation Version available: http://www.sarnoff.com/products services/video vision/jndmetrix/downloads.asp., 2003. [11] J. Lubin. A visual discrimination mode for image system design and evaluation. Visual Models for Target Detection and Recognition, E. Peli, Ed. Singapore: World Scientific Publishers, pp. 207 220, 1995. [12] A. B. Watson. DCTune: A technique for visual optimization of DCT quantization matrices for individual images. Society for Information Display Digest of Technical Papers, vol. XXIV, pp. 946 949, 1993. [13] M. Miyahara, K. Kotani, and V. R. Algazi. Objective Picture Quality Scale (PQS) 20
for image coding. IEEE Trans. Communications, vol. 46, no. 9, pp. 1215 1225, Sept. 1998. [14] N. Damera-Venkata, T. D. Kite, W. S. Geisler, B. L. Evans, and A. C. Bovik. Image quality assessment based on a degradation model. IEEE Trans. Image Processing, vol. 4, no. 4, pp. 636 650, Apr. 2000. [15] Z. Wang, E. P. Simoncelli, and A. C. Bovik. Multi-scale structural similarity for image quality assessment. Proc. IEEE Asilomar Conf. on Signals, Systems, and Computers, Nov. 2003. [16] H. R. Sheikh, A. C. Bovik, and G. de Veciana. An information fidelity criterion for image quality assessment using natural scene statistics. IEEE Trans. Image Processing, vol. 14, no. 12, Dec. 2005. [17] H. R. Sheikh and A. C. Bovik. Image information and visual quality. IEEE Trans. Image Processing, vol. 15, no. 2, Feb. 2006. [18] J. Canny. A computational approach to edge detection. IEEE Trans. PAMI, vol. 8, no. 6, pp. 679 698, Nov. 1986. [19] H.R. Sheikh, Z.Wang, L. Cormack and A.C. Bovik. "LIVE Image Quality Assessment Database Release 2". http://live.ece.utexas.edu/research/quality. 21
[20] H.R. Sheikh, M.F. Sabir and A.C. Bovik. "A statistical evaluation of recent full reference image quality assessment algorithms". IEEE Transactions on Image Processing, vol. 15, no. 11, pp. 3440-3451, Nov. 2006. [21] Z. Wang, A.C. Bovik, H.R. Sheikh and E.P. Simoncelli. "Image quality assessment: from error visibility to structural similarity". IEEE Transactions on Image Processing, vol.13, no.4, pp. 600-612, Apr. 2004. [22] M. Kostyukov and S. Fedorov. Detecting and Correcting Blur and Defocusing. http://www.faqs.org/patents/app/20120243792, Patent application number: 20120243792, Sep. 2012. 22