Enhanced image saliency model based on blur identification

Transcription

1 Enhanced image saliency model based on blur identification R.A. Khan, H. Konik, É. Dinet Laboratoire Hubert Curien UMR CNRS 5516, University Jean Monnet, Saint-Étienne, France. Abstract Detection of visual saliency is of great interest for a lot of computer vision applications in particular for contentbased image retrieval. The work presented in this paper is devoted to develop an algorithm of saliency detection that performs adequately in predicting human fixations for stimuli containing blur and sharp regions. This work is based on an experimental study on the effect of blurriness on visual attention when observers see images with no prior knowledge in free viewing conditions. A ground-truth has been derived from this experimental study to test the saliency model we developed. Keywords: visual saliency, blur detection, colour image processing 1 Introduction For machine vision applications, it is desirable to have the images entirely sharp. But when considering classical holiday pictures or portraits or other any kinds of images as illustrated in figure 1, blur can be more or less present. Blurry regions generally represent the background or, otherwise speaking, regions of less interest for the viewer. proposed during this last decade, apparently no model incorporating blur/sharp aspect has been published yet. The goal of this paper is then to develop a computational approach broadly useful to detect sharp and blur zones in diversified images as shown in figure 2. It should be noticed that blur can have various origins and that it can be generated by optical lenses, motion or any image modification with specific software. In this study, only optical blur will be considered i.e. images captured with a limited depth-of-field [6]. Figure 1: Examples of pictures where the sharp objects are of interest. Despite its limited neural resources, the human visual system is able to rapidly analyze complex scenes. As an explanation for such a performance, it has been proposed that only some visual inputs are selected by considering salient regions [1], [2], that is to say most noticeable or most important parts of a scene. In computer vision the notion of saliency was mainly popularized by Tsotsos et al. [3] and Olshausen et al. [4] with their work on visual attention, and by Itti et al. [5] with their work on rapid scene analysis. Even if some models of visual attention have been Figure 2: Extracting salient regions from a blurry background. This paper is organized as follows: section 2 briefly presents the experimental study on the effect of blur on visual attention. Section 3 deals with the problem of saliency detection and introduces the computational model developed to produce saliency maps where the visual attractors are similar to those highlighted in experimental gaze maps. Before concluding, different results are objectively compared with a ground-truth.

2 2 Assessing blur effect on visual attention In this experimental study, the hypothesis that sharp objects tend to capture attention irrespective of intensity, colour or contrast was investigated. Eye movements of 17 subjects were recorded with an eyetracker in free viewing conditions. Both male and female aging from 20 to 45 years with normal or corrected to normal vision were naïve to the purpose of the experiment. 2.1 Experiment The experiment was performed using a video based eye-tracker Eyelink II system from SR Research, Canada. The system consists of three miniature infrared cameras with one mounted on a comfortable leather padded, light-weight headband for head motion compensation and the other two mounted on arms attached to headband for tracking both eyes. Each camera has a built-in infrared illuminator. The Eyelink II system allows determination and tracking of subject's dominant eye without any mechanical configuration. Eye position was tracked at 500 Hz with an average noise less than Fixations were estimated from a comparison between the centre of the pupil and the reflection of the IR illuminator on the cornea. Before starting an experimental session, the eye-tracker was calibrated with a set of random points on a 3 3 matrix. For the observer, the calibration stage simply consisted of fixating the gaze at the nine points displayed sequentially and randomly at different locations on the screen. Each image was shown for 3 seconds, preceded by a black fixation cross displayed at the centre of the screen on a uniform neutral gray background. This has a twofold impact: firstly all observers start viewing images from the same point and secondly, it allows gaze position to be realigned if headband slippage or significant pupil size change has deteriorated the accuracy of eye movement recording. Head mounted eye-tracker allows flexibility to perform experiment in free viewing conditions as the system is designed to compensate for small head movements. Then the recorded data is not affected by head motions and participants can observe stimuli with no severe restrictions. Severe restrictions in head movements have been shown to alter eye movements and can lead to noisy data acquisition and corrupted results [7]. 2.2 Stimuli Considering the final objective, that is to automatically extract sharp salient zones from any blur background, 15 entirely sharp primary colour images were selected and manually blurred to create a database of 122 stimuli (see figure 3 for some examples). Figure 3: Examples of images from the database. Left column: original primary images which are entirely sharp. Middle and right columns: two secondary images derived from corresponding primary images. At least two secondary images were derived from each primary image to have different blurred versions in such a way that in every secondary image only one region remains sharp. Different regions were selected and preserved while the rest of the original image was blurred with the same ratio. Primary images have been processed in such a way that the result should not look artificial. The binary mask corresponding to its blurred version has also been stored. Sharp parts of secondary images are never located in their centre and selected primary images are not very complex in terms of their contents in order to not bias the results with emotional circumstances for example. More generally, the stimuli were chosen in order to respect the classical constraints imposed in such an experimental study. 2.3 Results Figure 4 presents some examples of gaze maps derived from eye-movement recordings. These gaze maps were obtained for each of the 122 images by averaging the data collected from the free observations of the 17 observers. The coloured blobs superimposed on images show the areas where observers gazed. The longer the gazing time is, the warmer the colour is. It is important to notice that all gaze maps were filtered to display only the significant fixation points according to a procedure commonly used in such a context [8]. Figure 4 clearly illustrates the attractiveness of the sharp regions. While the gaze is spread across the whole image when it is entirely sharp, the visual attention is on the contrary captured by the only sharp region otherwise. Moreover, according to our assumption that taking into account only intensity, color or contrast is not sufficient for visual saliency modeling, Figure 5 shows an example where sharpness prevails over

3 color. Objectively, the two original flowers are viewed with the same attention. On the contrary, only one is clearly discriminated when one of them is sharp. The curve in figure 6 was obtained by measuring the time spent by the 17 observers to gaze the sharp region of each blurred image of the database. It shows that more than 65% of the average fixation time of the 17 subjects was spent on gazing only the sharp region for 13 categories of stimuli out of 15. It suggests that blur information should be integrated in biologically inspired models of attention to efficiently improve the extraction of salient regions. Moreover, the same images were presented to observers in their greyscale version. Recorded data are reported in figure 7. Figure 4: Examples of gaze maps. Left column: results obtained for entirely sharp images. Other columns: results obtained for images having only one sharp region. Figure 5: An image with opponent colours and high chromatic contrasts illustrating the importance of sharpness in visual attention mechanisms. Figure 7: Average percentage of trial time spent on gazing sharp regions in greyscale images. The numbers ranging from 1 to 15 refer to the primary images chosen during the experiment. Measures of all secondary images derived from a same primary image were grouped together. It shows that the gaze trends for colour and greyscale stimuli are very similar. This suggests a minimal influence of colour on visual attention at any rates when the observed scenes have a sharp area on a blur background. As a conclusion, even if there is variability in the results, a generic behaviour can be noticeably extracted from the curves. 3 Salient region detection model From now on, after being convinced that blur/sharp aspect plays an important role in our visual attention, the aim of the work is to develop a computational model that integrates the blur/sharp aspect and that provides saliency maps correlated to human visual attention. After reviewing existing techniques, a combination of different models will be proposed. Figure 6: Average percentage of trial time spent on gazing sharp regions in colour images. The numbers ranging from 1 to 15 refer to the primary images chosen during the experiment. Measures of all secondary images derived from a same primary image were grouped together. 3.1 Sate of the art Models of visual attention can be classified as biologically, purely computational or combined approaches. None of the existing models integrates the sharpness information but the following methods will provide references and basis:

4 i. Biologically-inspired model developed by Itti et al. [5], referred as IT in the following. Figure 10: Achanta s model, referred as FT in the text. iv. Spectral residual model developed by Hou et al. [11], referred as SR in the following. ii. Figure 8: General architecture of Itti s model, referred as IT in the text. This model uses bottom-up saliency technique to compute saliency maps. It takes into account only the three low level parameters colour, intensity and orientation. Graph-based model developed by Harel et al. [9], referred as GBVS in the following. Figure 11: Hou s model, referred as SR in the text. This model is purely computational and it is not based on any biological principle. The power of log spectrum is exploited in this approach to explore the properties of the background of any given stimulus. The main idea is to remove the redundant part of any image s spectrum. 3.2 Results Figure 12 presents saliency maps obtained with the four previous models when using in input the database of stimuli constructed for the visual experiment. iii. Figure 9: Harel s model, referred as GBVS in the text. This method creates feature maps using Itti s model but performs their normalization using a graph-based approach. This leads to more robust results and to a predicting human performance. Actually, it acts in a centre-bias manner which corresponds to a natural human tendency. Frequency-based model developed by Achanta et al. [10], referred as FT in the following. This model extracts low-level, pre-attentive and bottom-up saliency. Frequency-based approach is used to estimate centre-surround contrast using colour and luminance features. It offers some advantages over other existing methods: well-defined regions, full resolution and computational efficiency. These results confirm that none of the above methods is able to extract sharp regions that have attracted the visual attention of observers. Nevertheless, GBVS and SR techniques seem better in predicting human fixations in comparison to other state-of-the-art saliency detection methods. In order to not only obtain a subjective evaluation of these models, the ground-truth built during the creation of the database was used. With this groundtruth, the sharp or blur property is accessible for each pixel of any image. In fact, let remember that from original entirely sharp images a database of 122 stimuli has been constructed, as explained in paragraph 2.2. Each sharp zone preserved during the blurring step was stored in a binary mask for each stimulus, called the ground-truth (see column b of figure 15 for some examples).

5 transition for contours in sharp regions is like a step function when a ramp characterizes them in blur zones (see figure 13). Figure 12: Comparison of saliency maps obtained with the selected models of visual attention. Table 1 presents the percentages of precision and recall, respectively for SR and GBVS models. Table 1: Precision and recall obtained with SR and GBVS models. SR model GBVS model Precision Recall Precision Recall These values confirm the previous subjective evaluation. Firstly, SR model fails to uniformly map the entire salient area as 13 times recall percentages are under 50%. Secondly, GBVS model detects always sharp regions as salient but at the same time the precision is very low, ranging from 6% to 34%. 3.3 SGHF method We developed an approach based on high-frequencies using the classical properties of blur regions. We propose to use the Laplacian of Gaussian operator to detect edges. As this operator detects edges in both blur and sharp regions, we exploited their respective characteristics to differentiate them. The intensity Figure 13: Edge aspects in sharp and blur regions. Then the resulting image derived from the Laplacian of Gaussian filtering is processed again after dividing it in non-overlapping blocks of 2 2 pixels. Within each block the standard deviation is computed giving an image that we will call map_ls. The output of this high frequencies detection module corresponds to edge information in sharp regions. As mentioned above, GBVS model detects sharp regions as salient areas but also extracts other regions that have no visual interest. SR model fails for uniformly mapping the entire salient region. In fact, SR model has high precision values but small recall scores while GBVS produces high recall values but bad precision scores (see Table 1). Then it seems interesting to combine map_ls with the output of both GBVS and SR models to achieve better results in terms of accurately predicting human fixations i.e. with higher precision and recall values. To deal with this problem a classifier able to automatically discriminate the different labels for each pixel can be selected. Classification by examples is one of the main machine learning problems [12]. It is well known that there is no general rule to guide how to choose learner for a specific task. For the purpose of the work, the decision tree C4.5 was retained [13]. This classical classifier is easily converted to a set of production rules and does not have a priori assumptions about the nature of data. This choice has been guided by the effective compromise between efficient results, rule size and learning time cost. The obtained rules for combining the different information are given in figure 14. This corresponds to the method referred as SGHF. Figure 15 shows very encouraging visual results since SGHF method not only successfully maps only sharp regions as salient but also completely maps them. Table 2 presents precision and recall values for the fifteen categories of images of the database. Not only higher precision values are obtained, but also higher recall values. Figure 16 compares the results obtained by SGHF with previous better models. It is clear that SGHF method gives a better trade-off between precision and recall values.

6 Figure 14: Decision used to combine map_ls (feature 1), SR output (feature 2) and GBVS output (feature 3). Figure 15: Comparison of the results given by SGHF method with the ground-truth built during the visual experiment. Table 2: Precision and recall scores obtained with SGHF method. SGHF method Precision Recall Figure 16: Precision-Recall obtained with SR, GBVS and SGHF methods for the database used in the visual experiment. 3.4 Results on a random database To confirm the validity of our approach, one hundred images with blur/sharp areas were gathered from Internet (examples are shown in figure 17). Corresponding ground-truth results were manually created. More precisely, a paintbrush was used by each observer to paint the different areas of each image. As it is sometimes difficult to answer on the sharp/blur aspect of a precise zone, each observer has the possibility to paint in gray the uncertain areas. All the results were averaged giving the ground-truth used for the evaluation. The white parts are definitively blur, while the black ones are on the contrary sharp. The grey ones are uncertain. Figure 17 shows some examples, while figure 18 illustrates precision and recall values obtained with the different models. Again SGHF method leads to better results, giving a good compromise between precision and recall values. For all images, the SGHF leads to good results where only the attractive part of the image is retained, whatever the color attractiveness. Let note the first one, where the classical approaches extract the orange flower while our method extracts really the grey and flat bird. 4 Conclusion An experimental study was firstly designed to investigate the influence of blur on human visual attention. In free viewing conditions, with no prior knowledge on stimuli, eye movement recordings from a collection of 122 colour and greyscale images confirmed a general visual behaviour of observers. This suggested that blur information has to be integrated in visual attention models, in the same way as colour for example.

7 Figure 17: Results with SGHF method on random selected images from Internet. Figure 18: Precision-Recall for random selected images having sharp/blur characteristics. A first robust learning method was secondly introduced to construct an efficient blur identification tool for real images. Detection was successfully realized by the proposed method, resulting in improved subjective and objective qualities according to a ground-truth dataset, giving promising results. Future works will now be based on this efficient detector allowing to envisage its use in algorithms of higher level of image processing also known as content aware processing. Without considering the ability to reach semantic level in images, such punctual tools must be considered only as links of a more global chain in charge to propose new services and usages around new multimedia technologies (auto-cropping, content-based image retrieval ). compression and data selection, Network: computation in neural systems, vol. 17, pp , [3] J.K. Tsotsos, M.C. Scan, W.Y.K. Wai, Y. Lai, N. Davis, and F. Nu_o, Modeling visual attention via selective tuning, Artificial Intelligence, vol. 78, pp , [4] B. Olshausen, C. Anderson, and D. Van Essen, A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information, Neuroscience, vol. 13, pp , [5] L. Itti, C. Koch, and E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, pp , [6] J. Da Rugna, and H. Konik, Blur identification for content aware processing in images, in Theory and Novel Applications of Machine Learning, M.J. Er and Y. Zhou, eds, In-Tech, 2009, pp [7] H. Collewijin, M.R. Steinman, J.C. Erkelens, Z. Pizlo, and J. Steen, The Head-Neck Sensory Motor System, Oxford University Press, [8] A.T. Duchowski, Eye tracking methodology: Theory and practice, Springer, [9] J. Harel, C. Koch, and P. Perona, Graph- Based Visual Saliency, Advances in Neural Information Processing Systems, vol. 19, pp , [10] R. Achanta, S. Hemami, F. Estrada, and S. Süsstrunk, Frequency-tuned Salient Region Detection, Proceedings IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), Miami Beach, Florida, [11] X. Hou, and L. Zhang, Saliency detection: A spectral residual approach, Proceedings IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Minneapolis, Minnesota, June 2007, pp [12] T. Mitchell, Machine learning, McGraw- Hill, New York, [13] J.R. Quinlan, Improved use of continuous attributes in C4.5, Journal of Artificial Intelligence Research, vol. 4, pp , References [1] W. James, The Principles of Psychology. London: Macmillan, [2] L. Zhaoping, Theoretical understanding of the early visual processes by data