Predicting Eye Fixations using Convolutional Neural Networks

Transcription

1 Predicting Eye Fixations using Convoutiona Neura Networks Nian Liu 1, Junwei Han 1*, Dingwen Zhang 1, Shifeng Wen 1 and Tianming Liu 2 1 Northwestern Poytechnica University, P.R. China 2 University of Georgia, USA {iunian228,junweihan2010,zhangdingwen2006yyy, wenshifeng90}@gmai.com tiu@cs.uga.edu Abstract It is beieved that eye movements in free-viewing of natura scenes are directed by both bottom-up visua saiency and top-down visua factors. In this paper, we propose a nove computationa framework to simutaneousy earn these two types of visua features from raw image data using a mutiresoution convoutiona neura network (Mr-CNN) for predicting eye fixations. The Mr-CNN is directy trained from image regions centered on fixation and non-fixation ocations over mutipe resoutions, using raw image pixes as inputs and eye fixation attributes as abes. Diverse top-down visua features can be earned in higher ayers. Meanwhie bottom-up visua saiency can aso be inferred via combining information over mutipe resoutions. Finay, optima integration of bottom-up and top-down cues can be earned in the ast ogistic regression ayer to predict eye fixations. The proposed approach achieves state-of-the-art resuts over four pubicay avaiabe benchmark datasets, demonstrating the superiority of our work. 1. Introduction When viewing visua scenes, human visua system has the abiity to seectivey ocate eye fixations on some informative contents. In computer science fied, researchers normay deveop computationa visua saiency modes to quantitativey predict human eye attended ocations using computer vision techniques. In recent years, a arge number of computationa modes [1-7] and appications [8-10] have been proposed. Inspired by the bioogica evidence that ocations distinctive from their surroundings are more ikey to attract our attention, most traditiona approaches typicay cope with the probem of saiency modeing by three steps in sequence: eary feature extraction, feature contrast inference, and contrast integration. For eary feature extraction, Itti et a. [1] proposed three ow-eve features Corresponding author. incuding intensity, coor, and orientation. Judd et a. [2] considered more eary features, for exampe, subbands of the steerabe pyramid, 3D coor histograms and coor probabiities based features. These works rey on hand-crafted features. To acquire powerfu hand-crafted features, sufficient and proper domain-specific knowedge is generay required. Nevertheess, a thorough understanding of human visua attention mechanisms has not been achieved yet. Meanwhie, the hand-crafted features may be not universay appropriate for different types of images. Athough some machine earning methods have been invoved in some modes, e.g. ICA [11] and sparse coding [4, 12, 13], it s sti very hard for these modes to mine high-eve information and atent patterns of compex images due to the imited representationa capabiity of their shaow architectures. In saiency modes, contrast computation over eary features is another key procedure. Itti et a. [1] designed the "center-surround difference" operator across mutipe scaes to cacuate contrast. Later on, a ot of works foowed Itti s idea to compute contrast from different views via using a variety of mathematica toos, for exampe, using information theories [12], frequency spectrum [14-17], sparse coding [13, 39] or autoencoder [40]. From these previous works, we can see that most of them resort to human-designed mechanisms to cacuate contrast, which woud be insufficient to hande arge-scae data with compex distributions. The ast step for saiency modeing is to integrate various contrast features to yied saiency maps. Itti et a. [1] ineary fused three contrast maps using fixed weights. Zhao and Koch [18] earned the optima weights associated with various contrasts using a east square technique upon a set of eye tracking data. Judd et a. [2] earned a inear SVM to fuse bottom-up features. Simiary, Borji [3] expored the inear regression mode and AdaBoost cassifier for optimized feature fusion. Athough most previous works mainy concentrate on contrast-based bottom-up saiency, it is beieved that at eary stage of free viewing, eye movements are mainy directed by bottom-up visua saiency and ater on, by high-eve factors (e.g., objects [19, 20], actions [21], and events) [22, 23]. Thus it is inevitabe to combine bottom-up saiency information and top-down factors to buid a

2 superior mode for predicting eye fixations. Some works have pursued this direction. For instance, Cerf et a. [24] combined face detection with ow-eve saiency. Judd et a. [2] and Borji [3] combined bottom-up features with more top-down factors, incuding humans, faces, cars, texts, and animas. Athough these methods achieve better performance than traditiona modes reying on visua saiency aone, there is sti much room for improvement because ony a sma number of hand-tuned factors are used in these modes. A of the issues discussed above motivate us to design a new unified earning mode to enhance the hand-crafted bottom-up saiency features and top-down factors for eye fixation prediction. To this end, this paper proposes a nove computationa mode based on a mutiresoution convoutiona neura network (Mr-CNN) which simutaneousy earns eary features, bottom-up saiency, top-down factors, and their integration from raw image data. To be specific, as shown in Figure 1 and Figure 2, we train a Mr-CNN directy from image regions centered on fixation and non-fixation ocations over mutipe resoutions, using raw image pixes as inputs and eye fixation attributes as abes. Benefitting from its hierarchica architecture and the purey supervised training manner, our mode can earn saiency-reated features with hierarchicay increasing compexity in convoutiona ayers, instead of resorting to various hand-crafted features. These features earned with hierarchica depth can represent origina image regions efficienty and discriminativey. In higher ayers, the proposed Mr-CNN can earn diverse high-eve top-down visua features due to its deep architecture. Meanwhie, it can aso earn bottom-up saiency via combining information over mutipe resoutions. Considering oca image regions with the same center ocation but with fine-to-coarse resoutions (see the three image regions of the traffic sign in Figure 1), finer image regions are actuay the centra parts of coarser ones. When the deep features of both the center (the finer image region) and the context (the coarser image region) are inputted to a neura network simutaneousy, the difference between them may be earned under the supervision of abes, which makes the proposed Mr-CNN have the capabiity to earn the bottom-up saiency, contrary to using various human-designed mechanisms in traditiona modes. Finay, the ast ogistic regression ayer earns to integrate bottom-up saiency with top-down cues to predict eye fixations. We notice that some researchers have appied deep earning agorithms to mode visua saiency atey. Shen et a. [25] used a 3-ayer convoutiona sparse coding mode to earn high-eve concepts from fixated image regions in an unsupervised way. Then, a inear SVM was utiized to detect saiency from those earned concepts. Simiary, Vig et a. [26] earned a inear SVM over hierarchica optima features which are optimized via a bio-inspired hierarchica modes (hierarchica neuromorphic networks) in their Ensembe of Deep Networks (edn) mode. These two methods mainy focus on earning deep features whie ignoring the importance of bottom-up visua saiency. Lin et a. [27] used a set of adaptive convoutiona ow-eve fiters earned by k-means agorithm to produce ow-eve and mid-eve features. Then, the center-surround difference was performed over the earned features to compute oca contrast. In this mode, top-down factors were not taken into account. In contrast to these previous works, our proposed work buids a unified framework to earn both the bottom-up saiency and top-down factors simutaneousy. The contributions of this paper can be summarized as foows. (1) By using a Mr-CNN, we impement the earning of eary features, bottom-up saiency, top-down factors, and their integration from image data itsef simutaneousy. The yieded method is automated and does not depend on hand-tuned features or cacuation mechanisms. (2) The proposed method is evauated on four widey used eye-tracking benchmark datasets and achieves better resuts compared to 11 state-of-the-art modes. (3) We visuaize the earned hierarchica features from the Mr-CNN. It demonstrates that the proposed Mr-CNN can earn both ow-eve features reated to bottom-up saiency and high-eve top-down factors to improve eye fixation prediction. Furthermore, the earned features can aso uncover nove insights for the psychophysics of fixation seection and the intrinsic bioogica mechanism, which we wish can offer nove inspiration to expore the human vision system. The rest of this paper is organized as foows. Section 2 describes the proposed mode using the Mr-CNN. Section 3 reports the quantitative and quaitative experimenta resuts on four benchmarks. Finay, we draw concusions in Section Proposed mode In this section, we eaborate the approach we propose. As iustrated in Figure 1 and Figure 2, the mode architecture is mainy based on a Mr-CNN. We first briefy review CNN, and then we depict the proposed Mr-CNN in detais and show how to use it to predict eye fixations A brief review of CNN A convoutiona neura network (CNN) [28] is usuay composed of aternate convoutiona and max-pooing ayers (denoted as C ayers and P ayers) to extract hierarchica features to represent the origina inputs, subsequenty with severa fuy connected ayers (denoted by FC ayers) foowed to do cassification.

3 Figure 1: Diagram of our Mr-CNN based mode. First, the given image is rescaed to three scaes, i.e , and , then sized image regions with the same center ocations are extracted from the rescaed image dupicates as inputs to the Mr-CNN. We extract fixation and non-fixation image regions to train the Mr-CNN. When testing, we just eveny sampe ocations per image to estimate their saiency vaues to reduce computation cost. The obtained down-samped saiency map is rescaed to the origina size to achieve the fina saiency map. Considering a CNN with L ayers, we denote the output state of the -th ayer as H, where {1,..., L}, additionay using H 0 to denote the input data. There are two parts of trainabe parameters in each ayer, i.e. the weight matrix W that connect the -th ayer and its previous ayer with state H 1, and the bias vector b. The input data is usuay connected to a C ayer. For a C ayer, a 2D convoution operation is performed first with convoutiona kernes W. Then the bias term b is added to the resutant feature maps, in which a pointwise non-inear activation operation Actv is typicay performed subsequenty. Finay a max-pooing ayer is usuay foowed to seect the dominant features over non-overapping square windows per feature map. The whoe process can be formuated as: H = poo( Actv( H 1 W + b )), (1) where denotes the convoution operation and poo denotes the max-pooing operation. Severa C ayers and P ayers can be stacked one by one to form the hierarchica feature extraction architecture. Then, the resutant features are further combined into 1D feature vectors by severa FC ayers. A FC ayer first processes its inputs with inear transformation by weight W and bias b, then the pointwise non-inear activation is foowed: H = Actv( H 1 W + b ). (2) Severa non-inear activation functions have been proposed. Here we choose the Rectified Linear Unit (ReLU) [29] in a C ayers and FC ayers for its high capabiity and efficiency: Actv( x) = max(0, x). (3) The ast cassification ayer is usuay a softmax ayer with the amount of neurons equaing the number of casses to be cassified. We use a ogistic regression ayer with one neuron to do binary cassification, which is simiar to a FC ayer except that the sigmoid activation function shoud be used: 1 Actv( x) = x 1 e (4) + The activation vaue represents the probabiity of the input beonging to the positive cass. The weights { W1,..., W L } and the biases { b 1,..., b L } compose the mode parameters, which are iterativey and jointy optimized through maximization of the cassification accuracy over the training set Saiency detection using Mr-CNN Inspired by [30-32], we deveop a CNN architecture with mutipe resoutions (or scaes) to simutaneousy earn eary features, bottom-up saiency, top-down factors and their integration from image data for predicting eye fixations. Speciay, we consider three propery designed resoutions. For the input ayer, we extract image regions of fixed size centered on the same ocations from images with different scaes to form mutiresoution inputs. We first rescae the input image to three scaes by simpy warping it directy and ignoring its origina size and aspect ratio. Then

4 Figure 2: Network architecture of our Mr-CNN. Convoutiona ayer, max-pooing ayer and fuy connected ayer are denoted as C, P and FC respectivey. The sizes of the input image, feature maps, FC ayers, convoution kernes and pooing windows are marked in the ast stream of the Mr-CNN, which shares the same network architecture and the same parameters in C ayers over 3 streams. we extract image regions of same size at the same center ocation from the three rescaed image dupicates mentioned above. Thus the three image regions constitute the mutiresoution architecture containing information fows with sma-to-arge contexts and coarse-to-fine granuarities. In this paper, the three scaes used to rescae the input image are empiricay chosen as , , and , respectivey. The size of image regions is set to 42 experimentay (see Figure 1 and Figure 2). As for the network architecture, as shown in Figure 2, our Mr-CNN starts from three streams in ower ayers. Each stream is composed of three C ayers, three P ayers, and a FC ayer. Subsequenty the three streams are fused using another FC ayer, which is foowed by one ogistic regression ayer at the end to perform cassification. The three streams are separated shouder to shouder before the second FC ayer and then are combined into one ayer for jointy inferring the bottom-up saiency among the muti-resoution inputs. Here we share the parameters in each C ayer across three streams to earn scae-invariant features. We use 96 fiters with size 7 7 in the first C ayer, 160 and 288 fiters with size 3 3 respectivey in the second and the third C ayer. We set the convoution stride to 1 and perform vaid convoution operations, disregarding the map borders. We aso choose to use 2 2 pooing windows in a P ayers, 512 neurons in a FC ayers and 1 neuron in the output ayer, resuting in the whoe network of size I[ ( 3)]-C[ ( 3)]-P[ ( 3)]-C[ ( 3)]-P[ ( 3)]-C[ ( 3)]-P[ ( 3)]-FC[512( 3)]-FC[512]-O[1] (see Figure 2), where we write the size and the attribute of each ayer in and out of brackets respectivey. The denotation ( 3) means the ayer has three dupicates in three streams. The input and the output ayers are abbreviated as I and O respectivey. In the training stage, we randomy sampe fixation and non-fixation ocations based on the saiency vaues in the ground truth density maps which are generated by appying Gaussian bur on the raw eye fixation point maps. Then, we extract image regions centered at the samped fixation or non-fixation ocations as the inputs of our Mr-CNN, together with their corresponding binary cassification abes. Here we consider fixation and non-fixation image regions as positive set and negative set respectivey. Afterwards, we train the Mr-CNN using back propagation agorithm [33] and gradient descent agorithm based on the minimization of the cross entropy between the predicted abes and the ground truth abes in the ast ayer. When testing, to reduce computation cost, we sampe 2500 ocations for each testing image as center ocations to extract image regions, which is impemented by eveny samping 50 ocations aong each side of the testing image. Then the activation vaue of the ast ayer in the Mr-CNN is obtained as the saiency vaue of each ocation to form the down-samped saiency map. Utimatey, the obtained down-samped saiency map is rescaed to the origina size of the testing image to achieve the fina saiency map. 3. Experiments In this section, we report experimenta resuts to evauate the proposed approach in eye fixation prediction. We first introduce the eye fixation benchmark datasets and the evauation metrics used in this paper, foowed by the impementation detais of our mode. Then the resuts of our approach and comparisons with 11 state-of-the-art saiency modes over four datasets are presented. Finay, the hierarchica features earned by the proposed Mr-CNN are visuaized and some fata parameters are anayzed Datasets We conducted evauation on four widey used eye fixation datasets with different characteristics. The first dataset, MIT [2], contains 1003 images coected from Ficker and LabeMe datasets, with resoution ranging from to pixes. It is the argest eye fixation dataset and consists of 779 andscape, 228 portrait and severa synthetic images free-viewed by 15 human subjects. The second dataset, Toronto [11], contains 120 coor images of indoor and outdoor scenes with a fixed resoution of pixes. These images are free-viewed by 20 human subjects. The third dataset, Cerf dataset [24], is made up of 181 images with resoution of pixes. The contents of interest in this dataset are usuay faces and some other sma objects ike ce phones, toys, etc. Each image in this dataset is viewed by 7 subjects.

5 Figure 3: Quaitative mode comparisons. Fixation prediction accuracy of our Mr-CNN mode compared with 11 state-of-the-art modes over 4 benchmark datasets. The resut of edn over the Cerf dataset is not shown. X-axis indicates the Gaussian bur STD σ (in image width) by which saiency maps are smoothed and Y-axis indicates the average shuffed-auc score on one dataset. Dataset AWS BMS CA edn HFT ICL IS JUDD LG QDCT SDSR Mr-CNN MIT Opt. σ Toronto Opt. σ Cerf Opt. σ NUSEF Opt. σ Average Tabe 1: Maximum performance of modes shown in Figure 3. Optima scores of each mode over different datasets and the corresponding Gaussian bur STD are reported. The highest scores over the compared 11 modes on each dataset are shown in bod face font, the highest ones over a modes are both in bod face font and underined. The resut of edn mode over the Cerf dataset is not avaiabe. The average score of edn is cacuated over three datasets The ast dataset, NUSEF [21], is a newy proposed dataset with 758 semanticay-rich images containing affective contents such as expressive faces, interesting objects, and actions. On average, each image in this database is viewed by 25 subjects. In our experiments, we use 431 images in this dataset due to the copyright issue Evauation metrics One of the most widey used metrics to evauate saiency modes is the Area Under the ROC Curve (AUC) [11]. For an image, human eye fixation points are considered as positive set and non-fixation points are regarded as negative set. Then, the computed saiency map is binariy cassified into saient region and non-saient region by a threshod. By varying the threshods, ROC curve is achieved by potting true positive rate vs. fase positive rate, with its underneath area cacuated as AUC score. However, AUC can be greaty infuenced by center-bias [34] and border cut [35]. Consequenty, it woud generate a arge vaue for a centra Gaussian bob, eading to unfair evauation. To cope with these issues, shuffed AUC is introduced by [34, 35]. Contrary to AUC, shuffed AUC adopts a fixation points (except for the positive set) over a images from the same dataset as the negative set. Using shuffed AUC, the score of a centra Gaussian bob is 0.5 whie the score of a perfect prediction is 1. Considering the sensitivity of the shuffed AUC score to different eves of burring appied on saiency maps, we foow many recent works [4, 6] to smooth the saiency maps using sma Gaussian fiters with various standard deviation (STD) σ. Then we show the curve of average shuffed AUC scores over a datasets vs. various σ and report the best score under the optima σ to evauate a mode Impementation detais Data processing. We did data augmentation by horizontay fipping each image to doube image sampes so as to enhance mode generaization. During training, we samped 10 fixation ocations and 20 non-fixation ocations per training image based on whether the corresponding saiency vaues in the eye fixation density maps are greater than 0.9 or smaer than 0.1. When testing, for an origina testing image, we averaged its saiency map and the one of its horizontay fipped version as the fina saiency map. When extracting image regions given the center ocations, if the center pixe cose to image borders, it wi resut in insufficient pixes to extract. In this situation, we copied image borders to form image regions with the same size. Before training, each dimension in the training image regions was mean-centered and normaized to unit variance over each training set, and the same normaization process was aso used in the testing stage.

6 Figure 4: Visua comparisons of different modes. We compare some saiency maps of our Mr-CNN mode with other 6 modes, i.e. AWS, BMS, IS, LG and QDCT, which perform best over 4 datasets based on the average shuffed AUC scores in Tabe 1. The first row shows the input images from the MIT (the first 4 coumns), Toronto (the 5th to 7th coumns), Cerf (the 8th coumn) and NUSEF (the ast 3 coumns) datasets. The second row shows the corresponding ground truth fixation density maps (GT) which are generated by appying Gaussian bur on the raw eye fixation point maps. CNN parameters and settings. We trained and tested our mode over each dataset using 10-fod cross-vaidation. Specificay, we averagey and randomy divided the dataset into 10 partitions. 9 partitions were used for training and the remaining 1 partition was used for testing. This was repeated such that each partition in the dataset is used once as the testing data. During the iterative process in training the Mr-CNN, we set the training step to 5,000 where one mini-batch was trained per step. Meanwhie we used 1/9 of the training set as the vaidation set to avoid overfitting. In detais, we evauated the performance of the Mr-CNN every 200 training steps, and seected the best trained network with the minima cross entropy over the vaidation set. We set the size of mini-batch to 256 and 128 respectivey for MIT dataset and other 3 datasets during training, with respect to the different image amount of these datasets. Besides, we used weight decay of and momentum ineary increased from 0.9 to 0.99 aong with the increasing training step in a networks. To aeviate overfitting, dropout [36] was used with the corruption probabiity of 0.5 in the third C ayer and the subsequent two FC ayers for a networks. We aso used a weight constraint [36] of 0.1 to the convoutiona kernes of the first C ayer so that once the 2 -norm of a kerne is arger than the constraint, it coud be renormaized by division. This aso may reieve overfitting. Transfer earning. Given that the MIT dataset contains the argest amount of images among the four datasets and consists of various saient contents, incuding both bottom-up and top-down ones, we utiized modes trained on it to transfer domain knowedge to other three datasets to overcome the probem of acking training images. We first trained the networks over MIT dataset with the earning rate initiay set to and subsequenty decay aong with the increasing training step. Then we simpy adopted one of the networks trained in the 10-fod cross-vaidation process as the pre-trained network for the other three datasets, instead of training a new mode using a MIT images. On other three datasets, the networks are fine-tuned given a reativey sma earning rate initiay set to and a smaer one fixed to respectivey for the ast 4 ayers and the first 2 C ayers, considering the ow-eve features can generaize we over natura scene images.

7 Figure 5: Feature visuaization on MIT and Toronto datasets. Best viewed in digita version. Patform and routine. The proposed mode was impemented using Matab, python and CUDA, run on a workstation with 2 2.8GHz 6-core CPUs, 32GB memory, 64-bit Windows sever 2008 OS, additionay with a GTX Titan back GPU for acceeration. The CNN routine we used is based on the deepnet 1 ibrary. The average time taken to test an image is 14s Resuts To demonstrate the effectiveness of the proposed Mr-CNN mode in predicting eye fixations, we evauated it by comparison to 11 state-of-the-art modes, incuding AWS [5], BMS [6], CA [7], edn [27], HFT [37], ICL [12], IS [16], JUDD [2], LG [4], QDCT [17], and SDSR [38]. These methods seected for comparison have been proposed in recent years and their codes or cacuated saiency maps are pubicy avaiabe 2. We first evauated the shuffed AUC scores over our mode and other 11 modes for quantitative comparison. The saiency maps were smoothed by Gaussian kernes with various bur STD σ first, then average shuffed-auc scores of each mode on different datasets over varying σ were presented in Figure 3. Optima scores of each mode over different datasets and the corresponding Gaussian bur STDs were reported in Tabe 1. As shown in Fig. 3 and Tabe 1, the proposed Mr-CNN mode achieves the best performance on a four benchmark datasets. Especiay, it is significanty better than other 11 methods on the MIT, Cerf, and NUSEF datasets. On the Toronto dataset, our mode is sighty better than other modes. We presume that this is because the Toronto dataset contains reativey ess images, which usuay hurts the performance of deep earning modes. From our comparison resuts, AWS and BMS ranked the second echeon. We aso notice that in [26], authors adopted AUC as the metric and edn method shows the best performance The authors of the edn mode ony pubished their saiency maps on three datasets, i.e. MIT, Toronto, and NUSEF. Thus we didn t evauate the edn mode on Cerf dataset. However, its performance is not good using the metric of shuffed-auc scores. It is we recognized that shuffed-auc is a better metric to fairy compare different saiency modes. We aso give the quaitative comparison of our mode with other 6 best modes in Figure 4. As we can see, our Mr-CNN mode can detect not ony bottom-up saiency patterns (e.g., Coumn 3, 5, 6, 7, 9), but aso diverse top-down factors, such as faces (e.g., Co 1, 8, 9), text (e.g., Co 4, 5), anima heads (e.g., Co 2, 11), which are difficut for traditiona methods Feature visuaization To further understand the earned Mr-CNN, we visuaized the hierarchica features of the C ayers earned on MIT dataset and the features of the third C ayer earned on Toronto dataset. Considering the ow-eve features can generaize we over different natura images, we didn t visuaize the features in ower ayers earned over Toronto dataset. As it is difficut to visuaize convoutiona kernes in higher ayers of CNNs, for each kerne we uniformy show 9 optima stimui which most strongy activate the corresponding neuron. We just show 64 kernes per ayerfor space imitation, forming an 8 8 matrix (see Figure 5). As shown in Figure 5, our Mr-CNN mainy earns various edges and coor bobs in ayer 1, diverse corners and edge/coor conjunctions in ayer 2. The features earned in ayer 3 are very informative. On MIT dataset, there contain many ow-eve patterns, for instance, compex corners ((Row 2, Co 7), and (Row 5, Co 2)), edge conjunctions ((Row 1, Co 3), (Row 6, Co 1), (Row 6, Co 2), (Row 6, Co 4) and so on), compex textures ((Row 4, Co 2), (Row 5, Co 7), (Row 7, Co 6) and so on), and other contrast-ike patterns ((Row 1, Co 7), (Row 3, Co 6), (Row 8, Co 8) and so on). These features are essentiay reated to bottom-up saiency. Meanwhie, we can see on MIT dataset, ayer 3 aso earns some high-eve semantic concepts, for instance, eyes or eye-ike patterns ((Row 4, Co 6)), faces ((Row 7, Co 1)), human heads ((Row 8, Co 1)), human body profies or simiar patterns ((Row 7, Co 8)), and text ((Row 2, Co 8), (Row 3, Co 8), (Row 4, Co

8 Figure 6: Network structure anaysis on MIT dataset. (a): Effect of number of resoutions. We rescaed images to when testing one resoution, and when testing 2 resoutions, 90 90, , and when testing 4 resoutions. (b): Effect of number of convoutiona ayers. We just abandoned the third C ayer from our mode when testing 2 C ayers. When testing 4 C ayers, we just got rid of the ast pooing ayer and added a C ayer with kernes above. The mode with 3 resoutions and 3 C ayers is the one we used in this paper. 8)). This indicates that our Mr-CNN can earn both bottom-up saiency cues and high-eve top-down factors. On Toronto dataset, ayer 3 mainy earns many ow-eve and mid-eve patterns. It seems it fais to earn much semantic concepts as this dataset mainy consists of diverse pain objects and acks obvious semantic contents Network structure anaysis Here we anayze how the network structure infuences the mode performance. We mainy tested two fata factors, namey, the number of resoutions we used and the number of convoutiona ayers, on MIT dataset. As shown in Figure 6(a), when we increase the number of resoutions, the mode performance goes up first, then reaches the peak when three resoutions are used as in our mode, subsequenty drops down. As for the effect of different numbers of C ayers, considering it s too naive to just use one C ayer in a deep convoutiona network, we just additionay test our mode with 2 and 4 C ayers. As shown in Figure 6(b), increasing the number of C ayers from 2 to 3 boosts the mode performance apparenty, then the mode performance neary saturates. Athough using 4 C ayers can sti enhance our mode performance a itte, it aso increases much more training and testing time. Thus we adopted three C ayers regarding the tradeoff between mode capabiity and computation cost. 4. Concusions and future works In this work, we have proposed a nove convoutiona neura network based eye fixation prediction mode. Our mode has achieved the best performance with significant improvement to 11 state-of-the-art saiency modes on four pubicay avaiabe benchmark datasets. The superior performance of our method indicates that the human visua system is more ikey to process ow-eve contrast and high-eve semantics jointy rather than separatey. The earned hierarchica features were visuaized to show that our Mr-CNN earns both ow-eve saiency cues and high-eve factors. The above resuts demonstrated that the proposed mode can obtain promising performance by simutaneousy earn eary features, bottom-up saiency, top-down factors, and their integration directy from image data. More importanty, the proposed mode architecture can aso hep to improve our understanding of the interna mechanism of fixation seection in the human visua system. In the future, we wi further extend the proposed work in two aspects. First, we can expore the effect of each saiency reated feature uncovered in the visuaization experiment section, this may offer nove insights for the understanding of human vision system. The second aspect is to extend our mode to predict eye fixations whie viewing video sequences. Acknowedgements: This work was partiay supported by the Nationa Science Foundation of China under Grant References [1] L. Itti, C. Koch, and E. Niebur. A mode of saiency-based visua attention for rapid scene anaysis. IEEE Trans. Pattern Ana. Mach. Inte., 20(11): , [2] T. Judd, K. Ehinger, F. Durand, and A. Torraba. Learning to predict where humans ook. In ICCV, [3] A. Borji. Boosting bottom-up and top-down visua features for saiency estimation. In CVPR, [4] A. Borji, and L. Itti. Expoiting oca and goba patch rarities for saiency detection. In CVPR, [5] A. Garcia-Diaz, X. R. Fdez-Vida, X. M. Pardo, and R. Dosi. Saiency from hierarchica adaptation through decorreation and variance normaization. Image Vision Comput., 30(1): 51-64, [6] J. Zhang, and S. Scaroff. Saiency detection: A booean map approach. In ICCV, [7] S. Goferman, L. Zenik-Manor, and A. Ta. Context-aware saiency detection. IEEE Trans. Pattern Ana. Mach. Inte., 34(10): , [8] F. Liu, and M. Geicher. Video retargeting: automating pan and scan. In ACM Mutimedia, [9] J. Sun, and H. Ling. Scae and object aware image retargeting for thumbnai browsing. In ICCV, [10] J. Han, K. Li, L. Shao, X. Hu, S. He, L. Guo, J. Han, and T. Liu. Video abstraction based on fmri-driven visua attention mode. Information Sciences, [11] N. Bruce, and J. Tsotsos. Saiency based on information maximization. In NIPS, [12] X. Hou, and L. Zhang. Dynamic visua attention: Searching for coding ength increments. In NIPS, [13] B. Han, H. Zhu, and Y. Ding. Bottom-up saiency based on weighted sparse coding residua. In ACM Mutimedia, 2011.

9 [14] X. Hou, and L. Zhang. Saiency detection: A spectra residua approach. In CVPR, [15] C. Guo, and L. Zhang. A nove mutiresoution spatiotempora saiency detection mode and its appications in image and video compression. IEEE Trans. Image Process., 19(1): , [16] X. Hou, J. Hare, and C. Koch. Image signature: Highighting sparse saient regions. IEEE Trans. Pattern Ana. Mach. Inte., 34(1): , [17] B. Schauerte, and R. Stiefehagen. Quaternion-based spectra saiency detection for eye fixation prediction. In ECCV, [18] Q. Zhao, and C. Koch. Learning a saiency map using fixated ocations in natura scenes. J. Vision, 11(3):9, [19] W. Einhäuser, M. Spain, and P. Perona. Objects predict fixations better than eary saiency. J. Vision, 8(14):18, [20] L. Eazary, and L. Itti. Interesting objects are visuay saient. J. Vision, 8(3):3, [21] S. Ramanathan, H. Katti, N. Sebe, M. Kankanhai, and T.-S. Chua. An eye fixation database for saiency detection in images. In ECCV, [22] R. Carmi, and L. Itti. Visua causes versus correates of attentiona seection in dynamic scenes. Vision Research, 46(26): , [23] B. W. Tater. The centra fixation bias in scene viewing: Seecting an optima viewing position independenty of motor biases and image feature distributions. J. Vision, 7(14):4, [24] M. Cerf, J. Hare, W. Einhäuser, and C. Koch. Predicting human gaze using ow-eve saiency combined with face detection. In NIPS, [25] C. Shen, M. Song, and Q. Zhao. Learning high-eve concepts by training a deep network on eye fixations. In NIPS Deep Learning and Unsup Feat Learn Workshop, [26] E. Vig, M. Dorr, and D. Cox. Large-scae optimization of hierarchica features for saiency prediction in natura images. In CVPR, [27] Y. Lin, S. Kong, D. Wang, and Y. Zhuang. Saiency detection within a deep convoutiona architecture. In Workshops at the Twenty-Eighth AAAI Conference on Artificia Inteigence, [28] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based earning appied to document recognition. Proceedings of the IEEE, 86(11): , [29] V. Nair, and G. E. Hinton. Rectified inear units improve restricted botzmann machines. In ICML, [30] K. He, X. Zhang, S. Ren, and J. Sun. Spatia pyramid pooing in deep convoutiona networks for visua recognition. In ECCV, [31] C. Farabet, C. Couprie, L. Najman, and Y. LeCun. Learning hierarchica features for scene abeing. IEEE Trans. Pattern Ana. Mach. Inte., 35(8): , [32] A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and F. Li. Large-scae video cassification with convoutiona neura networks. In CVPR, [33] D. E. Rumehart, G. E. Hinton, and R. J. Wiiams. Learning interna representations by error propagation. MIT Press, Cambridge, MA, USA, [34] B. W. Tater, R. J. Baddeey, and I. D. Gichrist. Visua correates of fixation seection: effects of scae and time. Vision Research, 45(5): , [35] L. Zhang, M. H. Tong, T. K. Marks, H. Shan, and G. W. Cottre. SUN: A Bayesian framework for saiency using natura statistics. J. Vision, 8(7):32, [36] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Saakhutdinov. Improving neura networks by preventing co-adaptation of feature detectors. arxiv preprint arxiv: , [37] J. Li, M. D. Levine, X. An, X. Xu, and H. He. Visua saiency based on scae-space anaysis in the frequency domain. IEEE Trans. Pattern Ana. Mach. Inte., 35(4): , [38] H. J. Seo, and P. Mianfar. Static and space-time visua saiency detection by sef-resembance. J. Vision, 9(12):15, [39] J. Han, S. He, X. Qian, D. Wang, L. Guo, and T. Liu. An object-oriented visua saiency detection framework based on sparse coding representations. IEEE Trans. Circuits Syst. Video Techno., 23(12): , [40] J. Han, D. Zhang, X. Hu, L. Guo, J. Ren, and F. Wu. Background prior based saient object detection via deep reconstruction residua. IEEE Trans. Circuits Syst. Video Techno., DOI: /TCSVT