Travel Photo Album Summarization based on Aesthetic quality, Interestingness, and Memorableness

Transcription

1 Travel Photo Album Summarization based on Aesthetic quality, Interestingness, and Memorableness Jun-Hyuk Kim and Jong-Seok Lee School of Integrated Technology and Yonsei Institute of Convergence Technology Yonsei University, Korea {junhyuk.kim, Abstract Photo album summarization refers to the process of choosing a representative subset of photos in a photo album. In this paper, we propose a novel system capable of automatic photo album summarization based on three fundamental criteria, namely, aesthetic quality, interestingness, and memorableness. Based on these criteria, steps for filtering and scoring photos are designed. Through an experiment with photo albums of different sizes, it is demonstrated that the proposed system works well consistently. I. INTRODUCTION The proliferation of mobile devices allows many people to take photos easily in their daily lives. Accordingly, the volume of personal photo collections has increased much more than before. The collected photos can be used for a variety of purposes including sharing with others and recalling past memories. Specifically, activities of choosing representative photos from a photo collection, called photo album summarization, for sharing them online and printing them in the form of book occur frequently [1]. However, it is very difficult and time-consuming to manually search through an unorganized photo album. In order to resolve such difficulty, we present a novel system that automatically summarizes photo albums based on three fundamental criteria, namely, aesthetic quality, interestingness, and memorableness. First, aesthetic quality of the photos in the summary must be high, as many photos taken by amateur users are of low aesthetic quality. Second, each photo in the summary must be interesting. Third, the photos in the summary must be memorable, i.e., contain valuable memories. The three criteria are implemented as filtering steps excluding unsuitable photos and scoring steps assigning importance of the photos. We aim at maximizing synergy of the criteria to optimize the system for travel photo collections. The studies [], [3] share some similar aspects with our work. In [], key photos in temporally grouped partitions are selected based on two rules. The first one considers the content of the photos by selecting photos that have large and centerpositioned frontal faces, and the second considers temporal importance of photos by selecting photos that are taken within small time intervals. However, aesthetic aspects of photos were not considered. Some notions used in [3] (e.g., diversity) share similarity with those in our work, but the method in [3] relies on social and geographical data, which are not available in many cases. II. PROPOSED METHOD The proposed photo album summarization method is based on the following three criteria: Aesthetic quality: When people summarize a photo album, they naturally select photos that are aesthetically pleasing. Photos taken by amateur users often contain aesthetic defects, which would not be candidates for summary. Aesthetic image quality assessment have been researched quite extensively [4]. Interestingness: In the album summary, it is not desirable to have multiple photos of similar content. Thus, the interestingness criterion enforces that the summary contains unique photos, and moreover, among multiple photos of similar content, the one having the best aesthetic quality is selected. It should be noted that this criterion is different from interestingness introduced in [5] that measures how much interesting content is included in an image. In our case, interestingness is not about the content of a particular photo but determined by similarity of temporally neighboring photos. In addition, we do not attempt to measure interestingness scores as in [5] but filter out duplicate photos to make each of the photos in the summary interesting. Memorableness: There are memorable photos that have special meaning for someone. It should be noted that this criterion is different from the memorablility noted in [6]. Whereas memorablility measures how easy a photo is to remember, which is not relevant in the context of photo album summarization, memorableness in our work is the criterion for measuring how much the user wants to remember. A. Overall procedure The overall procedure of the proposed system is summarized in Fig. 1. First, the system reads the photo album and deletes

2 Fig. 1: Flow chart of the proposed system. The criterion considered in each step is indicated by the box color. photos that are judged as blurred. Then, aesthetic quality scores of the remaining photos are calculated. Among similar photos neighboring temporally, the one having the highest aesthetic quality is selected and the rest is filtered out, which is based on the interestingness criterion. The memorableness scores are calculated for the remaining photos and used together with the aesthetic quality scores to obtain the final scores. Finally, the photos ranked top compose the final summary. B. Aesthetic quality Previous studies on aesthetic image quality evaluation often use a supervised learning approach to learn the mapping between visual features and aesthetic quality scores. Such an approach requires a large set of training images, which is often burdensome. Thus, we instead design an algorithm that does not require learning. 1) Aesthetic Saliency: Aesthetic saliency means how salient the main part of a photo is and how much the main part satisfies the compositional attribute, rule of thirds. The aesthetic saliency score is obtained by the weighted sum of the saliency score and the rule of thirds (ROT) score, which are explained below. The main part of a photo can be people or salient object(s). Thus, we consider the following three cases (Fig. ): Case 1: The photo contains one or more people. Case : The photo contains salient object(s). Case 3: There is no main person/object in the photo. (a) Case 1 (b) Case (c) Case 3 Fig. : Three cases considered for calculating aesthetic saliency scores. We use the Viola-Jones algorithm to detect faces in a photo. The number of faces, their locations, and sizes are obtained. The color, size, and location of each detected face are examined to remove detection errors. If no face is detected, we use the conditional random field (CRF)-based method [7] to detect salient objects in the photo. The outputs of the method Fig. 3: Scoring according to the rule of thirds. In the right panel, the red line shows our scoring scheme; linear scoring is also shown as the blue line for comparison. are the location of the salient part and the score representing how salient the part is. The saliency score is calculated differently for each case. In Case 1, if there are a certain number of persons, the maximum saliency score is given. If the photo contains salient objects and the output score of the salient object detection algorithm is higher than a threshold, we regard the photo as Case and use the output score as the saliency score. Finally, if the output score is less than the threshold, we regard the photo as Case 3 and the saliency score becomes zero. The rule of thirds is a compositional guideline proposing that an image is imagined as divided into nine equal parts by two equally spaced horizontal lines and two equally spaced vertical lines, and important compositional elements should be placed at their intersections. Thus, for Case 1 and Case, the ROT score ranging between 0 and 100 for an image having a size of M by N pixels is defined as follows (Fig. 3): x (1) ROT score = q 1 (M/3) + (N/3) where x is the distance between the center of the main object and the nearest intersection. In comparison to linear scoring (shown as the blue line in Fig. 3), it allows modest score decrease around the intersection but decreases the score rapidly as the location of the main object becomes more distant. For Case 3, the ROT score is set to zero. In Case 1, if there are more than a certain number of persons (four in our experiment) or they are not close to each other, the ROT score is also set to zero. ) Brightness: The brightness distribution of a photo plays an important role in determining the aesthetic quality. If it is too dark or too bright, it does not look aesthetically attractive.

3 more than 50% of the pixels in the upper one-third region of a photo have hue values within this range, we regard the photo as containing blue sky. If a photo has blue sky, we increase its aesthetic quality score in a way that the amount of increase is larger for a photo having a lower quality score. C. Interestingness (a) (b) Fig. 4: (a) Area used for calculating brightness scores and sharpness scores. (b) Two examples of the brightness histogram. We postulate that it is better if the brightness histogram of a photo is more similar to a uniform distribution function. Thus, we measure the distance between the brightness histogram and a uniform histogram by using the Kullback-Leibler (KL) divergence and use it as the brightness score after linear scaling to make the score ranging 0 to 100. Here, only the central part of a photo is used as shown in Fig. 4(a) because the boundary area is less important than the central part in judging aesthetic quality in the viewpoint of brightness. Two examples are shown in Fig. 4(b). The upper image has a well-distributed brightness histogram that is relatively similar to the uniform function, so the KL divergence is relatively low. On the other hand, the lower image has a highly skewed histogram, so the KL divergence is relatively high. 3) Sharpness: Blurred images are frequently found in a photo album created by an amatuer with a light handheld devices such as smartphones, which are not aesthetically preferable. We use the algorithm presented in [8], which compares the variances of the pixel values of the original image and those of a blurred version of the image. 4) Blue Sky: In the case of outdoor photos, presence of blue sky tends to enhance their aesthetic quality. We detect blue sky in a photo in the HSV color space by considering that hue values from 19 to 64 correspond to blue sky. If The interestingness criterion prevents containing similar images in the summary result in order to keep every image in the summary interesting. Two types of similarity are employed, i.e., temporal similarity and content similarity. The photos are listed in the chronological order, and two neighboring photos are compared. If the time difference between them is less than five minutes, their content similarity is further examined. We use the speeded up robust features (SURF) algorithm to measure the content similarity between two images. Interest points are extracted using SURF in each image and similar points across the images are matched. Then, the distance between each matched pair of features is calculated. If the sum of the distances is smaller than a threshold, the two images are considered as similar in content. Once it is determined that particular two images are similar, the one having a lower aesthetic quality score is removed. Then, the remaining image is compared with the next image, which is repeated for the entire photo album. D. Memorableness The degree of memorableness of a photo is determined in two ways. First, a photo taken with other people like friends, family, or people met during the travel is often memorable, as it allows one to recall the time spent with them. First, a photo taken with other people like friends, family, or people met during the travel is often memorable. Therefore, the memorableness score is set to be proportional to the number of people in the photo if more than a certain number of people are contained. Second, when a large number of photos have been taken within a short time period, it means that the particular moment or location is memorable. Thus, the memorableness score of a photo is set to be proportional to the number of photos lost against the photo during interestingness-based filtering. Other factors influencing memorableness can be also incorporated. For instance, the presence of a famous landmark in a photo can be detected, or the GPS coordinates can be used to check whether the photo was taken at a famous location. We do not take these into consideration because the photos in our database do not contain such landmarks and GPS information. III. EXPERIMENT A. Dataset Photos taken during two different travels were considered: Travel1: 1 university students visited Switzerland for a week. Travel: 19 university students visited Finland for 3 weeks.

4 Fig. 5: Comparison of the performance of different summarization methods. We created Album1: Album: Album3: three different non-overlapping photo 100 temporally consecutive photos in 00 temporally consecutive photos in 300 temporally consecutive photos in albums: Travel1. Travel. Travel1. (a) Summary results using the Uniform 1 system B. Ground truth In order to evaluate the performance of the proposed system, manual album summarization was performed by employing 0 participants (16 males and 4 females) aging between 19 and 5. They were instructed to manually select 5% or 10% of the photos in each album. The instruction given to the participants is: There are three photo albums obtained from the travels of about 0 college students. The photos in the albums are arranged in the chronological order. Suppose that you work at a company providing a service to summarize photo albums and create printed books with the selected photos. Select a subset of photos that are the most representative in order to satisfy the customer. (b) Summary results using the Uniform system (c) Summary results using the Uniform 3 system (d) Summary results using the proposed system Fig. 6: Selected photos in the case of Album1 (5%). C. Evaluation We created three baseline systems (Uniform 1, Uniform, and Uniform 3) that select photos uniformly with different starting points. The task of the systems was to summarize each album with 5% or 10% of the photos. The performance of a summary was measured as the average number of votes that a photo chosen in the summary received by human. In our system, we manually adjusted the contributions of different scores for combining them. The weights for the saliency score and the ROT score were set to 0.8 and 0., respectively, to obtain the aesthetic saliency score. The aesthetic saliency score and the brightness scores were equally weighted to obtain the aesthetic quality score. The final score was obtained by combining the aesthetic quality score and the memorableness score with weights of 0.8 and 0., respectively. Fig. 5 summarizes the evaluation results for the three albums (Album1, Album, and Album3) and two summary sizes (5% and 10%). It can be seen that the proposed system shows the best performance for Album1 (5%), Album (5%), Album (10%), Album3 (5%) and Album3 (10%), and similar performance with the baseline systems for Album1 (10%). It is also important to note that the proposed system performs well consistently across different albums and different target summary sizes, while the best baseline system (i.e., Uniform ) exhibits severe fluctuation in performance. Fig. 6 compares the results of the Uniform 1, Uniform, Uniform 3, and the proposed system for Album1 (5%). The proposed system tends to select photos that have prominent main objects or people, and thus the summary is more meaningful. Specifically, the second photos in Fig. 6(b) and Fig. 6(c) seem relatively less important and the fourth photo in Fig. 6(b) has poor composition. Compared to the others, the photos in Fig. 6(d) have relatively better composition. IV. C ONCLUSIONS In this paper, we presented a photo album summarization system based on the three criteria: aesthetic quality, interestingness, and memorableness. The experimental results showed that the proposed system yields consistently good summarization performance across different albums and different target summary sizes when compared with manual summarization. ACKNOWLEDGMENT This research was supported by the MSIP(Ministry of Science, ICT and Future Planning), Korea, under the ICT Consilience Creative Program (IITP-R ) supervised by the IITP(Institute for Information & communications Technology Promotion)

5 REFERENCES [1] S. Boll, P. Sandhaus, A. Scherp, and S. Thieme, Metaxa- contextand content-driven metadata enhancement for personal photo books, in Advances in Multimedia Modeling. Springer, 006, pp [] J. Li, J. H. Lim, and Q. Tian, Automatic summarization for personal digital photos, in Proc. PCM, vol. 3, 003, pp [3] P. Sinha, Summarization of archived and shared personal photo collections, in Proc. WWW, 011, pp [4] D. Joshi, R. Datta, E. Fedorovskaya, Q.-T. Luong, J. Z. Wang, J. Li, and J. Luo, Aesthetics and emotions in images, IEEE Signal Processing Magazine, vol. 8, no. 5, pp , 011. [5] H. Grabner, F. Nater, M. Druey, and L. Van Gool, Visual interestingness in image sequences, in Proc. ACM MM, 013, pp [6] P. Isola, J. Xiao, A. Torralba, and A. Oliva, What makes an image memorable? in Proc. CVPR, 011, pp [7] S. Dhar, V. Ordonez, and T. L. Berg, High level describable attributes for predicting aesthetics and interestingness, in Proc. CVPR, 011, pp [8] F. Crete, T. Dolmiere, P. Ladret, and M. Nicolas, The blur effect: perception and estimation with a new no-reference perceptual blur metric, in Proc. of SPIE Electronic Imaging, 007, pp