Automatic Organization of Photograph Collections

Transcription

1 Automatic Organization of Photograph Collections Michael Wallick University of Wisconsin-Madison 1210 West Dayton Street Madison, WI Michael Gleicher University of Wisconsin-Madison 1210 West Dayton Street Madison WI ABSTRACT In this paper we address the problem of interacting with large streams of photographs. To achieve this, we studied patterns of photography, and found that photo streams exhibit a strong burst pattern along multiple levels of the time line. Using this fact we developed an algorithm to automatically organize the photographs hierarchically, where each node corresponds to a different event that the photographs capture. In order to represent the hierarchy, a small number of photographs are automatically selected to represent that event. This selection is modeled on the results of an ethnography we conducted to determine how human carry out this task. Using these methods, we have developed applications for viewing, tagging, and searching large streams of digital photographs. 1. INTRODUCTION Advances in digital photography offers the power to collect, store and share more photographs than ever before. The average digital camera owner may accumulate between 3000 to 6000 photographs per year [5]. Unfortunately this leads to massively sized photo streams that are difficult to organize, or even select a few good pictures. 1 In this paper we demonstrate how large photo streams can be automatically organized into a tree structure, where each node in the tree is the set of photos for an event being photographed, and the children of each node are sub-events. We also present the results of a user study that shows how to model human selection of a small number of photographs to represent each grouping, in order to do this automatically. Rather than displaying all of the photographs at any one time, each event is abstracted by showing a single photograph that acts as both a summary of the event and a gateway to the photographs it is representing. Automatic organization of photographs can be a challenging problem. When performing this task, humans rely on the contents of the photographs to determine the context of the stream in order to make organizational choices. Rather than attempting to extract this high level information, we rely on low level cues in the photograph, 1 In this context, we define a photo stream as a collection of photographs taken over some period of time by a single photographer. Each photograph in the stream has the time it was captured associated with it. such as the metadata recorded by the camera and visual content of the image. In this way, our algorithms can approximate the semantic relationships between the different photographs to complete the task of organizing the photograph collection. There are two key insights in this work. The first is that by using the metadata in each photograph, our algorithm is able to cluster photographs together approximating the boundaries in the events. Each cluster can be further divided in order to create a full hierarchy of the photo collection. The second insight is that a subset containing a sampling of photographs from the full stream provides a summary of the collection without having to interact with all of the photos at once; and this summarization can be applied to any level of the hierarchy. The remainder of this paper is organized as follows. In the next section we describe some of the work that is related to our efforts. Next, in Section 3 we introduce our burst theory of photography, and describe our clustering algorithm, which makes use of this theory. In Section 4 we describe our photo selection method. After that we outline some applications for our methods (Section 5). We conclude in Section 6 with an evaluation of our methods. 2. RELATED WORK The ideas in this paper draw on those of photo browsing, automatic photograph organization, and photo layouts. In this section, we briefly describe some efforts in these three areas. There have been several commercial applications for photo browsing, such as Picasa and ACDSee. When browsing, users are presented with every photo, usually as a thumbnail. The user can annotate and sort the photos based on various criteria such as date taken. Zoomable User Interfaces (ZUI) have also been proposed as an improvement for browsing large collections of photos. One such example is Photomesa [1]. The photographs are laid out on a canvas. The closer the mouse is to a photograph, the larger it will appear on the screen. A problem with these approaches is that the user is always presented with all of the photos in the collection at once, making it difficult for these approaches to scale as collections grow. Other efforts have looked at using the time stamp of photographs, or some other piece of metadata, to automatically organize photo collections [3, 4, 9]. In these systems, the photographs are clustered into a single level, and a photograph may be selected to represent each grouping. If any one group is very large, it could have the same problems as other photo browsing systems.

2 Recently researchers have been looking at using collages in order summarize a collection of photographs [2, 10, 11, 14]. In this way the user can get an overall impression of the photo collection by viewing a single image. Currently, [10] is the only system that abstracts away the photographs. In this system, a completely new image is rendered taking key elements from each image. The other systems simply create a collage of all input photos. The methods we present addresses all of the above issues. Unlike traditional photo browsing tools, the user is never overwhelmed by too many photographs because they are never shown the full set, rather a summary is displayed. However, because all of the photographs are stored in the tree, no photograph is ever unreachable, as they may be in the collage based systems. Finally, our methods create a tree with as many levels as is necessary. Our work also draws on inspiration from work in photo retargeting [7, 13]. In both problems semantic information about the photographs is required but missing. Instead low-level cues are employed in order to approximate semantic meanings. Figure 1: Time line at multiple levels showing the burst pattern of photographs. 3. AUTOMATIC PHOTO CLUSTERING Researchers have pointed out that photographs are often captured in a burst pattern [3, 4]. Two pictures taken by the same camera close in time are likely to be semantically related, as the pictures must be taken close in space. Further, a photographer will take multiple pictures of something of interest, either to capture the entire event as it unfolds, or to make sure to get at least one good picture of the subject. We extend this theory of the burst pattern to multiple levels. It is our hypothesis that at any level of the time line, photographs will exhibit this pattern, such as in Figure 1. To motivate this point, consider a stream which contains a trip to Paris, France (an example that we have worked with). Looking at the entire stream, there is a large burst around the time of the trip. If we were to zoom-in around that time frame, we would again see different bursts around different events during the trip, such as photographs by the Eiffel Tower and the Louvre. Again, we can zoom in further around the visit to the Louvre and see a burst of photographs corresponding to each room, or individual works of art. Each of these bursts can approximate the semantic relationship between the photographs. We have studied many photo streams from different amateur photographers. Empirically we have found that as long as camera does take pictures at regular intervals, all streams follow this burst pattern. 3.1 Automatic Clustering Algorithm We have developed a novel clustering algorithm for creating the hierarchy of photographs. For a given set of photographs we determine the average time between each photograph. If two photographs have a time stamp difference greater than some constant times the average, they are placed in two separate clusters. Our implementation looks for spaces between photographs that is three times larger than the average space. This is done recursively on each cluster group until every photograph is placed into its own unique cluster. The recursive clustering produces a tree separating the photographs into an approximation of events at different levels of granularity. With this approach the photographs are properly clustered at all levels of the hierarchy; however, there is no guaranteed bound on the number of clusters. Since photographs that are clustered together are likely to be related, we do not want to combine clusters and create artificial relationships. Figure 2 is Figure 2: Example of a collage representing approximately 400 photos. Branching size automatically set. Each photograph is representative of one event in the photo stream. an example of a collage of photographs representing the automatic clustering. As an alternative to our approach, we have tested K-Means clustering to generate the hierarchy of photo clusters with a fixed number of clusters. In this context, K would be the number of bursts. Since the number of bursts (at all levels) is generally not known a priori, it is not surprising that our algorithm produces more a natural partitioning of the photographs. The one advantage of using K-means over our algorithm is that it gives a guaranteed upper bounds on the number of clusters (K) which can be useful if the display medium is small. However, again, the clusters will not be as meaningful as with our method, since the boundaries of clusters will be set to create the correct number of clusters, rather than around event structure. Figure 3 is an example of a collage of photographs representing the K-Means clusters. The photograph clustering can be computed in realtime. We capitalize on this fact and only generate the lower levels as the user traverses to a section of the tree. This reduces memory usage, as we only have to load photographs from the part of the subtree that the user is visiting.

3 Using the most simplistic, and common, methods (random selection or position in set) without looking at the visual information in the photograph may select a nonrepresentative photograph, and should be avoided. Figure 3: Example of a collage layout representing approximately 400 photos. Branching Size set to 20 images. Each photograph is one cluster, however several events have been clustered together to ensure only 20 images will be presented. Figure 4: Example of a tree of photographs. 4. SELECTING REPRESENTATIVE PHO- TOGRAPHS After we have computed the hierarchy, we need to have a means of visually representing this structure. Since photographs taken close together are related, we should be able to choose a single photograph (or small number of photographs) to represent each event. In other words, each node in the tree is a small collection of photographs, representing the photographs underneath it. Individual photographs are treated as leaf nodes. Figure 4 shows an example of such a tree. In order to truly be able to pick a representative photograph, some semantic understanding of the set is required. However, this information is not readily available. Systems which require selecting a small set of images as representative tend to use one simple lowlevel cue. These cues are often: physical location in the set (first or middle image), histogram comparison, or random selection. Since our method will hide many of the photographs and expect that one photograph will represent those that are hidden, this is an important choice to make and must be done with care. To this end, we conducted a user study to test the methods listed above, along with: appearance of faces, internal image contrast, human selected best photograph and human selected worst photograph. The full details of this study are included in the Appendix at the back of this paper. The results of the study were: There are three categories a photograph can fit into: representative, non-representative, and neither. There are likely to be multiple representative and nonrepresentative photographs in any given set. 4.1 Representative Photo Selection Model Since it is so imperative that the our methods do not select a nonrepresentative photograph, our goal is to build a better model for selecting representative photographs. To inform our new model, we conducted a second user study in the form of a talk aloud ethnography. In such a study, a participant is given a specific task and asked to discuss what she is thinking while performing the task. In our study, five participants (three males and two females, all with experience of dealing with large photograph collections) were asked to select representative and non-representative photographs from clusters at various levels of the tree from two different photo streams. More specifically, we described representative photographs as being a label for a shoe box full of photographs of set. Each set of photographs was displayed in a thumbnail view and the participant was permitted to open and view any individual photograph at any resolution. After completing their task, the participants were asked to briefly describe their procedure for making the selections. Although there was a lot of variance in their selections, and in some instances disagreements (photographs being marked as nonrepresentative by one participant and representative by another), their methodology was strikingly similar. First, each participant would try to identify a theme or context for the images in the set. Next, she would cull (mark as non-representative) any poorly captured photographs; e.g. out-of-focus, over light, etc. Next she would look for photographs that could be considered representative. This is generally photographs that fit the context of set, contained people, and were aesthetically pleasing or interesting to look at (in that order). Several participants commented that the appearance of people, in the correct context, would serve as a good memory trigger. Some participants went so far as to look for reoccurrence of the same people as an importance measure. Finally, photographs that did not fit the context of the set were marked as non-representative. It is important to note that the designation of representative/non-representative was not a total partitioning of the sets, in fact most photographs were left unmarked. Based on the results of our ethnography, we derived a formula for scoring photographs as being representative. For any given set, the photograph with the highest score is chosen as the representative photograph. If more than one photograph is desired, then the set should be further clustered, and a photograph from each sub-cluster can be chosen. The following is the formula that we derived: P i = α C(P i,s) + β F(P i ) + γ I(P i ) (1) In the above formula, P i is the i th photograph in set S. C is a function that returns the numerical score of context of P i relative to the set. F returns a score of the people in the image. I is a function that measures the interestingness of the image. α, β, and γ are each normalization and weighting constants to adjust the relative importance of each measure. For the given formula, the representative photograph, P r in set S is simply given as: P r = max(p i S) (2)

4 4.2 Implementation of Photo Selection Context, interestingness, and to a lesser extend, people are each abstract concepts that cannot be measured or scored automatically in unconstrained photo sets. In fact, even our five participants had disagreements in this regard. Rather, we rely on easily and reliably obtainable, low-level cues from the photographs to approximate each component of Equation 1. In order to approximate the people score, we use simple face detection from the Intel Open Source Computer Vision library (OpenCV). Each detected face adds one to the face score. An alternative method would be to combine this with face recognition, so that faces that occur throughout the set can carry a higher score. The β parameter can be used to weight the contribution of faces. To approximate interestingness, we use internal image contrast, please see [8] for the implementation details. An image with a large amount of internal image contrast is likely to contain some interesting feature, and the contrast will attract the eye. The interestingness score returns a value between 0 and 1 as a percentage of the image that contains contrast. The γ parameter is used to scale up the contribution of interestingness to the overall equation. Both the face and the interestingness scores can be computed for each photograph once and independently of the rest of the set. Since the photographs in the photo tree exist at multiple levels, context is something that changes at different levels. For example, the context of the full set of photographs of a trip to large city is different than the set of photographs taken at a museum in that city, although the same photograph may exist in both sets. For this reason, the context score for a photograph must be computed at every level of the tree. Our implementation uses the image color (RGB) histogram to approximate the context score. Photographs within the same context should have a similar color distribution. The context score of a photograph is inversely proportional to the L 2 distance of the photograph s quantized histogram to the average quantized histogram of the entire set. Here, the α parameter is used to normalize the distances relative to people and interestingness and weight the contribution appropriately. If the set is large and contains many different contexts, then the average histogram will tend towards a straight line and the distance to all of the histograms in the set will be roughly the same, thus canceling the context contribution out. When the time span of a cluster is large, we can ignore context and reduce the computation time. It should be noted that we have chosen one particular combination of approximations to Equation 1. We believe that this does a good job of modeling representative photo selection, however other approximations may outperform our methods. We give a through discussion of the selection performance in Section 6. Results of our photograph selection can be seen in figures throughout this paper. Figures 5 and 6 illustrate two different paths through the same tree, showing the clustering and representative photo selection. 5. APPLICATIONS Once we have completed clustering and selected representative photographs, there are several applications that can make use of these methods. In this section we describe a few such applications that we have developed which make use of our methods. Figure 5: A path through the collage tree. 5.1 Layout Each non-leaf node in the tree represents some subset of the photo collection underneath it. The layout acts as a summary showing the photographs that are in that particular node s subset. The photograph with the highest score in each cluster is included in the layout. By using the score, the chosen photographs should be visually interesting, contain the participants of the event, and be able to represent the other photographs in the set. If there is a small number of groups, each group may contribute more photographs to further populate the layout, either by calculating the next level of clustering or selecting multiple high scoring images. Depending on how the user wants to interact with the photographs, different layouts may be employed. We have currently implemented two different layout types: grid layout and collage layout. Future directions for this work include investigating alternate layouts that may be useful for different photo related tasks Grid Layout In the grid layout, the photos are organized in a simple grid, arranged in temporal ordering. We have found that this method is useful when trying to find a specific photo (or photos) in the collection, or go through and rapidly tag the photos (see Section 5.2). Figure 7 shows an example of photos laid out in a grid Collage Layout While a grid layout is useful to quickly finding a specific photograph (or event), we have found that a collage layout is one way to create more aesthetically interesting renderings. We employ two different collage layout algorithms. The first method is a free form generation. The images are laid out in order of their score (Section 4.2) starting at the highest and working towards the lowest scored photograph. The size given to each photograph is based on

5 Figure 7: A standard grid layout. Figure 8: A template based collage layout. the score, relative to the other photographs in the set. Each photograph is placed in the canvas trying to maximize the amount of space that it borders with other (already placed) photographs and be as close to the center as possible, without overlapping other photographs. The second method uses a predefined collage template, similar in method to [2], to place the photographs on the canvas. Each entry in the template is numbered, and photographs are again placed in on the canvas ordered by the interest score. The highest scoring photograph goes into position one of the template, the second highest scoring photograph goes into position 2, etc. The template is ordered so that position 1 is in the center of the canvas, and positions 2 and 3 are on either side of it. Positions 4 through 8 is directly above, and 9 through 12 is directly below. Positions 13 to 16 and 17 to 20 are columns on the side. This pattern continues until all of the photographs are placed. 5.2 Tagging Photographs There have been many methods presented to speed the process of tagging photographs, such as using a Drag and Drop [12] method, or using some type of computer vision approach [6]. We implemented a novel approach to tagging photographs by employing our methods. Figure 6: A path through the collage tree. Since each sub-tree represents a specific event in the set, a label given to a node can be propagated down to the children of that node; rather than having to separately label each photograph in the set. We were able to label complicated streams, containing hundreds of photographs, in approximately 10 minutes. The labeling can be used as either a method creating new combinations of trees, or to correct the event clustering when temporal information is not enough. Future investigation in this area includes integrating additional la-

6 Figure 9: A collage layout from the vacation stream for photos with the label Cayman Island. This represents several groups in the original tree. beling mechanisms with our methods. For example, a drag and drop interface can easily be combined with the tagging that we describe. Further, newer cameras are beginning to come with GPS data as part of the captured meta information, the location can be translated into labels for the tree. 5.3 Photo Browsing Using our methods as the control structure, we developed a desktop photo browsing tool. Whenever displaying a non-leaf node, the user is shown a layout that summarizes the photographs that are underneath the node. A single photograph is shown whenever the user reaches a leaf of the tree. The user is given the option of using either our adaptive clustering or K-Means clustering. The photo tree and layout is dynamically generated at run-time; only part of the photograph score (Section 4.2) is computed off-line. In order to reduce computation time and memory usage, layouts are generated as requested by the user. Traversing the tree, or browsing the collection, is done using the mouse. Left-clicking on an element of a layout moves down one level, bringing up a new layout based on the group the element represents. Right-clicking anywhere on the canvas will move up one level back to parent layout; if the root layout is being displayed this will have no effect. Example of paths through a collage layout can be seen in Figures 5 and 6, the root node for each layout is Figure 2. As the user mouses over elements of the layout, the thumbnails of the photographs that are represented by the element are displayed at the bottom of the screen. The number of images, and time range of the cluster is also displayed for the user. When moving between two layouts, a transition may be displayed. The transition between the layout helps to avoid jarring the viewer and give a visual connection between the two layouts. The transition we have implemented slowly fills the canvas, starting with the photograph that was clicked and continuing in descending order of score. Finally, the user is also given the ability to set the background color to help visually separate the background from the photo elements. Figure 10 shows a screen shot of the collage system. In addition to the desktop photo browser, we have also developed a web-based photo browser, also using our methods. The web based browser was built as an AJAX script. In the photographs can be placed on a web server, and the script does not need to be adjusted Figure 10: Screen shot of the photo tree browsing system. for different sets. Again, clicking on a photograph will traverse the tree down one level. A button is displayed for moving back up the tree to a higher level. For the web-based implementation we do not include a transition, since it is too difficult to ensure that the photographs will be transferred in a timely manner and the correct order. 6. EVALUATION There are two major components to this paper that we evaluate: the photograph clustering and representative photo selection; we discuss both in this section. 6.1 Clustering Evaluation To evaluate our automatic clustering method, we have conducted two different tests. In the first test, we compared the results of the automatic clustering versus clustering the photographs by hand using human knowledge of the contents. At the highest level of the tree, we have only found one discrepancy which occurred in a stream that spanned three years. In this case, the photographer was on vacation in a different state and returned to find massive storm damage to his home. He immediately took pictures of the storm damage for insurance purposes. In this case, both the storm damage and the vacation were clustered together. At the second level, the storm damage and vacation are separated into two different sub-clusters. This example demonstrates that while time is a very strong cue for clustering, there may be some instances when it is not enough. In the future, other information, such as GPS location could be combined to further improve the robustness of the clustering algorithm. Towards the bottom of the tree, there were several instances where the automatic and human clusters diverge, however, these are mostly matters of personal taste, and both clusters make logical sense. In some instances the automatic clustering out performed the human clustering, as the human gave up at a higher level than our clustering. Two such examples were from streams spanning six months and three years. The stream spanning three years included a trip to a zoo, which was the lowest cluster created by hand. The automatic clustering went a step further, creating distinct clusters exclusively of bears, the primate house, and the African animals (lions, tigers, etc.). From the trip Paris, in the stream spanning 6 months, trips to art museums were clustered into individual rooms which tended to contain all of the pieces of art by a particular artist. For the second evaluation, we created a subset stream where several photographs were randomly deleted, to simulate less photographs

7 being taken or photographs being culled from the set. As expected, the algorithm produced roughly the same boundaries of clusters, with some minor variations (especially towards the bottom of the tree) due to different exact spans of time between photographs. 6.2 Representative Selection Evaluation In order to evaluate the representative selection, we rely on the empirical results of the user study described in Section 4.1. The participants were shown 10 different levels of two different photo streams (one of a week long family vacation and one of photos taken over three years). We consider a result to be very good if one or more participants marked that photograph as representative; the result is acceptable if it is not marked as either; the result is bad if it was only marked as non-representative. Virtually all of the photographs selected by the system could be considered representative. For the most part, our selection method performed reasonably well, usually selecting photographs that were either marked as being representative or not marked, but still qualified as capturing the context and people in the event. There were two instances, however, where the system fails by selecting a photograph that is considered to be non-representative by one or more of our participants because it contains many people but does not convey the context, because the photograph was poorly staged and the participants are blocking too much of the background. Although selecting a photograph with several people, but no context is a better choice than most of the other non-representative photographs, further investigation into approximating context is required to improve these results. 7. CONCLUSION In this paper we presented methods for automatically organizing large collections of digital photographs. The photographs are recursively clustered, based on the metadata provided by the camera, into groups of related events. A photograph from each cluster is selected to represent the cluster, based on our model of how humans perform this task. We have evaluated our methods and find that in most cases our methods performs as well as human does, and in some cases even better. Although we do not formally evaluate our browsing tool, our experiments have lead to two interesting observations. First, locating a specific photograph (or small set of related photographs) can be done significantly faster using our browsing tool in grid layout compared to viewing the entire set in the file system. Second, users have told us that they find the collage layout to be more aesthetically interesting than any grid layout, and it is more enjoyable than viewing photographs one at a time in a linear fashion. APPENDIX We rely very heavily on automatic selection of a single photograph to represent a larger set. In general, selecting a representative image requires having semantic knowledge of the photo set, and the individual photographs. Lacking this information, low-level cues are often employed. To this end, we wanted to study different, commonly accepted procedures to achieve this goal. We studied six different single-cue methods, and two human selection methods for determining a representative image. The methods are: first image in the set, middle image in the set, closest to average histogram, total number of faces, internal contrast, random selection, human selected best, human selected worst. Our expectation is that a human can reliably select a representative image as well as a non-representative image. For our experiment we used twenty-one sets of twenty images each. Six of the 21 sets were donated explicitly for use in this research project. No one person donated more than two image sets, so if a donor participated in the study, his or her familiarity with the photographs should have minimal impact on the final results. The remaining fifteen sets were albums acquired from the Flickr web site, and are under a Creative Commons license, allowing for redistribution and modification of the original images. Only the first twenty images in each selected album was used in the experiment. For each set, six of the 20 images were selected as being potentially the most representative image in the set, using the following six methods: first image in set, middle image in set, closest to the average histogram, total number of faces, internal contrast, human selected best image. If less than six unique images were selected, either because of a lack of faces, or a single image qualified under two methods, then a random and/or worst image was also selected. In all 17 of the sets had a faces image, 11 had a worst image, and 9 had a random image. Every set was represented by one image selected by each of the other methods. We describe how we implemented each method later in this section. We invited participants to take part in the study over the World Wide Web. Initial invitations were sent to mailing lists for computer science and education graduate students. The invitation encouraged participants to forward the invitation to friends and family who they thought may be interested in participating. Our human subjects approval prohibited us from collecting any demographic or geographic information about the participants. After agreeing to participate in the study, each user was shown a set of 20 images. Below that they were shown the 6 candidate images to choose from and asked to select the one image that they felt was most representative. This was repeated a total of 21 times. The order of the sets and order of the candidate images images were independently random for each volunteer. Incomplete surveys were not recorded. Volunteers were also given the opportunity to leave comments about their experience at the conclusion of the survey, however this information was separated from individual answers. In total we received 63 completed surveys. Figure 11 shows a screen shot of a single image trial from our user study. For each set, the selection method was conducted in a consistent manner. All of the photographs contained EXIF metadata and were examined at the same resolution. All of the photographs were either 4 3 or 3 4 aspect ratio. The first image in the set was determined by the time stamp recorded in the photograph EXIF data by the camera. Since each photo set contained exactly 20 images, the 10 th image in the set was used as the middle image, again ordered by the time stamp. We used the Python Image Library (PIL) to compute the RGB histogram for each image, and the average histogram. We took the image with the smallest difference between its own and the average histogram. For internal contrast we used the algorithm described by [8] and chose the image whose sum of pixel salience was the highest. To determine the photograph with the most faces, we counted them by hand. A random number generator was used whenever we needed to randomly select an image. The most and least representative images were pre-selected manually by the experimenter. Again, our hypothesis is that a human can reliably perform significantly better than the other methods at picking a representative image. To test this, we perform a χ 2 test with a null hypothesis that each method should perform with roughly the same results. Table 1 shows number of times an image of each method was selected and

8 The reliable existence of non-representative images has an important implication for implementations: bad choices exist, and should be avoided. Therefore, systems should avoid random or fixed index methods that may inadvertently select a bad choice. Acknowledgements Cody Robson helped design the freeform collage layout algorithm. This work is supported by NSF Grants IIS Michael Wallick is funded by a fellowship from Microsoft Research. We would also like to thank the numerous participants of both user studies. Figure 11: Screen shot of our user study. Selection Method Total Votes Expected Vote First Image Middle Image Average Histogram Faces Contrast Least Representative Random Most Representative Table 1: The total number of votes for each selection method and the expected number of votes. the expected selections, assuming that each method should perform equally. Faces, least representative, and random selection have a lower expectation than the other methods since they were not used in all 21 sets. For the results, χ 2 = with 7 degrees of freedom. The P value is less than With extreme confidence we may reject the null hypothesis that all methods perform equally. A human can reliably select an image that is more representative than the ones chosen by other methods. Our single selection design creates a masking effect that makes it difficult to infer either the absolute performance of the top choice, or much about the methods that were not chosen. However, the extremely large number of times the human-identified best images was chosen and the extremely low number of times the humanidentified worst image was suggests that humans can reliably choose representative and un-representative images. A common comment among participants in our study was that for some sets, they would have chosen a different image that was not one of the six choices. Thus, participants had a different opinion of what the most representative image was from the experimenter. We feel that this suggests that there are multiple good answers. The implication is that finding one of this set of sufficiently good answers is sufficient for the selection process. References [1] Benjamin B. Bederson. Photomesa: a zoomable image browser using quantum treemaps and bubblemaps. In UIST 01: Proceedings of the 14th annual ACM symposium on User interface software and technology, pages 71 80, New York, NY, USA, ACM Press. [2] Nicholas Diakopoulos and Irfan Essa. Mediating photo collage authoring. In UIST 05: Proceedings of the 18th annual ACM symposium on User interface software and technology, pages ACM Press, [3] Andreas Girgensohn, John Adcock, Matthew D. Cooper, Jonathan Foote, and Lynn Wilcox. Human-Computer Interaction INTERACT, 3: , [4] Adrian Graham, Hector Garcia-Molina, Andreas Paepcke, and Terry Winograd. Time as essence for photo browsing through personal digital libraries. In JCDL 02: Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, pages ACM Press, [5] David F. Huynh, Steven M. Drucker, Patrick Baudisch, and Curtis Wong. Time quilt: scaling up zoomable photo browsers for large, unstructured photo collections. In CHI 2005: CHI 2005 extended abstracts on Human factors in computing systems, pages ACM Press New York, NY, USA, [6] J. Jeon, V. Lavrenko, and R. Manmatha. Automatic image annotation and retrieval using cross-media relevance models. In SIGIR 03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval, pages ACM Press, [7] F. Liu and M. Gleicher. Automatic image retargeting with fisheyeview warping. Proceedings of the 18th annual ACM symposium on User interface software and technology, pages , [8] Y.F. Ma and H.J. Zhang. Contrast-based image attention analysis by using fuzzy growing. Proceedings of the eleventh ACM international conference on Multimedia, pages , [9] J. Platt. AutoAlbum: Clustering digital photographs using probabilistic model merging. Proceedings of the IEEE Workshop on Contentbased Access of Image and Video Libraries (CBAIVL 00), [10] Carsten Rother, Lucas Bordeaux, Youssef Hamadi, and Andrew Blake. Autocollage. ACM Transactions on Graphics, 25(3), July [11] Carsten Rother, Sanjiv Kumar, Vladimir Kolmogorov, and Andrew Blake. Digital tapestry [automatic image synthesis]. volume 1, [12] Ben Shneiderman and H. Kang. Direct annotation: A drag-and-drop strategy for labeling photos. In IV 00: Proceedings of the International Conference on Information Visualisation, pages IEEE Computer Society, [13] B. Suh, H. Ling, B.B. Bederson, and D.W. Jacobs. Automatic thumbnail cropping and its effectiveness. Proceedings of the 16th annual ACM symposium on User interface software and technology, pages , [14] Jingdong Wang, Jian Sun, Long Quan, Xiaoou Tang, and Heung- Yeung Shum. Picture collage. Computer Vision and Pattern Recognition, CVPR IEEE Computer Society Conference on, 2006.