Jigsaw Puzzle Image Retrieval via Pairwise Compatibility Measurement

Transcription

1 Jigsaw Puzzle Image Retrieval via Pairwise Compatibility Measurement Sou-Young Jin, Suwon Lee, Nur Aziza Azis and Ho-Jin Choi Dept. of Computer Science, KAIST 291 Daehak-ro, Yuseong-gu, Daejeon , Korea (South) {nikkijin, leesuwon, ziza19, Abstract To solve jigsaw puzzles, pairwise compatibility scores for all pairs should be computed first. As puzzles are reassembled according to the compatibility scores, how to measure pairwise compatibility is crucial. In this paper, we propose a novel pairwise piece compatibility scoring approach that considers not only edge similarity but also content similarity between puzzle pieces. Specifically, we propose an algorithm that computes content similarity scores between two puzzle pieces and introduce two pairwise scoring measurements that assemble the content similarity scores and edge similarity scores. We designed a jigsaw puzzle image retrieval system that identifies the best matching puzzle piece from the candidate set given a target piece to evaluate the proposed pairwise compatibility measurements. We tested our approach on the jigsaw puzzle test images, and the experimental results show our approach is comparable to the state-of-the-art and even outperforms them for some test images. Keywords image retrieval; jigsaw puzzle solver; pairwise compatibility measurement I. INTRODUCTION The recent advancement of data-mining techniques allows us to tackle problems in large-scale images. The computerized jigsaw puzzle solving problem, useful not only for assembling shredded image pieces [1] but also for repositioning objects in an image [2], is a typical example of image data-mining. Advanced techniques need to be used to find the best-matching pieces among puzzles with hundreds or thousands of pieces. Recent studies [3], [4] approach the jigsaw puzzle problem with a graph model, where a node indicates a puzzle piece and an edge represents a compatibility score between two pieces. This approach involves two steps: (i) measuring the pairwise compatibility of all possible pairs and (ii) reassembling pieces by a tree-based algorithm. In this paper, we will focus only on the first step, viz., measuring pairwise compatibility, for a better compatibility measure that will yield more appropriate edge labels and eventually help reassemble the pieces. For this purpose, we define the problem of jigsaw puzzle image retrieval to address the task of identifying the best matching puzzle piece from the candidate set, given a target piece. For compatibility measure, existing approaches [2] [4] focus on edge compatibility, i.e., how compatible the pixels in an edge of a puzzle piece are with those in an edge of another piece. These approaches compute edge compatibility This work was supported by the Korea Ministry of Sciece, ICT and Future Planning, under the title Development of Knowledge Evolutionary WiseQA Platform Technology for Human Knowledge Augmented Services (No ). not by checking all pixels but by examining only a part of the pixels. Although machine learning enables the pixel-level analysis for the edge compatibility, which is almost impossible for humans, edge compatibility is not enough. In fact, two pieces whose edges are compatible could not be a groundtruth pair. In our observation, however, humans find the right puzzle piece not just by examining the edges but also by looking at similarity in color, graphics, etc. among the candidate pieces. That is, a puzzle piece colored reddish is more likely to be adjacent to red or orange pieces than to those of black or blue. Thus, we believe that considering puzzle content similarity among pieces together with edge compatibility will improve the performance of jigsaw puzzle image retrieval. Specifically, we propose an algorithm that computes content similarity scores between two puzzle pieces and introduces two overall pairwise scoring measurements that assemble the content similarity score and edge similarity score. We test our approach on the 20 jigsaw puzzle test images from [3]. The experimental results show that our approach is comparable to the state-of-the-art approaches and even outperforms them for some test images. The rest of this paper is organized as follows. Section II reviews related work on jigsaw puzzle solvers, Section III presents the proposed approach, and Section IV demonstrates the results of experiments. Finally, Section V concludes the paper. II. RELATED WORK The problem of computer-based jigsaw puzzle solving was first introduced by Freeman and Gardner [5]. Freeman and Gardner worked with apictorial puzzles and focus only on shape matching in pieces. By focusing on puzzle piece shape only, several works [6], [7] propose more global approaches by attempting to solve puzzles with a larger number of pieces. Wolfson et al. [6] mimic the human strategy in solving puzzles, which finds the frame first, prior to the inner part, and perform the assembly with the Travelling Salesman Problem approach. Goldberg et al. [7] follow a similar approach but use more difficult puzzle pieces. On the other hand, some works [8] [11] considered both puzzle piece shapes and image contents. Kosiba et al. [8] used image color matching and color compatibility of adjacent puzzle pieces, Webster et al. [9] used global critical points called isthmus points. Yao et al. [10] applied a classified algorithm to the shape of pieces, and Nielson et al. [11] employed the /14/$ IEEE 123 BigComp 2014

2 a b Top Right Bottom Left Top Right Bottom Left,,,,, Fig. 1. The proposed content similarity measurement. Pairwise pieces, i.e. a and b, are divided into four sub-pieces and each sub-piece is described by a feature vector, e.g. f a,t. The content similarity between the right sub-piece of a and the left sub-piece of b is computed by CSS(f a,r,f b,l ). divide-and-conquer paradigm by grouping puzzle pieces that are likely to be interconnected. In contrast to the aforementioned types of problem, several recent works [3], [4], [12] [15] remove the jigsaw shape of puzzle pieces and concentrate only on image content by using puzzles that have square pieces. In [12], puzzles are treated as binary images and a genetic algorithm is utilized to construct the whole puzzle. A recent work from [15] also applied a genetic algorithm, but the authors worked with a color puzzle with a very large number of pieces. Cho et al. [3] used a graphical model and proposed a probabilistic approach. They show that dissimilarity-based compatibility gives the most discriminative measurement. On the other hand, Gallagher [4] introduced a new compatibility measurement called Mahalanobis Gradient Compatibility (MGC). Unlike the other studies that assume the orientation of puzzle pieces are known, Gallagher [4] claims work with pieces of unknown orientation. Following [4], in this paper, we assume that the orientation of puzzle pieces is unknown. When humans solve jigsaw puzzles, pieces are usually firstly selected according to the piece content similarity. Edge similarity for the selected pieces is then examined closely. In this paper, we introduce an approach that improves the performance of a jigsaw puzzle retrieval system with square pieces of unknown orientation by considering both content similarity and edge similarities. Unlike the works that consider both shapes and contents [8] [11], we worked with the four-squared puzzles, which requires more advanced image processing techniques. III. JIGSAW PUZZLE IMAGE RETRIEVAL Our proposed jigsaw puzzle retrieval considers two aspects for piece compatibility: content similarity and edge similarity. As for the edge similarity, we utilize the MGC algorithm [4]. The MGC algorithm computes the Mahalanobis distance of pairwise pieces to consider intensity gradients and the covariance between color channels. As we use square-piece puzzles, there are four queries for each jigsaw piece. In order to provide more specified content information to each query, we divide a piece into four subpieces (top, right, bottom, left) as in Figure 1. Each sub-piece is described as a feature vector, and the content similarity is computed as described in Section III-A. In Section III-B, we introduce two approaches to merge similarity scores for jigsaw puzzle retrieval. A. Puzzle Content Similarity In order to compare the similarity between pairwise subpieces, we describe each sub-piece with a histogram of pixel intensity values in the sub-piece. Note that we do not consider color channels for the histogram and each histogram is quantized into 256 ranging from 0 to 255. The feature vector is represented as follow: f P,O = {i 0,i 2,i 3,..., i 255 }, (1) where P indicates a certain piece, O stands for the orientation of the piece, i.e. an identifier for a sub-piece, and i n shows the frequency value for the bin n. We refer to T, R, B, and L as top, right, bottom, and left, respectively. For example in Figure 1, the top sub-piece of piece a is described with a feature vector, f a,t. Then, the content similarity score for pairwise pieces is then measured by computing cosine similarity between two corresponding feature vectors as follows: CSS(f A,f B )= f A f B f A f B, (2) where CSS(f A,f B ) ranges from 0 to 1. In Figure 1, the content similarity between the right side of the piece a and the left side of the piece b is computed by CSS(f a,r,f b,l ). B. Overall Scores In order to merge our proposed content similarity score and the edge similarity score borrowed from [4], we add the two scores. The overall score of two sub-pieces, P A and P B, is then defined as follows: En ADD(P A,P B ) = α{mgc(p A,P B )} (3) +β{css(f PA,f PB )}, (4) where α + β =1. We set both α and β as 0.5 by default. We expect that the content similarity in intensity plays an important role in solving puzzles, as in Figure 3(a). However, unlike our expectation, there are many pieces that have 124

3 (a) (b) Fig. 3. (a) We expect that content similarity in intensity plays a major role in jigsaw puzzle retrieval. (b) However, many adjacent pairwise pieces do not share the similar intensity. Query Best-3 retrieved pieces by edge compatibility 1st 2nd 3rd, = 0.9 = 0.95 = 0.7 Fig. 2. Example of En RANK. As the content similarity scores of the first ranked piece, a, is higher than the second ranked piece, b, the final winner piece is b. different intensity histograms with their adjacent pieces, as in Figure 3(b). With this observation, we designed another score merging approach that first considers the edge score and then the content similarity score later. This approach is as follow for a given query: 1) Retrieve top-k pieces by edge scores for a query subpiece. 2) Re-rank the retrieved pieces by the content similarity scores. We call this overall scoring measurement as En RANK. In this paper, k is set to 3. Figure 2 shows an example of how En RANK works. The three retrieved pieces, i.e. a, b, and c, are re-ranked by the proposed content similarity scores, as in Equation (2). As the CSS of the first-ranked piece, a, is higher than that of the second-ranked piece, b, the rankings are changed. Finally, the piece b is ranked as the first. IV. EXPERIMENTS We tested our approach with the 20 jigsaw puzzle test images from [3]. One of the test images is shown in Figure 3. Each test image has 432 pieces, where each piece is composed of 28 pixels. We also designed a simple jigsaw puzzle image TABLE I. AVERAGE ACCURACY OF JIGSAW PUZZLE RETRIEVAL WITH DIFFERENT PAIRWISE COMPATIBILITY MEASUREMENTS. Accuracy of jigsaw puzzle retrieval Test image SSD MGC En ADD En RANK Average retrieval system that retrieves the best matching piece for a given query piece according to a scoring algorithm. For evaluation, we calculated the average accuracy for the first-retrieved pieces for all pairs. Table I shows the performance of the jigsaw puzzle retrieval for the proposed scores, i.e. En ADD and En RAN K, sum-of-squared differences (SSD) in RGB and MGC [4]. Each row shows a test puzzle image name, the performance results from SSD, MGC, En ADD, and En RANK in order. The average performance for each score measurement is also shown in the last row. The best accuracy for each test image is highlighted in the table. Comparing our measurements with the SSD, both of our overall scoring measurements outperformed existing measurements for almost all images. However, comparing with the MGC [4], only En RANK is comparable. Although MGC is superior to En RANK for the majority of the test images, 125

4 Query Best-3 retrieved pieces by edge compatibility 1st 2nd 3rd Q1 Q2 Q3 Fig. 4. Successfully re-ranked examples of our proposed En RANK approach. Each row shows a query and its retrieved puzzles by the edge scores. The En RANK successfully retrieves the groundtruth pieces at first. Accuracy (%) β, weights for the CSS Fig. 5. Accuracy change by α and β. indicates the performance of MGC from [4] for the test image 13 while shows the performance change of our proposed En ADD with β. the performance difference is less than In addition, for several test images, En RAN K outperforms MGC. We found the performance of jigsaw puzzle retrieval with a certain pairwise compatibility measurement relies heavily on the characteristics of a test image and the divided pieces of the image. Figure 4 shows three successfully re-ranked examples in test image 13. For an additional experiment, we changed the weights for the edge similarity and the overall similarity from Equation (3), i.e. α and β, to see how these two values affect the performance of En ADD. We tested on the test image 13 whose MRF accuracy is 76.82%. As shown in Figure 5, the performance of En ADD changes according to the change of β 1. We can check the best performance of En ADD when α =0.8 and β =0.2. V. CONCLUSION In this paper, we have proposed a novel jigsaw puzzle pairwise compatibility measurement that considers content similarity between pairwise pieces as well as edge similarity. In particular, we compute the piece content similarity scores by getting cosine similarity between pairwise pieces after describing each piece with an intensity histogram. Two overall measurements to merge the content similarity score and the edge similarity score have been also proposed: En ADD and En RANK. Our experimental results show that our En RAN K measurement is comparable to the state-of-the-art approaches and even outperforms SSD and MGC for several images. We also found that the performance of jigsaw puzzle retrieval with a certain pairwise compatibility measurement relies heavily on the characteristics of a test image and the divided pieces of the image. As our En RANK works for some test images, we are convinced that there must be some kinds of puzzle test images that take advantages of applying the content similarity scores. Hence, finding a test images suitable for the use of the content similarity scores will improve the performance of the jigsaw puzzle retrieval system. 1 The α changes automatically according to the β. 126

5 ACKNOWLEDGMENT We would like to give special thanks to Prof. Sungeui Yoon for his invaluable advice. REFERENCES [1] H. Liu, S. Cao, and S. Yan, Automated assembly of shredded pieces from multiple photos, vol. 13, no. 5, 2011, pp [2] T. S. Cho, M. Butman, S. Avidan, and W. T. Freeman, The patch transform and its applications to image editing, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [3] T. S. Cho, S. Avidan, and W. T. Freeman, A probabilistic image jigsaw puzzle solver, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [4] A. C. Gallagher, Jigsaw puzzles with pieces of unknown orientation, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [5] H. Freeman and L. Garder, Apictorial jigsaw puzzles: The computer solution of a problem in pattern recognition, IEEE Transactions on Electronic Computer, vol. EC-13, no. 2, pp , [6] H. Wolfson, E. Schonberg, A. Kalvin, and Y. Lamdan, Solving jigsaw puzzles by computer, Annals of Operation Research, vol. 12, pp , [7] D. Goldberg, C. Malon, and M. Bern, A global approach to automatic solution of jigsaw puzzles, Computational Geometry, vol. 28, pp , [8] D. A. Kosiba, P. M. Devaux, and S. Balasubramanian, An automatic jigsaw puzzle solver, in Proceedings of 12th International Conference on Pattern Recognition, [9] R. W. Webster, P. S. LaFollette, and R. L. Stafford, Isthmus critical points for solving jigsaw puzzles in computer vision, IEEE Transactions On Systems, Man, And Cybernetics, vol. 21, no. 5, pp , [10] F.-H. Yao and G.-F. Shao, A shape and image merging technique to solve jigsaw puzzles, Pattern Recognition Letters, vol. 24, pp , [11] T. R. Nielsen, P. Drewsen, and K. Hansen, Solving jigsaw puzzles using image features, Pattern Recognition Letters, vol. 29, no. 14, pp , [12] F. Toyama, Y. Fujiki, K. Shoji, and J. Miyamichi, Assembly of puzzles using a genetic algorithm, in IEEE Int. Conf. on Pattern Recognition, [13] N. Alajlan, Solving square jigsaw puzzle using dynamic programming and the hungarian procedure, American Journal of Applied Sciences, vol. 6, no. 11, pp , [14] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, A fully automated greedy square jigsaw puzzle solver, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [15] D. Sholomon, O. David, and N. S. Netanyahu, A genetic algorithmbased solver for very large jigsaw puzzles, in IEEE Conference on Computer Vision and Pattern Recognition (CVPR),