An Automatic Portuguese Tile Panel Jigsaw Puzzle Solver

Transcription

1 An Automatic Portuguese Tile Panel Jigsaw Puzzle Solver João Tiago Fonseca Abstract The main goal of this work is to assemble Portuguese tile panels, from the individual tile images and no other information. Using the color of the tile images as a feature, multiple compatibility functions were tested: Euclidean distance, l-½ norm, (L p) q norm (with p and q rational), Pomeranz prediction based distance, Mahalanobis Gradient distance and a distance function learned via KLFDA. After that, the Portuguese tile was assembled in two different ways: firstly knowing the final orientation of each tile and secondly without any kind of information. For the first problem two different assembly methods were proposed: one approaching the problem via an Augmented Lagrange Method and one using a greedy approach based on a modified Nearest Neighbor rule. For the second problem, only the second approach proved to be useful. Two different image databases were used, a 20 natural images database and a 12 Portuguese tile database.. The greedy method solved both settings with performance comparable to the state-of-the-art; surpassing it in the most challenging case, where the orientations of the pieces is unknown. In particular, the proposed system accurately reconstructs 92.63% of the natural image puzzles and 35.91% of the Portuguese tiles panels. It was also developed a specific software for image capturing and assembly testing platform. Index Terms Portuguese Tile, Jigsaw Puzzles, Generalized Quadratic Assignment Problem, Nearest Neighbor I. INTRODUCTION For hundreds of years people have been challenged by the complexity of a jigsaw puzzle. In 1767, John Spilsbury created the first jigsaw puzzle using a world map as an educational tool [1]. Almost two hundred years after, Freeman and Garder assembled the first jigsaw puzzle using a computer [2]. In this paper we elevate the jigsaw puzzle assembly to a different level, proposing a method that automatically assembles a Portuguese tile panel using only the information provided by the individual pictures of each one of the tiles. Firstly, we study six Compatibility functions: Euclidean distance, l-½ norm, (L p ) q norm, Pomeranz prediction-based compatibility, Mahalanobis Gradient Compatibility and a distance function learned via KLFDA. After that, we propose two different methods to assemble a tile panel when the pieces orientation is known, one based on the BALM method [3] and one greedy algorithm. For the assembly of a tile panel when the pieces orientation is unknown, we shown that our greedy approach challenges the state of the art [4]. A. Related work Jigsaw Puzzle solving has been object of study since Freeman and Garder [2] first started studying it in Using Figure 1: Portuguese tile panel assembled with the proposed method: greedy strategy with Lowe Scores. the shape of each piece, they manage to solve a 9 pieces jigsaw puzzle. Since then many other works tried to solve this problem using only the pieces shape [5], [6], [7], [8], [9], [10]. In 1994, Kosiba et al. introduced a new feature to this kind of problem, using image information and the shape of each piece to solve jigsaw puzzles. Their work was followed by [6], [7], [9], [10] and only in 2007 Demaine and Demaine [11] manage to solve the first jigsaw puzzle using only image information, being followed by [12], [13], [14], [4]. In 2010, Cho et al. proposed a solver based on the patch transform [15]. In their work, they studied five compatibility metrics: Dissimilarity-based, Boosting-based, Set-based, Image statistics-based and Cho et al. metric. Of the five metrics, the best one was Dissimilarity-based compatibility, with 80% classification accuracy. With the help of a low resolution image or anchor patches, they manage to solve a puzzle with 432 pieces (with jigsaw pieces orientation known). In 2011, Pomeranz et al. took a greedy approach to this problem. Firstly, they studied three compatibility metrics: Dissimilarity-based, (L p ) q and Prediction-based, being all of them applied in the CIELAB color space. After proving that Prediction-based compatibility was the best metric, they proceeded with the assembly. Here they introduced a new concept - best buddies - which identified a pair of jigsaw pieces that have mutually the lowest distance between all possible connections. The puzzle solver was divided in three steps: the placement problem, the segmentation problem and the shifting 1

2 problem. In the placement problem, the puzzle is assembled using a growing block that minimizes the distance of new connections, with priority to best buddies. In the segmentation problem, the solution found is checked by a growing region algorithm that connects best buddies pieces. Finally, in the shifting problem the largest region is found and the rest of the pieces are placed using the placement problem again. This algorithm managed to solve a 3300 jigsaw puzzle with known orientation. In 2011, Andrew Gallagher was the first to solve jigsaw puzzles with unknown orientation of each jigsaw piece. For that he introduced a new metric, called Mahalanobis Gradient Compatibility, which computes the Mahalanobis distance between two pieces using the covariance between color channels. The assembly is made using a Minimum Spanning Tree (MST) found by a modified Kruskal algorithm that provides planar MST. This solver is currently the state of the art, being able to solve a single 9600 pieces jigsaw puzzle and multiple 1064 pieces jigsaw puzzle. II. PORTUGUESE TILE JIGSAW PUZZLE SOLVER The assembly of a Portuguese tile panels is two-fold, choosing the compatibility function between Portuguese tiles pieces and assembly strategy. For the first subproblem, six compatibility functions are proposed on Section II-A. The assembly strategy was divided in two different problems: firstly on Section II-B we propose two assembly methods for Portuguese tile jigsaw puzzles with known pieces orientation; and secondly on Section II-C we propose an assembly method for Portuguese tile jigsaw puzzle with unknown pieces orientation. A. Compatibility functions The compatibility function is needed to quantify how distant two Portuguese title margins are. The main difficulties on getting a compatibility function that makes a good classification for the Portuguese tiles are related with their aging. Most of the Portuguese tiles that require to be assembled are very old (some of them with more than 500 years) and hence the techniques and fabrication methods of that time were very primitive. Variations of color and cracks on the glass coating made during the baking stage, texture created by the brushing and discontinuity on the drawings occurring at the borders of the tiles are present in most of the Portuguese tiles. Adding to these problems the fact that the Portuguese tiles were originally placed on buildings, making them subject of wear, and then removed from original place, breaking some corners and edges. Because there are so many difficulties, we tested six compatibility functions computed in three different color spaces (RGB, HSV and CIELAB), trying to find one that best classifies the Portuguese tiles. 1) Euclidean distance: We compute the Euclidean distance between two margins, m i and m j, by square rooting the sum of the squared difference along the edges of two Portuguese tile margins, s(i,j) = 3 d max [m i (c,d max,d) m j (c,1,d)] 2 (1) c=1 d=1 Wherem i andm j are two matrices of dimension3 d max d max, c is the color channel and d is the pixel of the margin, with values between 1 and d max. 2) l-½ norm: s(i,j) = ( 3 d max c=1 d=1 m i (c,d max,d) m j (c,1,d) ) 2 (2) 3) (L p ) q norm: This norm was proposed by Pomeranz et al. [14] and the reason behind it, is that the Euclidean distance penalizes severely large differences even though such large differences do exist in natural images. Because of that, they studied the L p norm and found that for p 0.3 is optimal. The value of q is said to be optimal for q = 1/16. Thus, q/p = 5/24 resulting in, ( 3 d max s(i,j) = ( m i (c,d max,d) m j (c,1,d) ) 3 10 c=1 d=1 ) 5 24 (3) 4) Distance function learned via KLFDA: This compatibility function is learned using a non-linear extension of the method Local Fisher Discriminant Analysis (LFDA). The non-linearity of this extension is guarantied by a kernel trick [16]. Let a Portuguese tile be represented by four descriptors, one for each margin. Since four points make a very small training set, white Gaussian noise is added to each original point creating nine additional poits. This way, each Portuguese tile is represented by 40 points in R dmax. Hence the decision boundary of the four classes (one for each margin) may not always be defined by hyperplanes, a kernel it is used to map the points into a space where linear boundaries are more appropriate. 5) Pomeranz prediction-based Compatibility: This compatibility function was proposed by Pomeranz et al. [14] and computes the distance between two margins, m i and m j, by measuring the distance between a predicted and a candidate margin. The predicted margin is computed by summing the difference between the last two columns of the margin m i and adding it to the last column. After that, the distance is computed using the (L p ) q norm between the predicted margin and the candidate margin,m j. This method is repeated for both margins in a symmetric fashion and added together. s(i,j) = 3 d max c=1 d=1 [(2m i (c,d max,d) m i (c,d max 1,d) m j (c,1,d)) (2mj (c,1,d) m j (c,2,d) m i (c,d max,d)) 3 10 ] (4) 2

3 6) Mahalanobis Gradient Compatibility: This compatibility function was proposed by Gallagher [4] and it penalizes changes in the intensity gradient instead of changes in the intensity. Let, G il (d,c) = m i (d,d max,c) m i (d,d max 1,c) (5) and compute the Mahalanobis distance between the margins m i and m j as, D LR (m i,m j ) = 3 d max c=1 d=1 [G ilr (d,c) µ il (c)]s 1 il [G ilr(d,c) µ il (c)] T (6) where: G ilr is a d max 3 column vector of the difference between the last an first columns of m i and m j respectively; µ il is the mean of each color channel of G il ; S il is the covariance matrix of G il. This method is repeated for both margins in a symmetric fashion and added together, creating a symmetric compatibility function. Depending on the type of tile panel assembly, the Distance Matrix, S, might change. In the case of the Portuguese tile assembly with known pieces orientation, instead of the Distance Matrix, S, it is used two distance matrices: Horizontal Distance Matrix, S lr, and Vertical Distance Matrix, S tb. The Horizontal Distance Matrix, S lr, is a n n matrix that defines the distance between any tile s right side margin and another tile s left side margin. The Vertical Distance Matrix, S tb, is a n n matrix that defines the distance between any tile s bottom side margin and another tile s up side margin. B. Portuguese tile panel jigsaw puzzle solver with known pieces orientation 1) GQAP: The assembly of a Portuguese tile panel when the orientation of the pieces is known can be seen as an assignment of each tile to a specific position on the tile panel. In each position of the panel, an assembly method will have to minimize two distance components: Horizontal and Vertical, making this subproblem a Generalized Quadratic Assignment Problem (GQAP). Figure 2: Assignment Problem formulation. In the left image are the numbered positions of the tile panel, in the center image is the permutation matrix, P, which assigns a tile to a specific tile panel position and on the right image a set of unsorted Portuguese tiles Referring only to the horizontal connections on a tile panel, the cost function of the GQAP can be described as, f(p) = trace(a h PS lr P T ), (7) where P is a n n permutation matrix, applied to the rows and columns of S lr and A h is a n n binary Horizontal Rules Matrix, where A h (i,j) = 1 represents that position i is on the right side of position j. Using the same reasoning, a second term represents the vertical contribution with the respective A v. The problem is formally described by, P = argmin trace(a h PS lr P T +A v PS tb P T ) subject to P P(n n) (8) This problem is a GQAP and has a NP-hard complexity [17], some relaxations had to be made for a sub-optimal solution be found. Orthogonal Procrustes - linear Assigniment: Here one of the permutation matrix is substituted by an orthogonal matrix and after that it is applied the Augmented Lagrangian method, obtaining the following function: L σ (P,Q;R) = trace(a h PS lr Q T +A v PS tb Q T ) trace(r T (P Q))+ σ 2 P Q 2 F, (9) where Q is an orthogonal matrix, R are the Lagrange multipliers and σ > 0 is the weight of the penalty term. Problem 8 is the permutation matrix that minimizes (9) and it can be obtained using the Bilinear modeling via Augmented Lagrange Multipliers (BALM) [3], described in the algorithm 1. Algorithm 2 solves Problem (10) by alternating the minimization direction. The first problem (11) is equivalent to, Q [l+1] = argmin trace(z T Q) subject to Q O(n n) Z = A h P [l] S lr +A v P [l] S tb + R (k) σp (l) (13) Applying a Singular Value Decomposition (SVD) to Z results in, Q [l+1] = VU T, (14) where V and U respectively collects the right and left singular vectors of Z. The second problem (12) is equivalent to, P [l+1] = argmin trace(wp) subject to P P(n n) W = S lr Q (l+1)t A h + S tb Q (l+1)t A v R (k)t σq (l+1)t, (15) which can be solved using the Hungarian method [18]. 3

4 Algorithm 1 BALM 1: k = 0 e ε best = + 2: initialize R (0), σ (0), γ > 1 e 0 < η < 1 3: initialize P (0) 4: repeat 5: Solve (P (k+1),q (k+1) ) = argmin L σ (k)(p,q;r (k) ) subject to P P(n n) Q O(n n) (10) using the Algorithm 2 6: ε = P Q 2 F 7: if ε > ηε best 8: R (k+1) = R (k) σ (k) (P (k+1) Q (k+1) ) 9: σ (k+1) = σ (k) 10: ε best = ε 11: else 12: R (k+1) = R (k) 13: σ (k+1) = γσ (k) 14: end if 15: update k k +1 16: until ε < ε min ou k = k max Algorithm 2 Iterative method to solve Problem (10) 1: l = 0 e choose L max 2: P [0] = P (k) 3: repeat 4: solve 5: solve Q [l+1] = argmin L σ (k)(p [l],q;r (k) ) subject to Q O(n n) P [l+1] = argmin L σ (k)(p,q [l+1] ;R (k) ) subject to P P(n n) 6: l l+1 7: until l = L max 8: P (k+1) = P [Lmax] e Q (k+1) = Q [Lmax] (11) (12) As shown in Section III, this method does not solve the Problem 8 and hence another relaxation was proposed. Two Assignment problems: This method consists in replacing the orthogonal matrix by another permutation matrix, changing the Problem 11 by, Q [l+1] = argmin trace(z T Q) subject to Q P(n n) Z = A h P [l] S lr +A v P [l] S tb + R (k) σp (l) (16), which can also be solved using the Hungarian method. The reason is that (13) may have negative values, which is inconsistent with (8). Figure 3: Example of two possible tile panel positions with multiple margin connections. 2) Greedy: Another method proposed to assemble a Portuguese tile panel when the orientation of the pieces is known is based on greedy algorithm. Since this method is proposed to solve a problem more complex than this one, this method will be presented in Section II-C. The assembly will proceed in the same way whether the orientation of the pieces is known or unknown. The only difference lies on the distance matrix, when the orientation of the pieces is known, the Horizontal Distance Matrix, S lr, is used and the Vertical Distance Matrix, S tb. When the orientation of the pieces is unknown, the (full) Distance Matrix, S, is used giving the assembly method full flexibility in the tile placement. C. Portuguese tile panel jigsaw puzzle solver with unknown pieces orientation The assembly of a Portuguese Tile panel when the orientation of each piece is unknown is a NP-hard problem [11]. If the proportions of a tile panel with n tiles are know, there are 4 n 1 (n!) possible solutions. In our case the proportions are not known so there are even more solutions, making the search for an optimal solution impossible to preform in a exhaustive way. The method we propose to solve this problem is based on a greedy algorithm. This method starts by searching on the Distance Matrix, S, for the smallest connection between two margins. Connecting these two margins creates six possible margins where the next tile may create a new connection. To connect the following tiles a mask matrix is created, which multiplied element-wise by the Distance Matrix, S, selects only the legal connections (illegal connections: connections between two tiles that are not already placed on the tile panel, connections with margins of tiles that are already in use and connections between margins of the same tile). At every iteration a mask matrix is created, choosing the connection with smaller distance between the possible ones. This process is repeated until every tile is placed. With the aim of improving the effectiveness of this method, two modifications of the Nearest Neighbors rule are proposed: 1) Multiple Margins: During the tile panel s assembly process, many positions where a new tile makes more than one connection with already placed tiles occur. This heuristic forces the new tile to be placed in this positions whenever they 4

5 1 Table I: AUC for the best four mean ROC curves for Cho et al. database Taxa de Verdadeiros Positivos Compatibility function (color space) AUC Mahalanobis Gradient Compatibility (CIELAB) (L p) q norm (CIELAB) l-½ norm (CIELAB) Mahalanobis Gradient Compatibility (HSV) dist. grand. de Mahalanobis (CIELAB) norma (L p ) q (CIELAB) norma l 1/2 (CIELAB) dist. grand. de Mahalanobis (HSV) Taxa de Falsos Positivos Figure 4: Best four mean ROC curves for 20 images in Cho et al. database [13] appear, expecting that multiple margin compatibility improves the confidence of a new tile placement. 2) Lowe Scores: Whenever one of the margins involved in a new connection has a second best candidate with a distance value close enough to the first candidate, the connection is rejected. A threshold is defined and whenever the relation between the first and second candidates is higher than this threshold, the connection is rejected. This heuristic was proposed by Lowe [19] and what it intuitively states is that connections with ambiguous compatibilities should not be preformed since there is no confidence. With the evolution of the assembly method and consequently less candidate margins the confidence will increase. D. Performance metric In order to evaluate the assembly of the Portuguese tile panels, we used a performance metric proposed by Cho et al. [13]. The Neighbor Comparison calculates the fraction of pairwise tiles adjacencies that are correct on a tile panel. III. RESULTS In the Section III-A, the compatibility functions are studied using a natural image database proposed by Cho et al. [13] and a Portuguese tile panel database made specifically for this work. After that, the assembly of the tile panels is divided in two sections, whether the orientation of each piece is known (Section III-B) or it is not (Section III-C). A. Compatibility functions We study the compatibility functions through ROC analysis and using the Area Under the Curve (AUC) to quantify the quality of a classification. We compute the Distance Matrices, S, for every compatibility function and every image projected in three different color spaces (RGB, HSV and CIELAB). We compute a mean Figure 5: Best four mean ROC curves for the Portuguese tile panel database ROC for every pair of compatibility function/color space by averaging over the database images. For every Distance Matrix, a ROC curve was computed and, after that, the mean ROC curve by each compatibility function/color space was calculated. In figure 4 are presented the best four mean ROC curves and on table I the respective AUCs. As it can be seen in Figure 4 and on the Table I, the Mahalanobis Gradient Compatibility computed on natural images projected on CIELAB color space, proved to be the best compatibility function/color space pair. The same evaluation was applied to the Portuguese tile panel database. Figure 5 presents the best four mean ROC and table II the respective AUCs. Analyzing Figure 5 and the respective AUCs it is clear that classifying the tile panels is a harder task than it is to classify natural images. Every difficulty described in Section II-A contributes to lower the AUCs and from that it is easy to predict that the assembly of Portuguese tile panels will be Table II: AUC for the best four mean ROC curves for the Portuguese tile panel database Compatibility function (color space) AUC l-½ norm (CIELAB) (L p) q norm (CIELAB) Mahalanobis Gradient Compatibility (CIELAB) Pomeranz prediction-based compatibility (HSV)

6 0 3.2 x funcao de custo original Taxa de Comparacao de Vizinhanca funcao Lagrangiana Aumentada iteracao x iteracao Indice da imagem da base de bados Cho et al erro Taxa de Comparacao de Vizinhanca iteracao Figure 7: Example of the BALM method evolution. Green functions represent the reference values for the correctly reassembled tile panel and the blue ones represent the BALM method obtained values Indice da imagem da base de bados Cho et al Figure 6: Mean Neighbor Comparison from the assembly of the 20 puzzles from Cho et al. database using a) Orthogonal Procrustes and b)two Assignment problems. For each image the method was used 1000 different times to assemble the puzzle using random initial permutation matrix seeds. The red lines represent the median values of Neighbor Comparison and the blue box limits the 1st and 3rd quartiles. The edges of the doted lines represent the extreme values of Neighbor Comparison (disregarding the outliers that are represented by the red crosses) much harder when compared to natural images puzzles. For the tile panels, the compatibility functions presented curves with similar areas and similar shapes but l-½ norm computed on tile pieces projected on CIELAB color space proved to be the best compatibility function/color space pair. B. Assembly with known pieces orientation Two methods were proposed to assembly Portuguese Tile panels with known pieces orientation. The results for the BALM method and greedy method are as follows: BALM method: This method was firstly tested on natural images and using Mahalanobis Gradient Compatibility/CIELAB color space, since this was the compatibility function/color space pair that got best results in Section III-A. Considering that the BALM method needs a permutation matrix seed, P, for each one of the 20 images the method was tested in 1000 runs with random initial seeds, in order to minimize the dependency of the initial seed. This procedure was applied for the two heuristics proposed. Figure 6 presents the results for natural image puzzles. As an example, on Figure 7 it is presented the evolution of this method for the panel of Figure 2 and a given permutation matrix seed. For this example, BALM method managed to connect correctly 1 connection of the 12 possible. On Figure 7a it is possible to identify only two different values for the original cost function, suggesting that the Permutation Matrix, P, in (10) changed once. The behavior of this example represents most of the cases: Algorithm 1 favors changes in Q so that it matches P. In conclusion, it has been noticed that the BALM method showed a greater availability to match Q and P matrices than to match P to the solution of the optimization problem. Since this method did not solve the simplest problem of assembling natural image puzzles with known pieces orientation, it was not presented to the problem of qassembling Portuguese tile panels. Greedy method: The Distance Matrices, S lr and S tb, used in this method were computed using the compatibility function/color space pair that got the best results in Section III-A - Mahalanobis Gradient Compatibility computed on natural images projected on CIELAB color space and l-½ norm computed on tile panels projected on CIELAB color space. In the study of this assembly method, the two heuristics proposed on Section II - Multiple Margins and Lowe Scores - were considered. Since Lowe Scores requires a threshold value, values ranging from 0.05 to 0.95 with increments of 0.05 were tested. The results for the 20 puzzles with dimensions 18x24 pieces are presented on Table III. As it can be seen on Table III, the heuristic Multiple Margins produced lower Neighbor Comparison values almost every time. The reason is that compatibility functions do not classify the puzzles perfectly, therefore placing some pieces wrongly. The wrong placement of a piece implies that in the a future placement it will minimize simultaneously a distance to a correctly placed piece and a wrongly placed piece. As a result the new piece will most probably be misplaced. Without 6

7 Table III: Comparison between results obtained for natural image puzzles assembled with known pieces orientation. without any Lowe Scores Multiple Multiple Margins and Lowe Scores heuristic Margins Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Mean Variance Table IV: Comparison between Jigsaw puzzle solving methods for natural image database from Cho et al. and known pieces orientation. Figure 8: Example of the assembly of Image 14 from Cho et al. natural image database. On the left image the Lowe Scores heuristic is used, assembling the sky region at the end. On the right image no heuristic was used, assembling the sky region at an early stage. Proposed Cho et al. Pomeranz et al. Gallagher (2012) (2010) [20] (2011) [21] (2012) [22] Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Mean Variance Multiple Margins heuristic this will not happen, since for each new placed piece only one connection is minimized, making it possible to create at least one correct connection. On the other hand, the Lowe Scores heuristic produced higher Neighbor Comparison values almost every time, proving that the rejection of new connections anytime they do not stand out from the rest, it is an asset. This heuristic is extremely useful on homogeneous regions of the puzzles. These regions, e.g. sky, usually have connections with very low distance values. Without this heuristic, a greedy algorithm will assemble these regions in an early stage of the assembly, making them a center region of the final puzzle. Figure 8 is an example of the use of Lowe Scores heuristic. On the left image the heuristic is used, assembling the sky region at the end, and on the right image no heuristic was used, assembling the sky region at an early stage. To study the effect of the threshold value on the assembly, we tested values from 0.05 to 0.95, with increments of On Table III only 0.50 to 0.75 threshold values are presented, since these were the most significant ones. In conclusion, the proposed method producing best results for the assembly of natural image puzzles when the orientation of each piece is known is the greedy algorithm with Lowe Scores heuristic and a 0.55 threshold value. Our best methodology was compared with three other methods, tested using the available software from their authors [20] [21] [22]. As it can be concluded from Table IV, the proposed method consistently achieved results close to the state-of-theart method [4]. The results for the 12 Portuguese tile panels are presented on Table V. As for the results obtained for the natural image puzzles, Lowe Scores heuristic produced higher Neighbor Comparison values almost every time. The rejection of a new connection anytime it does not stand out from the rest of the possible connections as the best one, proved its value again on tile panel assembly. 7

8 Table V: Comparison between results obtained for Portuguese tile panels assembled with known piece orientation. without any Lowe Scores Multiple Multiple Margins and Lowe Scores heuristic Margins Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Media Variance It is further noted that the threshold value that produced best results is 0.90, which is much higher that the one obtained for the natural image puzzles. The reason behind this is related to the compatibility functions performance, which as studied before on Section III-A produced inferior results on the tile panels than the natural image puzzles. Because of that, a new connection must not stand out as much from the rest of the possible connections from the tile panels as it did on the natural images puzzles. In conclusion, the proposed method which produced best results for the assembly of Portuguese tile panels when the orientation of each piece is known was the greedy algorithm with Lowe Scores heuristic and a 0.90 threshold value. C. Assembly with unknown pieces orientation The Distance Matrices, S, used in this method were computed using the compatibility function/color space pair that got the best results in Section III-A, Mahalanobis Gradient Compatibility computed on natural images projected on CIELAB color space and l-½ norm computed on tile panels projected on CIELAB color space. In the study of this assembly method, the two heuristics proposed on Section II - Multiple Margins and Lowe Scores - were considered. Since Lowe Scores requires a threshold value, values ranging from 0.05 to 0.95 with increments of 0.05 were tested. The results for the 20 puzzles from the Cho et al. database are presented on Table VI. Lowe Scores heuristics once again produced higher Neighbor Comparison values almost every time, perfectly reconstructing 12 puzzles (even if with different threshold values). In conclusion, the proposed method which produced best results for the assembly of natural image puzzles when the orientation of each piece is unknown was the greedy algorithm with Lowe Scores heuristic and a 0.65 threshold value. The proposed method was compared with the only other method that proposes to solve the problem. The results of this comparison are presented on Table VII. As it can be concluded from Table IV, the proposed method surpasses the state-of-the-art method and presents more consistent results, managing to perfectly reconstruct 9 of the 20 natural image puzzles. Table VII: Comparison between Jigsaw puzzle solving methods for natural image database from Cho et al. and unknown pieces orientation. Proposed Gallagher Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Mean Variance The results for the 12 Portuguese tile panels are presented on Table VIII. Just like the assembly of natural image puzzles, the best Neighbor Comparison values were obtained with Lowe Scores heuristic. The rejection of a new connection anytime it does not stand out from the rest of the possible connections as the best one, proved its value again in tile panel assembly. It is further noted that, just like in the assembly of Portuguese tile panels when the orientation of each tile is known, the threshold values that produced better Neighbor Comparison values is much higher than the one obtained for the assembly of natural image puzzles when the orientation of each tile is unknown. Section III-A showed that the compatibility functions performance decayed for the tile panels classification, when compared to the classification of natural image puzzles. As a consequence a new connection will not stand out as much from the rest of the possible connections for the tile panels as it did on the natural images puzzles. In conclusion, the proposed method which produced best results for the assembly of Portuguese tile panels when the orientation of each piece is unknown was the greedy algorithm with Lowe Scores heuristic and a 0.90 threshold value. 8

9 Table VI: Comparison between results obtained for natural image puzzles assembled with unknown piece orientation. without any Lowe Scores Multiple Multiple Margins and Lowe Scores heuristic Margins Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Image Mean Variance Table VIII: Comparison between results obtained for Portuguese tile panels assembled with unknown piece orientation. without any Lowe Scores Multiple Multiple Margins and Lowe Scores heuristic Margins Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Panel Mean Variance IV. APPLICATION SOFTWARE During the development of this work two software applications were created. One application was created to capture Portuguese tiles and correct the images, while the other had the task of gathering all the compatibility functions and assembly methods while providing a test platform. On the first application it is possible to live capture Portuguese tiles or load a group of previously taken photographies. After that, the images are automatically rectified - the corners of the Portuguese tiles are found using the Hough transform [23] and then applied an homography, correcting the perspective and cropping the original image so that the squared tile fills the hole image. The result is displayed to the user, which can either save the image or manually correct the original image by clicking directly on the corners of the Portuguese tile. The second application was created with the aim of gathering all the compatibility functions and assembly methods used through this work. The user starts by choosing a group of Portuguese tile images or by choosing a natural image to split into a puzzle. Using a menu, the user is now able to choose a color space and a compatibility function to compute the Distance Matrix, S. The application provides tools to analyze the Distance Matrix, giving the possibility to graphically understand the Distance matrix. After computing the Distance Matrix, it is possible to proceed to the tile panel assembly. The user is able to select two different assembly settings and compare their results. Some details about the assembly are available to the user, e.g. placement order, connections distance value and neighbor comparison. V. CONCLUSIONS This work studies the assembly of a Portuguese tile panel by dividing this problem in two parts: measuring the compatibility between tiles and devising an assembly strategy. For the first subproblem, six compatibility functions were proposed: Euclidean distance, l-½ norm, (L p ) q norm, Pomeranz prediction-based compatibility, Mahalanobis Gradient Compatibility and a distance function learned via KLFDA, being all of the compatibility function computed in three different color spaces: RGB, HSV and CIELAB. By the study of the ROC curves obtained for each compatibility function/color space pair, it was possible to identify the ones that best classified natural image puzzles and Portuguese tile panels. For the natural image puzzles, the compatibility function/color space pair that provided best results was Mahalanobis Gradient Compatibility computed on natural images projected on CIELAB color space and for the tile panels it was l-½ norm computed on tile panels projected on CIELAB color space. We proceeded to study the assembly of a Portuguese tile panel when the orientation of the pieces was known. The first proposed method, BALM method, produced unsatisfying results. The second proposed method, greedy approach, 9

10 achieved great results assembling natural image puzzles, when assisted by the Lowe Scores heuristic with a threshold of This method was able to assemble, in average, 93.80% of a natural image puzzle which is comparable to the state-ofthe-art. When applied to Portuguese tile panels, the proposed method was able to assemble, on average, 57.82% of a tile panel, when assisted by the Lowe Scores heuristic with a threshold of We study the assembly of a Portuguese tile panel when the orientation of the pieces was unknown. The only method proposed was a greedy algorithm, that surpasses the state-ofthe-art, when assembling natural image puzzles. Assisted by the Lowe Scores heuristic with a threshold of 0.65, this method was able to assemble, in average, 92.63% of a natural image puzzle. When applied to Portuguese tile panels, this method was able to assemble, on average, 35.91% of a tile panel, when assisted by the Lowe Scores heuristic with a threshold of Future Work In order to improve the quality of the automatic Portuguese tile panel jigsaw puzzle solver, some improvements might be applied. Due to the difficulties of working with Portuguese tiles, relevance feedback from the user can be a major improvement. At the end of the assembly, the user could be able to select which connections are correct and incorrect. With this knowledge, the Distance Matrix can be modified, changing the correct connections to zero distance and the incorrect connections to an extremely high value of distance. After that, the method is rerun, hoping to achieve better results. Usually the Portuguese tile panels have a border. This border makes a big discontinuity to the inside image, creating a bigger difficulty to the assembly. If the border tile pieces are identified (automatically or manually), it is possible to estimate the dimensions of the panel. As Pomeranz et al. [14] used in their assembly method, the dimensions of the tile panel would be a major gain. Taking in consideration the development of a software end product, the two applications developed in this work can be merged together making it possible to capture the Portuguese tile images, correct the perspective and proceed to the tile panel assembly in a single application. [7] M. G. Chung, M. M. Fleck, and D. A. Forsyth, Jigsaw puzzle solver using shape and color, Proceedings of the International Conference on Signal Processing, pp , [8] D. Goldberg, C. Malon, and M. Bern, A global approach to automatic solution of jigsaw puzzles, Symposium on Computational Geometry, pp , [9] F.-H. Yao and G.-F. Shao, A shape and image merging technique to solve jigsaw puzzles, Pattern Recognition Letters, vol. 24, pp , [10] M. Weiss-Cohen and Y. Halevi, Knowledge retrieval for automatic solving of jigsaw puzzles, Computational Intelligence for Modelling, Control and Automation 2005 and International Conference on Intelligent Agents, Web Technologies and Internet Commerce, International Conference, vol. 2, pp , [11] E. Demaine and M. Demaine, Jigsaw puzzles, edge matching, and polyomino packing: Connections and complexity, Graphs and Combinatorics, vol. 23, pp , [12] T. Murakami, F. Toyama, K. Shoji, and J. Miyamichi, Assembly of puzzles by connecting between blocks, ICPR, pp. 1 4, [13] T. S. Cho, S. Avidan, and W. T. Freeman, A probabilistic image jigsaw puzzle solver, Proc. CVPR, pp , [14] D. Pomeranz, M. Shemesh, and O. Ben-Shahar, A fully automated greedy square jigsaw puzzle solver, Proc. CVPR, pp. 9 16, [15] T. S. Cho, M. Butman, S. Avidan, and W. T. Freeman, The patch transform and its applications to image editing, Proc. CVPR, [16] M. Sugiyama, Local fisher discriminant analysis for supervised dimensionality reduction, Proc. 23rd International Conference on Machine Learning, pp , [17] S. Arora and B. Barak, Complexity Theory: A Modern Approach. Cambridge University Press, [18] H. W. Kuhn, The hungarian method for the assignment problem, Naval Research Logistic Quarterly, vol. 2, pp , [19] D. G. Lowe, Distinctive image features from scale-invariant keypoints, International Journal Compututer Vision, vol. 60, pp , [20] T. S. Cho, S. Avidan, and W. T. Freeman, A probabilistic image jigsaw puzzle solver, consultado a 10 de Setembro [Online]. Available: [21] D. Pomeranz, M. Shemesh, and B.-S. O., Icvl - jigsaw solver project, consultado a 10 de Setembro [Online]. Available: icvl/projects/project-jigsaw.html [22] A. Gallagher, Jigsaw puzzles with pieces of unknown orientation, consultado a 10 de Setembro [Online]. Available: [23] R. Duda and P. Hart, Use of the hough transformation to detect lines and curves in pictures, AI Center, SRI International, Tech. Rep. 36, REFERENCES [1] A. D. Williams, Jigsaw puzzles: an illustrated history and price guide. Wallace-Homestead Book Company, [2] H. Freeman and L. Garder, Apictorial jigsaw puzzles: the computer solution of a problem in pattern recognition, Electronic Computers, IEEE Transactions, vol. 13, pp , [3] A. Del Bue, J. Xavier, L. Agapito, and M. Paladini, Bilinear modelling via augmented lagrange multipliers (balm), Pattern Analysis and Machine Intelligence, pp. 1 14, [4] A. Gallagher, Jigsaw puzzles with pieces of unknown orientation, Proc. CVPR, [5] H. Wolfson, E. Schonberg, A. Kalvin, and Y. Lamdan, Solving jigsaw puzzles by computer, Annals of Operations Research, vol. 12, pp , [6] D. Kosiba, P. Devaux, S. Balasubramanian, T. Gandhi, and K. Kasturi, An automatic jigsaw puzzle solver, Pattern Recognition, vol. 1, pp ,