Solving Jigsaw Puzzles by a Computer

1
Solving Jigsaw Puzzles by a Computer

• Ananda Chowdhury
• Visual and Parallel Computing Laboratory
• Department of Computer Science,
• University of Georgia.

2
Abstract

• An algorithm is presented for assembling large
  jigsaw puzzles using curve matching and
  combinatorial optimization techniques. The
  algorithm was successfully applied in the
  assembly of 104 piece puzzle (arranged in a 13 x
  8 grid) with many almost similar pieces.
  Furthermore, the proposed technique was also able
  to solve an intermixed puzzle assembly problem.

Keywords Computer Vision, Curve Matching, Jigsaw
Puzzle Assembly, Traveling Salesman Problem,
Assignment problem, Pattern Recognition.
3
Problem Definition

• 104 pieces of rectangular jigsaw puzzles of
  overall size of 18 x 13. The intermixed puzzle
  experiment has used the same two puzzles.
• Note that since all the puzzles have same size,
  they cannot be distinguished by any external
  parameter such as size.
• Every interior puzzle piece has 4 neighbors and
  every frame piece except the corner pieces has 3
  neighbors. The puzzle is arranged in a grid.

4
Basic Philosophy

• Local 2-D curve matching by Schwartz-Sharir
  algorithm
• In order to handle a relatively large number of
  puzzle pieces, many strong similarities between
  their boundaries and additional level of noise
  (since the pieces were photographed, some noise
  were introduced), we need to consider some
  global matching.
• Thus we have a 2 phase algorithm, where the first
  phase uses local matching and the second phase
  involves global matching with the result of the
  first phase as its input.

5
Preprocessing Steps

• All the puzzle pieces photographed by a camera
  and then thresholded to get binary image of each
  piece.
• Then boundary of each piece is extracted,
  followed by a polygonal approximation. This
  smoothes the boundary as well as eliminates some
  noise.
• The boundary curve of each piece is divided into
  4 sub-curves corresponding to its four sides.
  These will be used in the Schwartz-Sharir 2-D
  curve matching.
• Next corners are determined for each of the
  pieces using following methods
• - (a) finding all points where tangents has a
  sharp change in direction and hence the second
  derivative reaches a peak.
• - (b) A small portion of a right angle is
  matched against the boundary curve using
  Schwartz-Sharir algorithm.

6
Local Matching Procedure using the
Schwartz-Sharir algorithm

• Given two curves C and C with a sequence of
  evenly spaced points on it, (uj), j 1,, n
  (vj), j 1,, m.
• Matching thus amounts to find an Euclidean motion
  E of the plane which will minimize the L2 norm
  between the two sequences (E uj), j 1,, n and
  (vj), j 1,, m
• n
• ? min S Euj - vj2
  
• j1

First translate C so that Suj 0. Then write Eu
R? u a, where R? is a counterclockwise
rotation by ?. The value of a which yield the
best match is given by
a (1/n) S vj The least
square distance for this best match is given by
? S vj2
(1/n) S vj2 S vj2 2S uj vj

For a N-piece puzzle, we get a 4N x 4N symmetric
matrix of the best matching scores between every
two such sides. O(N2) matchings in total.
7
The Puzzle Assembly Algorithm

• This can be broken down into two subparts frame
  assembly and interior assembly.
• The frame assembly problem is formulated as an
  equivalent Traveling Salesman Problem (TSP).
• The interior assembly problem is solved using a
  greedy algorithm. (with some backtracking or
  branch and bound)
• For the intermixed puzzle assembly, an equivalent
  Assignment problem is formulated.

8
Some basic TSP stuff

• Actual TSP problem Given K cities and the
  distances between each pair of them, find the
  shortest close path which passes through every
  city exactly once.
• A tour is a closed path that visits every city
  exactly once. The problem is to find a tour with
  minimal total length. We can take F all cyclic
  permutations ? on K cities
• A cyclic permutation ? represents a tour if we
  interpret ?(j) to be the city visited after city
  j, j 1,2,.., K. Then the cost z maps ? to S dj
  ?(j)
• TSP becomes asymmetric when the direction of
  travel from one city to the other matters. Then
  we can introduce one-zero variables i.e. if the
  edge i - j exists in a tour, where i and j are
  two cities , then the corresponding one-zero
  variable say xij 1, else xij 0.

9
TSP to Assignment

• The constraint of single tour with precisely one
  cycle is removed from the TSP to make it an
  Assignment problem.
• Now the problem becomes

with two added constraints
j 1,2,, K
i 1,2,., K
10
Two different frame piece ?
Three different interior puzzle pieces ?
11
Assembled 104-piece puzzle ?
An assembled frame of the puzzle shown above ?
12
Frame Assembly part

• Pieces having an almost straight section between
  adjacent corners are identified as frame pieces.
  The four corner pieces will have two bottom sides
  and one right and one left side. (We name the
  sides counter-clockwise as bottom, right, top and
  left)
• A matrix M (i, j), i, j 1,2,,K, is defined.
  K denotes the no. of frame pieces. Each M (i, j)
  is the inter-curve L2 distance obtained by best
  matching the right side of piece i to the left
  side of piece j.
• Note that any choice of exactly K entries, one
  from each row and one from each column, defines a
  K-permutation P of the frame pieces so that P(i)
  ji , where ji is the entry chosen from the ith
  row.

13
TSP formulation

• Our goal is to find K entries in the matrix M,
  one in each row and one in each column in such a
  way that the sum of these K entries is minimal
  and the K-permutation P is a cycle.
• Actual TSP problem Given K cities and the
  distances between each pair of them, find the
  shortest close path which passes through every
  city exactly once.
• For the present case, we must solve the
  asymmetric version of the TSP, which means that
  the distance from city i to city j is not
  necessarily equal to the distance from city j to
  city i.

14
Approximate Solution of the TSP problem

• TSP is NP-Complete. However for small K, we can
  find efficient algorithms to solve it. For our
  case K ?
• Two classes of algorithms can be useful for
  solving TSP
• (a) time-efficient algorithms which use
  heuristic methods and assure discovery of the
  optimal solution with high probability (Raymond
  1969, Christophides 1975 etc., Fencl in CACM)
• (b) algorithms which guarantees optimal
  solution but may be very time-consuming (Belmore
  et al., 1971)
• A type (a) algorithm (Fencl in CACM) was chosen
  for solving the frame assembly problem.

15
Solution in greater details

• A 3-step interactive/iterative solution is
  proposed as below -
• (a) TSP problem is applied to the problem
  defined by the matrix M and the solution is fed
  to a robot.
• (b) Check the proposed solution by the robot
  and return to the computer information about
  correct, incorrect and undecidable matches
  between suggested and neighboring pieces.
• (c) If all the matches are correct, stop.
• Else If a match (i, j) was correct, assign
  zero to the (i, j) entry and almost infinity
  to all other entries of row i and column j.
• Else if a match (i, j) was incorrect, assign
  almost infinity to the entry (i, j).
• Else if a match (i, j) was undecidable, make
  no change in the matrix.
• Go back to step (a).
• In the present paper, the robotic assembly
  is not used. Since the initial M matrix is quite
  favorable to yield an optimal solution (why ?) ,
  so the correct frame assembly was obtained by
  just applying the TSP algorithm once on M.

16
Arrangement of the puzzle interior

• A plain greedy algorithm will not succeed (why ?)
• Use some backtracking or branch and bound
  algorithm instead.
• Steps of this algorithm (outline)
• (a) Corners processed sequentially, beginning
  with the lower left side of the puzzle interior
  and advancing from left to right within each row
• (b) For the first corner, all non-frame puzzle
  pieces in all possible rotations are tried and
  their local matching score is computed. The
  results are stored and KBEST solutions are passed
  onto next stage.
• (c) At the second corner, the same procedure
  is repeated for each KBEST partial solutions
  (from (b)) and only KBEST overall best solutions
  are passed to the next stage.
• (d) The algorithm proceeds iteratively. At
  the last corner in each row, we have ? sides to
  match. for the last piece ? sides to match
• In the present problem KBEST 200, the
  correct solution lay among the 10 best solutions.

17
Complexity of the proposed algorithm

• N piece puzzle with K frame pieces and O(m)
  sample points on the boundary of a piece.
• For frame pieces O(max(K3, K2 m log m))
• operations if we treat this as an Assignment
  problem instead of an exact TSP.
• For interior pieces O(N2 X KBEST X m log m)
  operations
• Polynomial time algorithm !

18
Solution of an intermixed double puzzle

• Two 104-pieces puzzles were intermixed and
  treated as a single 208-piece puzzle.
• Pre-processing and local matching were done in
  the same way as before
• The frame pieces after being recognized and
  separated was assembled using the Assignment
  problem. (Why not TSP ?)
• of cycles of different puzzles processed
  simultaneously
• size of the cycle size of the puzzle frames

19
Intermixed double puzzle problem (contd.)

• Once frame pieces are assembled, the interior
  pieces can be arranged by same branch and bound
  method.
• All the interior pieces compete on all the
  corners in the different puzzles. The assemblies
  of the different interiors may proceed in
  parallel.
• Same time complexity i.e.
• O( max (K3, K2 m log m)) Frame pieces
• O(N2 x KBEST x m log m) Interior pieces

20
Reference to Craniofacial Reconstruction

• We have different broken human mandible
  fragments. The goal is to reconstruct the
  mandible in silico.
• For multiple fragments, we would like to classify
  the fragments into terminal and non-terminal
  fragments. Then we can plan for a reconstruction
  strategy of what pieces will go together. For a
  large number of fragments, the reconstruction
  problem bears close resemblance to a 3D jigsaw
  puzzle.

21
References

• H. Wolfson, E. Schonberg, A. Kalvin and Y.
  Lamdan, Solving Jigsaw Puzzles By Computer.
  Annals of Operations Research. 12(1988), pp. 51 -
  64.
• H. Freeman and L. Garder, Apictorial jigsaw
  puzzles The computer solution of a problem in
  pattern recognition, IEEE Trans. on Electronic
  Comp. EC-13, 2(1964), pp. 118 - 127.
• J.T. Schwartz and M. Sharir, Identification of
  partially obscured objects in two dimensions by
  matching of noisy characteristic curves.
  International Journal of Robotics Research.
  6(1987), pp. 29 44.
• Z. Fencl. CACM Algorithm 456.
• N. Christophides. Graph Theory (Academic Press,
  1975).
• M.R. Garey and D.S. Johnson. Computers and
  Intractability A Guide to the Theory of
  NP-Completeness (W.H. Freeman and Co. 1979)
• E.L. Lawler, J.R. Lenstra, A.H.G. Rinnooy Kan and
  D.B. Shmoys. The Traveling Salesman Problem A
  Guided Tour of Combinatorial Optimization (Wiley,
  1985)
• M. Belmore and J.C. Malone, The
  traveling-salesman problem A survey, Annals of
  Operations Research. 16(1968) 538.
• T.C. Raymond, Heuristic algorithm for the
  traveling-salesman problem, IBM J. Research
  Development. 13, 4(1969)400.

22

• Thank you very much for your patience !!!