Dynamic Programming: The Edit Distance Problem

1
Dynamic ProgrammingThe Edit Distance Problem

• CS 2 Introduction to Programming Methods
• 9 February 2004

2
Today

• A review of dynamic programming.
• The Edit Distance problem.

3
ReviewDynamic Programming
4
Dynamic programming

• A method for developing algorithms.
• Given a problem to solve, dynamic programming
  says
• Solve subproblems of the problem first.
• Remember these solutions for later.
• Then put together a solution to the problem.
• Think of this as a bottom-up approach.
• Solve problems in order of increasing size.

5
Example The fibonacci sequence

• fib(0) 0
• fib(1) 1
• fib(2) 1
• fib(3) 2
• fib(4) 3
• fib(5) 5
• fib(n) fib(n-1) fib(n-2)

6
Naïvely computing fib(n)

• int fib(int n)
• if (n 0 n 1)
• return n
• else
• return (fib(n-1) fib(n-2))

7
Naïvely computing fib(n)
fib(n)
fib(n-2)
fib(n-1)
fib(n-4)
fib(n-3)
fib(n-3)
fib(n-2)
fib(n-6)
fib(n-5)
fib(n-5)
fib(n-4)
fib(n-5)
fib(n-4)
fib(n-4)
fib(n-3)
8
Naïvely computing fib(n)
fib(n)
fib(n-2)
fib(n-1)
fib(n-4)
fib(n-3)
fib(n-3)
fib(n-2)
fib(n-6)
fib(n-5)
fib(n-5)
fib(n-4)
fib(n-5)
fib(n-4)
fib(n-4)
fib(n-3)
9
Naïvely computing fib(n)
fib(n)
fib(n-2)
fib(n-1)
fib(n-4)
fib(n-3)
fib(n-3)
fib(n-2)
fib(n-6)
fib(n-5)
fib(n-5)
fib(n-4)
fib(n-5)
fib(n-4)
fib(n-4)
fib(n-3)
10
Naïvely computing fib(n)
fib(n)
fib(n-2)
fib(n-1)
fib(n-4)
fib(n-3)
fib(n-3)
fib(n-2)
fib(n-6)
fib(n-5)
fib(n-5)
fib(n-4)
fib(n-5)
fib(n-4)
fib(n-4)
fib(n-3)
11
How to fix things...

• The solution to fib(n) depends on
• The solution to fib(n - 1).
• The solution to fib(n - 2).
• Use dynamic programming to organize everything
• Each subproblem is characterized by the value of
  its input.
• To compute fib(n), we need fib(n - 2) and fib(n
  - 1).
• So compute fib(0) first, and work upwards.
• Use an array to keep track of everything.

12
Computing fib(n) Take 2

• int fib(int n)
• int seq new intn1
• seq0 0
• seq1 1
• for (int i 2 i lt n i)
• // looks like fib(i) fib(i - 1) fib(i - 2)
• seqi seqi - 1 seqi - 2
• return seqn

13
Computing fib(n) Take 2

• int fib(int n)
• // This array will keep track of the solutions
  to
• // all n1 problems.
• int seq new intn 1
• ...

14
Computing fib(n) Take 2

• int fib(int n)
• ...
• // Assume that (n gt 1). Otherwise, seq is to
  small.
• // The smallest subproblems have trivial
  solutions.
• seq0 0
• seq1 1
• ...

15
Computing fib(n) Take 2

• int fib(int n)
• ...
• // Solve other subproblems in order of
  increasing size.
• // Use the fact that weve already solved even
  smaller
• // subproblems already.
• for (int i 2 i lt n i)
• seqi seqi - 1 seqi - 2
• ...

16
Computing fib(n) Take 2

• int fib(int n)
• ...
• // Finally, return solution to the actual
  problem.
• return seqn

17
The Edit Distance Problem
18
The Edit Distance problem

• Problem what is the cheapest way to transform
  one word (the source) into another word (the
  output)?
• Example transform algorithm into alligator.
• Initially, you start at the first character of
  the source and have an empty output.
• At any point, you can
• Delete the current character of the source.
• Insert a new character into the output word.
• Copy the current character of the source into the
  output.
• Copying and deleting move you to the next
  character.

19
Example algorithm ? alligator

• Source a l g o r i t h m
• Output
• Operations (none)

20
Example algorithm ? alligator

• Source a l g o r i t h m
• Output
• Operations (none)

21
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a
• Operations Copy

22
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l
• Operations Copy, Copy

23
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l
• Operations Copy, Copy, Insert(l)

24
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i
• Operations Copy, Copy, Insert(l), Insert(i)

25
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy

26
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete

27
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete

28
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete, Delete

29
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g a
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete, Delete, Insert(a)

30
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g a t
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete, Delete, Insert(a), Copy

31
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g a t
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete, Delete, Insert(a), Copy,
  Delete,
• Delete

32
Example algorithm ? alligator

• Source a l g o r i t h m
• Output a l l i g a t o r
• Operations Copy, Copy, Insert(l), Insert(i),
  Copy
• Delete, Delete, Delete, Insert(a), Copy,
  Delete,
• Delete, Insert(o), Insert(r)

33
The Edit Distance problem

• Problem what is the cheapest way to transform
  one word (the source) into another word (the
  output)?
• At any point, you can
• Delete the current character of the source.
• Insert a new character into the output word.
• Copy the current character of the source into the
  output.
• Each operation has a cost associated with it.
• The cost of a transformation is the sum of the
  costs of each operation in the sequence.

34
Example algorithm ? alligator

• Operations Copy, Copy, Insert(l), Insert(i),
  Copy Delete, Delete, Delete, Insert(a), Copy,
  Delete, Delete, Insert(o), Insert(r)
• Assume that
• Copying costs 3
• Inserting costs 5
• Deleting costs 2
• The cost of the above transformation is 47
• This is just 3 3 5 5 3 2 2 ...

35
The Edit Distance problem

• Problem what is the cheapest way to transform
  one word (the source) into another word (the
  output)?
• At any point, you can
• Delete the current character of the source.
• Insert a new character into the output word.
• Copy the current character of the source into the
  output.
• By cheapest, we mean the transformation with
  the least cost.

36
How to solve this problem?

• Will trying out all possible transformations work?

37
How to solve this problem?

• Will trying out all possible transformations
  work?
• Answer It can, but it would take way too long.

38
Will dynamic programming work?

• Depends on our problem.
• Can our problem be solved based on the solution
  of some subproblems?
• What are the subproblems, if there any?

39
Will dynamic programming work?

• Depends on our problem.
• Can our problem be solved based on the solution
  of some subproblems?
• What are the subproblems, if there any?
• As is typical when trying out dynamic
  programming, it helps to find some key
  observation or insight.
• We will need three observations for this problem.

40
Notation

• (s1, s2, s3, ..., sn) stands for a sequence of n
  elements.
• Used for a list of operations.
• Example (Copy, Copy, Insert(c), Delete).
• Also used for strings.
• Example help would correspond to (h, e, l ,p).

41
Key Observation 1

• Given Strings X (x1, ..., xn) and Y (y1,
  ..., yt), anda sequence S of operations (s1, s2,
  ..., sm).
• Let S (s1, ..., s(m-1)), X (x1, ...,
  x(n-1)), andY (y1, ..., y(t-1)).
• Suppose that S is the cheapest sequence of
  operations to transform X into Y, and that sm is
  a Copy operation.

42
Key Observation 1 (cont.)

• Claim then, S is the cheapest sequence of
  operations to transform X to Y.
• Proof by contradiction. Suppose that T (t1,
  ..., tk) is cheaper than S and tranforms X to
  Y.
• Then (t1, ..., tk, sm) is cheaper than S and
  transforms X to Y. This is a contradiction, as S
  was supposed to be the cheapest way to transform
  X to Y.

43
Key Observation 2

• Given Strings X (x1, ..., xn) and Y (y1,
  ..., yt), anda sequence S of operations (s1, s2,
  ..., sm).
• Let S (s1, ..., s(m-1)) and X (x1, ...,
  x(n-1)).
• Suppose that S is the cheapest sequence of
  operations to transform X into Y, and that sm is
  a Delete operation.

44
Key Observation 2 (cont.)

• Claim then, S is the cheapest sequence of
  operations to transform X to Y.
• Proof by contradiction. Suppose that T (t1,
  ..., tk) is cheaper than S and tranforms X to
  Y.
• Then (t1, ..., tk, sm) is cheaper than S and
  transforms X to Y. This is a contradiction, as S
  was supposed to be the cheapest way to transform
  X to Y.

45
Key Observation 3

• Given Strings X (x1, ..., xn) and Y (y1,
  ..., yt), anda sequence S of operations (s1, s2,
  ..., sm).
• Let S (s1, ..., s(m-1)) and Y (y1, ...,
  y(t-1)).
• Suppose that S is the cheapest sequence of
  operations to transform X into Y, and that sm is
  an Insert(yn) operation.

46
Key Observation 3 (cont.)

• Claim then, S is the cheapest sequence of
  operations to transform X to Y.
• Proof by contradiction. Suppose that T (t1,
  ..., tk) is cheaper than S and tranforms X to
  Y.
• Then (t1, ..., tk, sm) is cheaper than S and
  transforms X to Y. This is a contradiction, as S
  was supposed to be the cheapest way to transform
  X to Y.

47
An observation about the observations...

• Given X, Y, and S, we have covered all possible
  cases for sm (the last operation).

48
Putting the observations to use

• The best sequence of operations to transform X to
  Y depended on one of the following
• (1) The best way to transform X to Y.
• (2) The best way to transform X to Y.
• (3) The best way to transform X to Y.
• The three cases become the subproblems to
  consider.
• Given the solution to all three, we can find the
  solution to our actual problem (transforming X to
  Y).
• Why? The best solution to transforming X to Y
  must contain a solution to one of the three cases
  by the three observations.

49
How to organize the subproblems

• Each subproblem is characterized by some initial
  part of the original strings X and Y.
• So use a matrix.

X
Y
50
How to organize the subproblems (cont.)

• The marked location contains the score for the
  cheapest way to transform algo to a.

X
Y
51
How to organize the subproblems (cont.)

• The marked location contains the score for the
  cheapest way to transform alg to .

X
Y
52
How to organize the subproblems (cont.)

• The marked location contains the score for the
  cheapest way to transform to a.

X
Y
53
How to organize the subproblems (cont.)

• The marked location contains the score for the
  cheapest way to transform to .
• This is the smallest problem possible.

X
Y
54
Additional information to store

• Each location should store the last operation in
  the cheapest sequence of operations used to get
  there.

X
Y
55
Additional information to store (cont.)

• Example the marked location would contain a
  score, and possibly a Copy operation.

X
Y
56
How to fill out the matrix

• For concreteness, assume that Copy costs 5,
  Insert(c) costs 10, and Delete costs 10.

X
Y
57
How to fill out the matrix (cont.)

• Start at the upper left.
• This is trivial score is zero, operation is null.

X
Y
58
How to fill out the matrix (cont.)

• All other locations depend on the values in up to
  three other places.

X
Y
59
How to fill out the matrix (cont.)

• Diagonal movement corresponds to a Copy
  operation.
• Vertical ? Insert(c).
• Horizontal ? Delete.

X
Y
60
How to fill out the matrix (cont.)

• Fill out each row in turn.
• Only one option for the first row...

X
Y
61
How to fill out the matrix (cont.)

• Fill out each row in turn.
• Only one option for the first row...

X
Y
62
How to fill out the matrix (cont.)

• Fill out each row in turn.

X
Y
63
How to fill out the matrix (cont.)

• Three possibilities to consider for the marked
  square.
• Clear that Copy is cheapest. (5 versus 20 for
  Insert(a) or Delete).

X
Y
64
How to fill out the matrix (cont.)

• Three possibilities to consider for the marked
  square.
• Clear that Copy is cheapest. (5 versus 20 for
  Insert(a) or Delete).

X
Y
65
How to fill out the matrix (cont.)

• Two possibilities to consider for the marked
  square.
• Copy is not possible since a ! l.

X
Y
66
How to fill out the matrix (cont.)

• If we chose Insert(a)....

X
Y
67
How to fill out the matrix (cont.)

• But Delete is cheaper!

X
Y
68
How to fill out the matrix (cont.)

• Fill the rest of the matrix out in a similar
  fashion.

X
Y
69
Transforming algori to al

• The cheapest sequence of operations has cost 50

X
Y
70
Transforming algori to al (cont.)

• Recover the sequence by working backwards.
• We get Copy, Copy, Delete, Delete, Delete,
  Delete.

X
Y
71
Why this worked

• This worked becuase of the three observations we
  made earlier.

X
Y
72
Why this worked (cont.)

• The best answer to put in a location must use the
  best solution to one of three possible
  subproblems.

X
Y
73
Why this worked (cont.)

• So we solved those subproblems first.
• Then we considered cases and figured out how best
  to solve our current problem.

X
Y
74
Why this worked (cont.)

• We could recover the cheapest sequence of
  operations since we stored operations at each
  step.

X
Y
75
Why this worked (cont.)

• Each locations operation tells us where to look
  for the previous one in the sequence.

X
Y