Dynamic Programming

1
Dynamic Programming

• Longest Common Subsequence

2
Common subsequence

• A subsequence of a string is the string with zero
  or more chars left out
• A common subsequence of two strings
• A subsequence of both strings
• Ex x A B C B D A B , y B D C A B A
• B C and A A are both common subsequences of x
  and y

3
Longest Common Subsequence

• Given two sequences x1 . . m and y1 . . n,
  find a longest subsequence common to them both.

4
Brute-force LCS algorithm
Check every subsequence of x1 . . m to see if
it is also a subsequence of y1 . . n.

• Analysis
• 2m subsequences of x (each bit-vector of length m
  determines a distinct subsequence of x).
• Hence, the runtime would be exponential !
• Towards a better algorithm a DP strategy
• Key optimal substructure and overlapping
  sub-problems
• First well find the length of LCS. Later well
  modify the algorithm to find LCS itself.

5
Optimal substructure

• Notice that the LCS problem has optimal
  substructure parts of the final solution are
  solutions of subproblems.
• If z LCS(x, y), then any prefix of z is an LCS
  of a prefix of x and a prefix of y.
• Subproblems find LCS of pairs of prefixes of x
  and y

m
i
x
z
n
y
j
6
Recursive thinking
m
x
n
y

• Case 1 xmyn. There is an optimal LCS that
  matches xm with yn.
• Case 2 xm? yn. At most one of them is in LCS
• Case 2.1 xm not in LCS
• Case 2.2 yn not in LCS

7
Recursive thinking
m
x
n
y

• Case 1 xmyn
• LCS(x, y) LCS(x1..m-1, y1..n-1) xm
• Case 2 xm ? yn
• LCS(x, y) LCS(x1..m-1, y1..n) or
• LCS(x1..m, y1..n-1), whichever is
  longer

Reduce both sequences by 1 char
concatenate
Reduce either sequence by 1 char
8
Finding length of LCS
m
x
n
y

• Let ci, j be the length of LCS(x1..i,
  y1..j)
• gt cm, n is the length of LCS(x, y)
• If xm yn
• cm, n cm-1, n-1 1
• If xm ! yn
• cm, n max cm-1, n, cm, n-1

9
Generalize recursive formulation
10
Recursive algorithm for LCS
LCS(x, y, i, j) if xi y j then ci, j ?
LCS(x, y, i1, j1) 1 else ci, j ?
max LCS(x, y, i1, j), LCS(x, y, i, j1)
Worst-case xi ¹ y j, in which case the
algorithm evaluates two subproblems, each with
only one parameter decremented.
11
Recursion tree
m 3, n 4
3,4
2,4
3,3
1,4
3,2
2,3
2,3
1,3
2,2
1,3
2,2
Height m n ? work potentially exponential.
12
DP Algorithm

• Key find out the correct order to solve the
  sub-problems
• Total number of sub-problems m n

n
0
j


C(i, j)


0
i
m
13
DP Algorithm

• LCS-Length(X, Y)
• 1. m length(X) // get the of symbols in X
• 2. n length(Y) // get the of symbols in Y
• 3. for i 1 to m ci,0 0 // special case
  Y0
• 4. for j 1 to n c0,j 0 // special case
  X0
• 5. for i 1 to m // for all Xi
• 6. for j 1 to n // for all Yj
• 7. if ( Xi Yj)
• 8. ci,j ci-1,j-1 1
• 9. else ci,j max( ci-1,j, ci,j-1 )
• 10. return c

14
LCS Example

• Well see how LCS algorithm works on the
  following example
• X ABCB
• Y BDCAB

What is the LCS of X and Y?
LCS(X, Y) BCB X A B C B Y B D C
A B
15
Computing the Length of the LCS
16
LCS Example (0)
ABCB BDCAB
j 0 1 2 3 4
5
i
Yj
B
B
A
C
D
Xi
0
A
1
B
2
3
C
4
B
X ABCB m X 4 Y BDCAB n Y
5 Allocate array c5,6
17
LCS Example (1)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
B
2
0
3
C
0
4
B
0
for i 1 to m ci,0 0 for j 1 to n
c0,j 0
18
LCS Example (2)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
19
LCS Example (3)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
20
LCS Example (4)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
21
LCS Example (5)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
0
1
1
B
2
0
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
22
LCS Example (6)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
0
0
1
0
1
B
2
0
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
23
LCS Example (7)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
24
LCS Example (8)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
1
1
1
2
3
C
0
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
25
LCS Example (9)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
2
1
1
1
1
3
C
0
1
1
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
26
LCS Example (10)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
27
LCS Example (11)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
28
LCS Example (12)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
29
LCS Example (13)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1
1 else ci,j max( ci-1,j, ci,j-1 )
30
LCS Example (14)
ABCB BDCAB
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
if ( Xi Yj ) ci,j ci-1,j-1 1
else ci,j max( ci-1,j, ci,j-1 )
31
LCS Algorithm Running Time

• LCS algorithm calculates the values of each entry
  of the array cm,n
• So what is the running time?

O(mn) since each ci,j is calculated in
constant time, and there are mn elements in the
array
32
How to find actual LCS

• The algorithm just found the length of LCS, but
  not LCS itself.
• How to find the actual LCS?
• For each ci,j we know how it was acquired
• A match happens only when the first equation is
  taken
• So we can start from cm,n and go backwards,
  remember xi whenever ci,j ci-1, j-11.

2
2
For example, here ci,j ci-1,j-1 1 213
2
3
33
Finding LCS
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
Time for trace back O(mn).
34
Finding LCS (2)
j 0 1 2 3 4
5
i
B
B
A
C
D
Yj
Xi
0
0
0
0
0
0
0
A
1
0
1
0
0
0
1
B
2
0
1
2
1
1
1
3
C
0
1
1
2
2
2
3
4
B
0
1
1
2
2
LCS (reversed order)
B
C
B
LCS (straight order)
B C B (this string turned out to be a
palindrome)
35
Compute Length of an LCS
c table (represent b table)
source 91.503 textbook Cormen, et al.
36
Construct an LCS
37

• LCS-Length(X, Y) // dynamic programming solution
• m X.length()
• n Y.length()
• for i 1 to m do ci,0 0
• for j 0 to n do c0,j 0
  O(nm)
• for i 1 to m do // row
• for j 1 to n do // cloumn
• if xi yi then
• ci,j ci-1,j-1 1
• bi,j
• else if ci-1, j ? ci,j-1 then
• ci,j ci-1,j
• bi,j
• else ci,j ci,j-1
• bi,j lt

38
First Optimal-LCS initializes row 0 and column 0
39
Next each ci, j is computed, row by row,
starting at c1,1. If xi yj then ci, j
ci-1, j-11 and bi, j
40
If xi ltgt yj then ci, j max(ci-1, j, ci,
j-1) and bi, j points to the larger value
41
if ci-1, j ci, j-1 then bi,j points up
42
To construct the LCS, start in the bottom
right-hand corner and follow the arrows. A
indicates a matching character.
43
LCS
A
B
C
B
44
Constructing an LCS

• Print-LCS(b,X,i,j)
• if i 0 or j 0 then
• return
• if bi,j then
• Print-LCS(b, X, i-1, j-1)
• print xi
• else if bi,j then
• Print-LCS(b, X, i-1, j)
• else Print-LCS(b, X, i, j-1)