ICS 353: Design and Analysis of Algorithms

1
ICS 353 Design and Analysis of Algorithms
King Fahd University of Petroleum
Minerals Information Computer Science Department

• Dynamic Programming

2
Reading Assignment

• M. Alsuwaiyel, Introduction to Algorithms Design
  Techniques and Analysis, World Scientific
  Publishing Co., Inc. 1999.
• Chapter 7

3
Dynamic Programming

• Dynamic Programming algorithms address problems
  whose solution is recursive in nature, but has
  the following property The direct implementation
  of the recursive solution results in identical
  recursive calls that are executed more than once.
• Dynamic programming implements such algorithms by
  evaluating the recurrence in a bottom-up manner,
  saving intermediate results that are later used
  in computing the desired solution

4
Fibonacci Numbers

• What is the recursive algorithm that computes
  Fibonacci numbers? What is its time complexity?
• Note that it can be shown that

5
Computing the Binomial Coefficient

• Recursive Definition
• Actual Value

6
Computing the Binomial Coefficient

• What is the direct recursive algorithm for
  computing the binomial coefficient? How much does
  it cost?
• Note that

7
Optimization Problems and Dynamic Programming

• Optimization problems with certain properties
  make another class of problems that can be solved
  more efficiently using dynamic programming.
• Development of a dynamic programming solution to
  an optimization problem involves four steps
• Characterize the structure of an optimal solution
  
• Optimal substructures, where an optimal solution
  consists of sub-solutions that are optimal.
• Overlapping sub-problems where the space of
  sub-problems is small in the sense that the
  algorithm solves the same sub-problems over and
  over rather than generating new sub-problems.
• Recursively define the value of an optimal
  solution.
• Compute the value of an optimal solution in a
  bottom-up manner.
• Construct an optimal solution from the computed
  optimal value.

8
Longest Common Subsequence Problem

• Problem Definition Given two strings A and B
  over alphabet ?, determine the length of the
  longest subsequence that is common in A and B.
• A subsequence of Aa1a2an is a string of the
  form ai1ai2aik where 1?i1lti2ltltik ?n
• Example Let ? x , y , z , A xyxyxxzy,
  Byxyyzxy, and C zzyyxyz
• LCS(A,B)yxyzy Hence the length
• LCS(B,C) Hence the length
• LCS(A,C) Hence the length

9
Straight-Forward Solution

• Brute-force search
• How many subsequences exist in a string of length
  n?
• How much time needed to check a string whether it
  is a subsequence of another string of length m?
• What is the time complexity of the brute-force
  search algorithm of finding the length of the
  longest common subsequence of two strings of
  sizes n and m?

10
Dynamic Programming Solution

• Let Li,j denote the length of the longest
  common subsequence of a1a2ai and b1b2bj, which
  are substrings of A and B of lengths n and m,
  respectively. Then
• Li,j when i 0
  or j 0
• Li,j when i gt 0,
  j gt 0, aibj
• Li,j when i gt 0,
  j gt 0, ai?bj

11
LCS Algorithm

• Algorithm LCS(A,B)
• Input A and B strings of length n and m
  respectively
• Output Length of longest common subsequence of A
  and B
• Initialize Li,0 and L0,j to zero
• for i ? 1 to n do
• for j ? 1 to m do
• if ai bj then
• Li,j ? 1 Li-1,j-1
• else
• Li,j ? max(Li-1,j,Li,j-1)
• end if
• end for
• end for
• return Ln,m

12
Example (Q7.5 pp. 220)

• Find the length of the longest common subsequence
  of Axzyzzyx and Bzxyyzxz

13
Example (Cont.)
x z y z z y x
0 0 0 0 0 0 0 0
z 0
x 0
y 0
y 0
z 0
x 0
z 0
14
Example (Cont.)
x x z z y y z z z z y y x x
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
z 0 ? 0 ? 1 ? 1 ? 1 ? 1 ? 1 ? 1
x 0 ? 1 ? 1 ? 1 ? 1 ? 1 ? 1 ? 2
y 0 ? 1 ? 1 ? 2 ? 2 ? 2 ? 2 ? 2
y 0 ? 1 ? 1 ? 2 ? 2 ? 2 ? 3 ? 3
z 0 ? 1 ? 2 ? 2 ? 3 ? 3 ? 3 ? 3
x 0 ? 1 ? 2 ? 2 ? 3 ? 3 ? 3 ? 4
z 0 ? 1 ? 2 ? 2 ? 3 ? 4 ? 4 ? 4
15
Complexity Analysis of LCS Algorithm

• What is the time and space complexity of the
  algorithm?

16
Matrix Chain Multiplication

• Assume Matrices A, B, and C have dimensions 2?10,
  10?2, and 2?10 respectively. The number of scalar
  multiplications using the standard Matrix
  multiplication algorithm for
• (A B) C is
• A (B C) is
• Problem Statement Find the order of multiplying
  n matrices in which the number of scalar
  multiplications is minimum.

17
Straight-Forward Solution

• Again, let us consider the brute-force method. We
  need to compute the number of different ways that
  we can parenthesize the product of n matrices.
• e.g. how many different orderings do we have for
  the product of four matrices?
• Let f(n) denote the number of ways to
  parenthesize the product M1, M2, , Mn.
• (M1M2Mk) (M k1M k2Mn)
• What is f(2), f(3) and f(1)?

18
Catalan Numbers

• Cnf(n1)
• Using Stirlings Formula, it can be shown that
  f(n) is approximately

19
Cost of Brute Force Method

• How many possibilities do we have for
  parenthesizing n matrices?
• How much does it cost to find the number of
  scalar multiplications for one parenthesized
  expression?
• Therefore, the total cost is

20
The Recursive Solution

• Since the number of columns of each matrix Mi is
  equal to the number of rows of Mi1, we only need
  to specify the number of rows of all the
  matrices, plus the number of columns of the last
  matrix, r1, r2, , rn1 respectively.
• Let the cost of multiplying the chain MiMj
  (denoted by Mi,j) be Ci,j
• If k is an index between i1 and j, what is the
  cost of multiplying Mi,j considering multiplying
  Mi,k-1 with Mk,j?
• Therefore, C1,n

21
The Dynamic Programming Algorithm
C1,1 C1,2 C1,3 C1,4 C1,5 C1,6
C2,2 C2,3 C2,4 C2,5 C2,6
C3,3 C3,4 C3,5 C3,6
C4,4 C4,5 C4,6
C5,5 C5,6
C6,6
22
Example (Q7.11 pp. 221-222)

• Given as input 2 , 3 , 6 , 4 , 2 , 7 compute the
  minimum number of scalar multiplications

23
Example (Q7.11 pp. 221-222)
0 M1
0 M2
0 M3
0 M4
0 M5
24
Example (Q7.11 pp. 221-222)
0 M1 36 M1..M2 84 (M1..M2) M3 96 M1 (M2..M4)
0 M2 72 M2..M3 84 M2(M3..M4) 126 (M2..M4)M5
0 M3 48 M3..M4 132 (M3..M4) M5
0 M4 56 M4..M5
0 M5
25
MatChain Algorithm

• Algorithm MatChain
• Input r1..n1 of ve integers corresponding to
  the dimensions of a chain of matrices
• Output Least number of scalar multiplications
  required to multiply the n matrices
• for i 1 to n do
• Ci,i 0 // diagonal d0
• for d 1 to n-1 do // for diagonals d1 to dn-1
• for i 1 to n-d do
• j id
• Ci,j ?
• for k i1 to j do
• Ci,j minCi,j,Ci,k-1Ck,jrir
  krj1
• end for
• end for
• return C1,n

26
Time and Space Complexity of MatChain Algorithm

• Time Complexity
• Space Complexity

27
All-Pairs Shortest Paths

• Problem For every vertex u, v ? V, calculate
  ?(u, v).
• Possible Solution 1
• Cost of Possible Solution 1

28
Dynamic Programming Solution

• Define a k-path from u to v, where
• u,v ? 1 , 2 , , n
• to be any path whose intermediate vertices all
  have indices less than or equal to k.
• What is a 0-path?
• What is a 1-path?
• What is an n-path?

29
Floyds Algorithm

• Algorithm Floyd
• Input An n ? n matrix length1..n, 1..n such
  that lengthi,j is the weight of the edge (i,j)
  in a directed graph G (1,2,,n, E)
• Output A matrix D with Di,j ?i,j
• 1 D length //copy the input matrix length into
  D
• 2 for k 1 to n do
• 3 for i 1 to n do
• 4 for j 1 to n do
• 5 Di,j minDi,j , Di,k Dk,j

30
Example
2
11
5
4
12
15
8
2
4
1
11
1
3
5
31
Example (Cont.)
0-p 1 2 3 4
1
2
3
4
1-p 1 2 3 4
1
2
3
4
2-p 1 2 3 4
1
2
3
4
3-p 1 2 3 4
1
2
3
4
32
Example (Cont.)
0-p 1 2 3 4
1 0 12 5 ?
2 15 0 8 11
3 ? 4 0 2
4 1 5 11 0
1-p 1 2 3 4
1 0 12 5 ?
2 15 0 8 11
3 ? 4 0 2
4 1 5 6 0
2-p 1 2 3 4
1 0 12 5 23
2 15 0 8 11
3 19 4 0 2
4 1 5 6 0
3-p 1 2 3 4
1 0 9 5 7
2 15 0 8 10
3 19 4 0 2
4 1 5 6 0
33
Example (Cont.)
4-p 1 2 3 4
1 0 9 5 7
2 11 0 8 10
3 3 4 0 2
4 1 5 6 0
3-p 1 2 3 4
1 0 9 5 7
2 15 0 8 10
3 19 4 0 2
4 1 5 6 0
34
Time and Space Complexity

• Time Complexity
• Space Complexity

35
The Knapsack Problem

• Let U u1, u2, , un be a set of n items to be
  packed in a knapsack of size C??.
• Let sj and vj be the size and value of the jth
  item, where sj, vj ??, 1 ? j ? n.
• The objective is to fill the knapsack with some
  items from U whose total size does not exceed C
  and whose total value is maximum.
• Assume that the size of each item does not exceed
  C.

36
The Knapsack Problem Formulation

• Given n ve integers in U, we want to find a
  subset S?U s.t.
• is maximized subject to the constraint

37
Inductive Solution

• Let Vi,j denote the value obtained by filling a
  knapsack of size j with items taken from the
  first i items u1, u2, , ui in an optimal way
• The range of i is
• The range of j is
• The objective is to find V ,
• Vi,0 V0,j
• Vi,j Vi-1,j
  if
• max Vi-1,j, V ,
  vi if

38
Example (pp. 223 Question 7.22)

• There are five items of sizes 3, 5, 7, 8, and 9
  with values 4, 6, 7, 9, and 10 respectively. The
  size of the knapsack is 22.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
0 0 0
0 0 0
0 0 0
0 0 0
39
Example (pp. 223 Question 7.22)

• There are five items of sizes 3, 5, 7, 8, and 9
  with values 4, 6, 7, 9, and 10 respectively. The
  size of the knapsack is 22.

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
0 0 0 4 4 6 6 6 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
0 0 0 4 4 6 6 7 10 10 11 11 13 13 13 17 17 17 17 17 17 17 17
0 0 0 4 4 6 6 7 10 10 11 13 13 15 15 17 19 19 20 20 22 22 22
0 0 0 4 4 6 6 7 10 10 11 13 14 15 16 17 19 20 20 21 23 23 25
40
Algorithm Knapsack

• Algorithm Knapsack
• Input A set of items U u1,u2,,un with sizes
  s1,s2,,sn and values v1,v2,,vn, respectively
  and knapsack capacity C.
• Output the maximum value of subject to
• for i 0 to n do
• Vi,0 0
• for j 0 to C do
• V0,j 0
• for i 1 to n do
• for j 1 to C do
• Vi,j Vi-1,j
• if si ? j then
• Vi,j maxVi,j, Vi-1,j-sivi
• end for
• end for
• return Vn,C

41
Time and Space Complexity of the Knapsack
Algorithm