COMP108 Algorithmic Foundations Divide and Conquer

1
COMP108Algorithmic FoundationsDivide and
Conquer
Prudence Wong http//www.csc.liv.ac.uk/pwong/tea
ching/comp108
2
Divide and Conquer
3
Learning outcomes

• Understand how divide and conquer works and able
  to analyze complexity of divide and conquer
  methods by solving recurrence
• See examples of divide and conquer methods

4
Learning outcomes

• Understand how divide and conquer works and able
  to analyze complexity of divide and conquer
  methods by solving recurrence
• See examples of divide and conquer methods

5
Divide and Conquer

• One of the best-known algorithm design
  techniques.
• Idea
• A problem instance is divided into several
  smaller instances of the same problem, ideally of
  about same size
• The smaller instances are solved, typically
  recursively
• The solutions for the smaller instances are
  combined to get a solution to the original problem

6
Binary Search

• Recall that we have learnt binary search
• Input a sequence of n sorted numbers a0, a1, ,
  an-1 and a number X
• Idea of algorithm
• compare X with number in the middle
• then focus on only the first half or the second
  half (depend on whether X is smaller or greater
  than the middle number)
• reduce the amount of numbers to be searched by
  half

7
Binary Search (2)
we first work on n numbers, from a0..an-1

• 3 7 11 12 15 19 24 33 41 55 24
• 19 24 33 41 55 24
• 19 24 24

then we work on n/2 numbers,from an/2..an-1
further reduce by half
8
Recursive Binary Search

• RecurBinarySearch(A, first, last, X)
• begin
• if (first gt last) then
• return false
• mid ?(first last)/2?
• if (X Amid) then
• return true
• if (X lt Amid) then
• return RecurBinarySearch(A, first, mid-1, X)
• else
• return RecurBinarySearch(A, mid1, last, X)
• end

invoke by calling RecurBinarySearch(A, 0, n-1,
X) return true if X is found, false otherwise
9
Recursive Binary Search (RBS)
To find 24

• 3 7 11 12 15 19 24 33 41 55
• RBS(A, 0, 9, 24)
• if 0 gt 9? No mid 4 if 24 A4? No if 24 lt
  A4? No
• RBS(A, 5, 9, 24)
• if 5 gt 9? No mid 7 if 24 A7? No if 24 lt
  A7? Yes
• RBS(A, 5, 6, 24)
• if 5 gt 6? No mid 5 if 24 A5? No if 24 lt
  A5? No
• RBS(A, 6, 6, 24)
• if 6 gt 6? No mid 6 if 24 A6? YES return
  true
• RBS(A, 6, 6, 24) is done, return true
• RBS(A, 5, 6, 24) is done, return true
• RBS(A, 5, 9, 24) is done, return true
• RBS(A, 0, 9, 24) is done, return true

10
Recursive Binary Search (RBS)
To find 23

• 3 7 11 12 15 19 24 33 41 55
• RBS(A, 0, 9, 23)
• if 0 gt 9? No mid 4 if 23 A4? No if 23 lt
  A4? No
• RBS(A, 5, 9, 23)
• if 5 gt 9? No mid 7 if 23 A7? No if 23 lt
  A7? Yes
• RBS(A, 5, 6, 23)
• if 5 gt 6? No mid 5 if 23 A5? No if 23 lt
  A5? No
• RBS(A, 6, 6, 23)
• if 6 gt 6? No mid 6 if 23 A6? No if 23 lt
  A5? No
• RBS(A, 7, 6, 23) if 7 gt 6? Yes return false
• RBS(A, 7, 6, 23) is done, return false
• RBS(A, 6, 6, 23) is done, return false
• RBS(A, 5, 6, 23) is done, return false
• RBS(A, 5, 9, 23) is done, return false
• RBS(A, 0, 9, 23) is done, return false

11
Time complexity

• Let T(n) denote the time complexity of binary
  search algorithm on n numbers.
• We call this formula a recurrence.

1 if n1 T(n/2) 1 otherwise
T(n)
12
Recurrence

• A recurrence is an equation or inequality that
  describes a function in terms of its value on
  smaller inputs.
• E.g.,

To solve a recurrence is to derive asymptotic
bounds on the solution
13
Substitution method
1 if n1 T(n/2) 1 otherwise
T(n)

• Make a guess, T(n) ? 2 log n
• We prove statement by MI.

Base case? When n1, statement is FALSE! L.H.S
T(1) 1 R.H.S c log 1 0 lt L.H.S Yet, when
n2, L.H.S T(2) T(1)1 2 R.H.S 2 log 2
2 L.H.S ? R.H.S
14
Substitution method
1 if n1 T(n/2) 1 otherwise
T(n)

• Make a guess, T(n) ? 2 log n
• We prove statement by MI.
• Assume true for all n' lt n assume T(n/2) ? 2
  log (n/2)
• T(n) T(n/2) 1
• ? 2 log (n/2) 1 ? by hypothesis
• 2(log n 1) 1 ? log(n/2) log n log 2
• lt 2log n

i.e., T(n) ? 2 log n
15
Example
1 if n1 2T(n/2) 1 otherwise

• Prove that is O(n)
• Guess T(n) ? 2n 1
• Assume true for all n' lt n assume T(n/2) ?
  2(n/2)-1
• T(n)
• ? 2 (2(n/2)-1) 1
• 2n 2 1
• 2n-1

T(n)
For the base case when n1,L.H.S T(1)
1 R.H.S 2-1 1 L.H.S ? R.H.S
i.e., T(n) ? 2n-1
16
More Example
1 if n1 2T(n/2) n otherwise

• Prove that is O(n log n)
• Guess T(n) ? 2 n log n
• Assume true for all n'ltn assume T(n/2) ? 2
  (n/2) log(n/2)
• T(n)
• ? 2 (2 (n/2) log (n/2) ) n
• 2 n (log n - 1) n
• 2 n log n - 2n n
• ? 2 n log n

T(n)
For the base case when n2,L.H.S T(2)
2T(1)2 4,R.H.S 2 2 log 2 4L.H.S ? R.H.S
i.e., T(n) ? 2 n log n
17
Summary

• Depending on the recurrence, we can guess the
  order of magnitude
• T(n) T(n/2)1 T(n) is O(log n)
• T(n) 2T(n/2)1 T(n) is O(n)
• T(n) 2T(n/2)n T(n) is O(n log n)

18
Learning outcomes

• Understand how divide and conquer works and able
  to analyze complexity of divide and conquer
  methods by solving recurrence
• See examples of divide and conquer methods

19
Merge Sort
20
Merge sort

• using divide and conquer technique
• divide the sequence of n numbers into two halves
• recursively sort the two halves
• merge the two sort halves into a single sorted
  sequence

21
51, 13, 10, 64, 34, 5, 32, 21
we want to sort these 8 numbers, divide them into
two halves
22
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
divide these 4 numbers into halves
similarly for these 4
23
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
further divide each shorter sequence until we
get sequence with only 1 number
24
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
51
13
10
64
34
5
32
21
merge pairs of single number into a sequence of 2
sorted numbers
25
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
51
13
10
64
34
5
32
21
13, 51
10, 64
5, 34
21, 32
then merge again into sequences of 4 sorted
numbers
26
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
51
13
10
64
34
5
32
21
13, 51
10, 64
5, 34
21, 32
10, 13, 51, 64
5, 21, 32, 34
one more merge give the final sorted sequence
27
51, 13, 10, 64, 34, 5, 32, 21
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
51
13
10
64
34
5
32
21
13, 51
10, 64
5, 34
21, 32
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13, 21, 32, 34, 51, 64
28
Summary

• Divide
• dividing a sequence of n numbers into two smaller
  sequences is straightforward
• Conquer
• merging two sorted sequences of total length n
  can also be done easily, at most n-1 comparisons

29
10, 13, 51, 64
5, 21, 32, 34
Result
To merge two sorted sequences, we keep two
pointers, one to each sequence
Compare the two numbers pointed, copy the smaller
one to the result and advance the corresponding
pointer
30
10, 13, 51, 64
5, 21, 32, 34
5,
Result
Then compare again the two numbers pointed to by
the pointer copy the smaller one to the
result and advance that pointer
31
10, 13, 51, 64
5, 21, 32, 34
5, 10,
Result
Repeat the same process
32
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13
Result
Again
33
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13, 21
Result
and again
34
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13, 21, 32
Result

35
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13, 21, 32, 34
Result
When we reach the end of one sequence, simply
copy the remaining numbers in the other sequence
to the result
36
10, 13, 51, 64
5, 21, 32, 34
5, 10, 13, 21, 32, 34, 51, 64
Result
Then we obtain the final sorted sequence
37
Pseudo code

• Algorithm Mergesort(A0..n-1)
• if n gt 1 then begin
• copy A0..?n/2?-1 to B0..?n/2?-1
• copy A?n/2?..n-1 to C0..?n/2?-1
• Mergesort(B0..?n/2?-1)
• Mergesort(C0..?n/2?-1)
• Merge(B, C, A)
• end

Algorithm Merge(B0..p-1, C0..q-1,
A0..pq-1) Set i0, j0, k0 while iltp and
jltq do begin if Bi?Cj then set AkBi
and increase i else set Ak Cj and
increase j k k1 end if ip then copy
Cj..q-1 to Ak..pq-1 else copy Bi..p-1 to
Ak..pq-1
38
Pseudo code

• Algorithm Mergesort(A0..n-1)
• if n gt 1 then begin
• copy A0..?n/2?-1 to B0..?n/2?-1
• copy A?n/2?..n-1 to C0..?n/2?-1
• Mergesort(B0..?n/2?-1)
• Mergesort(C0..?n/2?-1)
• Merge(B, C, A)
• end

39
MS(
)
51, 13, 10, 64, 34, 5, 32, 21
MS(
MS(
)
)
51, 13, 10, 64
34, 5, 32, 21
51, 13
10, 64
34, 5
32, 21
MS(
)
MS(
)
MS(
)
MS(
)
51
13
10
64
34
5
32
21
MS(
MS(
)
MS(
MS(
MS(
MS(
)
MS(
MS(
)
)
)
)
)
)
13, 51
10, 64
5, 34
21, 32
M(
)
,
M(
)
,
)
10, 13, 51, 64
5, 21, 32, 34
M(
,
5, 10, 13, 21, 32, 34, 51, 64
40
MS(
)
51, 13, 10, 64, 34, 5, 32, 21
1
11
)
MS(
MS(
)
51, 13, 10, 64
34, 5, 32, 21
6
2
16
12
51, 13
10, 64
34, 5
32, 21
MS(
)
MS(
)
MS(
)
MS(
)
3
4
7
8
13
14
17
18
51
13
10
64
34
5
32
21
MS(
MS(
)
MS(
MS(
MS(
MS(
)
MS(
MS(
)
)
)
)
)
)
15
19
5
9
13, 51
10, 64
5, 34
21, 32
M(
)
,
M(
)
,
10
20
)
10, 13, 51, 64
5, 21, 32, 34
M(
,
21
5, 10, 13, 21, 32, 34, 51, 64
41
Pseudo code
Algorithm Merge(B0..p-1, C0..q-1,
A0..pq-1) Set i0, j0, k0 while iltp and
jltq do begin if Bi?Cj then set AkBi
and increase i else set Ak Cj and
increase j k k1 end if ip then copy
Cj..q-1 to Ak..pq-1 else copy Bi..p-1 to
Ak..pq-1
42
p4
q4
10, 13, 51, 64
5, 21, 32, 34
B
C
i j k A
Before loop 0 0 0 empty
End of 1st iteration 0 1 1 5
End of 2nd iteration 1 1 2 5, 10
End of 3rd 2 1 3 5, 10, 13
End of 4th 2 2 4 5, 10, 13, 21
End of 5th 2 3 5 5, 10, 13, 21, 32
End of 6th 2 4 6 5, 10, 13, 21, 32, 34
5, 10, 13, 21, 32, 34, 51, 64
43
Time complexity
Let T(n) denote the time complexity of running
merge sort on n numbers.
1 if n1 2T(n/2) n otherwise
T(n)
See Slide 16, T(n) O(n log n)
44
Tower of Hanoi
45
Tower of Hanoi - Initial config

• There are three pegs and some discs of different
  sizes are on Peg A

1
2
3
A
B
C
46
Tower of Hanoi - Final config

• Want to move the discs to Peg C

1
2
3
A
B
C
47
Tower of Hanoi - Rules

• Only 1 disk can be moved at a time
• A disc cannot be placed on top of other discs
  that are smaller than it

Target Use the smallest number of moves
48
Tower of Hanoi - One disc only

• Easy!

1
A
B
C
49
Tower of Hanoi - One disc only

• Easy! Need one move only.

1
A
B
C
50
Tower of Hanoi - Two discs

• We first need to move Disc-2 to C, How?

by moving Disc-1 to B first, then Disc-2 to C
1
2
A
B
C
51
Tower of Hanoi - Two discs

• Next?

Move Disc-1 to C
1
2
A
B
C
52
Tower of Hanoi - Two discs

• Done!

1
2
A
B
C
53
Tower of Hanoi - Three discs

• We first need to move Disc-3 to C, How?
• Move Disc-12 to B (recursively)

1
2
3
A
B
C
54
Tower of Hanoi - Three discs

• Then move Disc-3 to C

• We first need to move Disc-3 to C, How?
• Move Disc-12 to B (recursively)

3
1
2
A
B
C
55
Tower of Hanoi - Three discs

• Only task left move Disc-12 to C (similarly as
  before)

1
3
2
A
B
C
56
Tower of Hanoi - Three discs

• Only task left move Disc-12 to C (similarly as
  before)

3
2
1
A
B
C
57
Tower of Hanoi - Three discs

• Done!

1
2
3
A
B
C
58
Tower of Hanoi

• ToH(num_disc, source, dest, spare)
• begin
• if (num_disc gt 1) then
• ToH(num_disc-1, source, spare, dest)
• Move the disc from source to dest
• if (num_disc gt 1) then
• ToH(num_disc-1, spare, dest, source)
• end

invoke by calling ToH(3, A, C, B)
59
ToH(3, A, C, B)
move 1 disc from A to C
ToH(2, A, B, C)
ToH(2, B, C, A)
ToH(1, A, C, B)
ToH(1, C, B, A)
ToH(1, B, A, C)
ToH(1, A, C, B)
move 1 disc from A to B
move 1 disc from B to C
move 1 disc from A to C
move 1 disc from C to B
move 1 disc from B to A
move 1 disc from A to C
60
ToH(3, A, C, B)
1
8
7
move 1 disc from A to C
ToH(2, A, B, C)
ToH(2, B, C, A)
9
2
5
12
11
4
ToH(1, A, C, B)
ToH(1, C, B, A)
ToH(1, B, A, C)
ToH(1, A, C, B)
move 1 disc from A to B
move 1 disc from B to C
3
13
6
10
move 1 disc from A to C
move 1 disc from C to B
move 1 disc from B to A
move 1 disc from A to C
from A to C from A to B from C to B from A to
C from B to A from B to C from A to C
61
Time complexity
Let T(n) denote the time complexity of running
the Tower of Hanoi algorithm on n discs.

• T(n) T(n-1) 1 T(n-1)

move n-1 discs from B to C
move n-1 discs from A to B
move Disc-n from A to C
1 if n1 2T(n-1) 1 otherwise
T(n)
62
Time complexity (2)

• T(n) 2T(n-1) 1
• 22T(n-2) 1 1
• 22 T(n-2) 2 1
• 22 2T(n-3) 1 21 20
• 23 T(n-3) 22 21 20
• 2n-1 T(1) 2n-2 22 21 20
• 2n-1 2n-2 22 21 20
• 2n-1

1 if n1 2T(n-1) 1 otherwise
T(n)
In Tutorial 2, we prove by MI that 20 21
2n-1 2n-1
63
Summary - continued

• Depending on the recurrence, we can guess the
  order of magnitude
• T(n) T(n/2)1 T(n) is O(log n)
• T(n) 2T(n/2)1 T(n) is O(n)
• T(n) 2T(n/2)n T(n) is O(n log n)
• T(n) 2T(n-1)1 T(n) is O(2n)

64
Iterative Method
65
Iterative method (to solve recurrence)

• The above method of solving a recurrence is
  called the iterative method.
• The method is to expand (iterate) the recurrence
  until we arrive the base condition.

66
Iterative method - Example 1

• T(n) T(n/2) 1
• T(n/4) 1 1
• T(n/22) 2
• T(n/23) 1 2
• T(n/23) 3
• T(n/2log n) log n
• T(1) log n
• 1 log n

1 if n1 T(n/2) 1 otherwise
T(n)
Hence, T(n) is O(log n)
67
Iterative method - Example 2

• T(n) 2T(n/2) 1
• 22T(n/4) 1 1
• 22 T(n/22) 2 1
• 22 2T(n/23) 1 2 1
• 23 T(n/23) 22 2 1
• 2log n T(n/2log n) 2logn-1 2logn-2
  22 2 1
• n T(1) n/2 n/4 ... 4 2 1
• n n/2 n/4 ... 4 2 1 2n-1

1 if n1 2T(n/2) 1 otherwise
T(n)
Hence, T(n) is O(n)
68
Iterative method - Example 3
1 if n1 2T(n/2) n otherwise
T(n)

• T(n) 2T(n/2) n
• 22T(n/4) n/2 n
• 22 T(n/22) n n 22 T(n/22) 2n
• 22 2T(n/23) n/22 2n
• 23 T(n/23) 3n
• 2i T(n/2i) in
• 2log n T(n/2log n) n log n
• n T(1) n log n n n log n

Hence, T(n) is O(n log n)
69
Exercise on iterative method
T(n) 2T(n-1) 4 22T(n-2) 4 4 22
T(n-2) 2x4 4 22 2T(n-3) 4 21x4
20x4 23 T(n-3) 22x4 21x4 20x4 2n-1
T(1) 2n-2x4 22x4 21x4 20x4 2n-1x4
2n-2x4 22x4 21x4 20x4 4x(2n-1)
4 if n1 2T(n-1) 4 if n gt 1
T(n)
Use 20 21 2n-1 2n-1
70
Fibonacci number
71
Fibonacci number

• Fibonacci number F(n)

1 if n 0 or 1 F(n-1) F(n-2) if n gt 1
F(n)
n 0 1 2 3 4 5 6 7 8 9 10
F(n) 1 1 2 3 5 8 13 21 34 55 89
Pseudo code for the recursive algorithm Algorithm
F(n) if n0 or n1 then return
1 else return F(n-1) F(n-2)
72
The execution of F(7)
F7
F6
F5
F5
F4
F4
F3
F2
F3
F3
F2
F2
F1
F0
F1
F1
F0
F1
F0
F4
F3
F2
F2
F1
F1
F1
F0
F1
F0
F2
F3
F2
F1
F0
F1
F1
F0
F2
F1
F1
F0
73
The execution of F(7)
1
F7
2
F6
F5
3
F5
F4
F4
F3
F2
F3
F3
F2
F2
13
F1
4
F0
F1
F1
F0
F1
F0
F4
F3
F2
F2
10
F1
F1
5
F1
F0
F1
F0
F2
F3
F2
F1
6
F0
F1
F1
F0
F2
F1
9
7
F1
F0
8
74
Learning outcomes

• Understand how divide and conquer works and able
  to analyze complexity of divide and conquer
  methods by solving recurrence
• See examples of divide and conquer methods