Title: Lectures on Recursive Algorithms
1Lectures onRecursive Algorithms
- COMP 523 Advanced Algorithmic Techniques
- Lecturer Dariusz Kowalski
2Announcements
- Web page
- www.csc.liv.ac.uk/darek/comp523.html
- Slides constantly updated based on your feedback
check for a new version a few hours after the
lecture(s)
3Overview
- Previous lectures
- Algorithms correctness, termination, performance
- Asymptotic notation
- Popular asymptotic complexities log n, n, n log
n, n2, ... - Graphs - basic definitions and examples
- These lectures
- Recursive algorithms - basic recursion
- Searching algorithm in time O(log n)
- Finding majority in time O(n)
- Sorting in time O(n log n)
- Other examples
4Algorithms based on recurrence
- Algorithmic scheme
- Reduce the input to smaller sub-input(s)
- Solve the problem for (some) sub-inputs
- Merge the obtained solution(s) into one global
solution
5Complexity analysis of recursive process
- Let T(m) denote time (or other complexity
measure) of solving a given problem by a given
algorithm working on input length m - Simple recursive formula
- T(n) ? qT(n/2) f(n)
- Solve the problem for half of the input q times,
and spend at most f(n) time for partitioning
and/or merging
6Example 1Searching in time O(log n)
7Searching problem and algorithm
- Input sorted array A of n numbers and number x
- Problem find if x is in array A
- Solution Algorithm SEARCH(x,1,n)
- Procedure SEARCH(x,i,j)
- If i j then
- if x Ai then answer YES
- if x ? Ai then answer NO
- Else compare x with element A(ij)/2
- If x A(ij)/2 then answer YES
- If x lt A(ij)/2 then SEARCH(x, i, (ij)/2-1)
- If x gt A(ij)/2 then SEARCH(x, (ij)/21, j)
8Tree structure of searching
root
4
find if value 5 is there
6
2
height 3
1
3
5
7
1
2
3
4
5
6
7
8
8 leaves
9Time and memory consumption
- Each recursive call
- needs additional constant memory, and
- the size of input decreases by factor 2
- Formula for time ( comparisons)
- T(n) ? T(n/2) 4
- Additional memory size log n O(log n)
- In other words recursive algorithm must store a
path from the root to the searched leaf, and it
has logarithmic length
10Case q 1, f(n) c
- T(n) ? T(n/2) c , T(2) ? c
-
- T(n) ? T(n/2) c
- ? (T((n/2)/2) c) c T(n/4) 2c
- ? (T((n/4)/2) c) 2c T(n/8) 3c
-
- ? (T(n/2i) c) (i-1) . c T(n/2i) i . c
-
- ? (T(n/2log n 1) c) (log n - 2) . c
- T(n/(n/2)) (log n - 2) . c T(2) (log n
1) . c - ? c (log n 1) . c c log n
11Case q 1, f(n) cformal analysis
- T(n) ? T(n/2) c
- T(2) ? c
- Solution T(n) ? c log n
- Proof by induction.
- For n 2 straightforward T(2) ? c ? c log 2
- Suppose T(n/2) ? c log (n/2) then
- T(n) ? T(n/2) c ? c log (n/2) c
- ? c log n - c c ? c log n
12Example 2Finding a majority in time O(n)
13Finding a majority
- Input array A of n elements
- Problem find if there is an element x that
occurs in array A more than n/2 times (a majority
value) - Solution Algorithm Majority(A) returning a
majority element in A, if it exists, or null
otherwise
14Finding a majority algorithm
- Algorithm Majority(A1..n)
- If A 0 then output null, else if A 1 then
output A1 else - If n A is odd then
- check whether An is a majority in A by counting
the number of occurrences of value An - if yes then output An, otherwise decrease n by
one - Initialize additional array B of size A/2
- Set j to 0
- For i 1,2,,n/2 do
- if A2i-1 A2i then
- increase j by one
- set Bj to A2i
- Find if there is a majority in B1..j by
executing Majority(B1..j) - If a majority value x in B1..j is returned then
check whether x is a majority in A, by going
through array A and counting the number of
occurrences of value x in A if successful output
x otherwise null
15Tree structure of finding majority
root
2
odd, no majority
height 2
2
2
1
2
2
2
2
1
2
1
1
8 leaves
16Correctness of Majority Algorithm
- Lemma (to be proved later)
- If x is a majority in A then x is a majority in
B. - Correctness (proof by induction, based on the
Lemma) - Inductive assumption
- Majority(B) works correctly for all Bs of size
smaller than A - Case 1 There is a majority in A.
- Then it is a majority in B, by Lemma, therefore
it will be found in the recursive stage of the
algorithm Majority(B) (by inductive argument),
and will be checked (successfully) in the last
point of the algorithm Majority(A). - Case 2 There is no majority in A.
- Then even if there is a candidate value found in
the recursive stage Majority(B) of the algorithm,
it will be rejected anyway in the last point of
the algorithm Majority(A).
17Proof of the Lemma
- Suppose, to the contrary, that x is a majority in
A but is not a majority in B. Let - m be the number of occurrences of x in A, and
- k be the number of occurrences of x in B.
- It follows that the other values appear at least
k times in B. - Consequently, the other values appear in A
- at least 2k times those pairs that are
represented in B by a value different than x
plus - m-2k times each occurrence of x in A that is
not paired by another x (there are k pairs xx in
A, thus m-2k of other occurrences of x in A) is
paired by some other value (different than x) - which gives at least 2k(m-2k) m occurrences in
total, and contradicts the fact that x is a
majority in A. Contradiction! Therefore x must be
also a majority in B.
18Memory consumption
- Each call to the recursive procedure
- needs additional memory of half of the current
input size - reduces the size of the input by factor at least
two - Formula for time ( comparisons)
- T(n) ? n -1 3n/2 T(n/2) n T(n/2) 7n/2
- Additional memory used (for smaller arrays B)
- M(n) ? n/2 M(n/2) ? n/2 n/4 M(n/4)
- ? n/2 n/4 1 n - 1 O(n)
- Intuition about memory size the algorithm needs
a binary tree of linear size, in the worst case,
to store winners
19Case q 1, f(n) cn
- T(n) ? T(n/2) cn , T(1) ? c
-
- T(n) ? T(n/2) cn
- ? (T(n/4) cn/2) cn T(n/4) (3/2)cn
- ? (T(n/8) cn/4) (3/2)cn T(n/8)
(7/4)cn -
- ? (T(n/2i) cn/2i-1) ((2i-1-1)/2i-2) . cn
- T(n/2i) ((2i-1)/2i-1) . cn
-
- ? T(n/2log n) ((2log n-1)/2log n-1) . cn
- T(1) (n 1)/(n/2) . cn ? c 2cn 2c
? 2cn
20Case q 1, f(n) c nformal analysis
- T(n) ? T(n/2) c n
- T(1) ? c
- Solution T(n) ? 2c?n
- Proof by induction.
- For n 1 straightforward T(1) ? c ? 2c ? 1
- Suppose T(n/2) ? 2c?(n/2) then
- T(n) ? T(n/2) c?n ? 2c?(n/2) c n
- 2c?n
21Textbook and exercises
- READING Chapter 5, up to Section 5.2
- EXERCISES
- Solve the following recursive formulas the best
you can - T(n) ? T(n/2) 4
- T(n) ? T(n/2) 5n
- T(n) ? T(n/2) 3n2
- T(n) ? (3/2)?T(n/2) 1
- Compare and sort the obtained (asymptotic)
formulas according to the asymptotic order. For
each of them try to find an algorithmic example
with performance giving by this formula. - Sort the following formulas according to
big/small Oh order - log (n1/2), log (9n), log (n3), 2log n, 23log n,
2log (9n), n2, n log n - Propose and analyse an algorithm for checking if
there is a majority in a given sorted array A.
22Example 3Sorting in time O(n log n)
23Divide and conquer
- Algorithm
- Partition input into two halves in (at most)
linear time - Solve the problem for each half of the input
separately - Merge solutions into one in (at most) linear time
- Recursive formula on time complexity
- T(n) ? 2 T(n/2) c?n
24Case q 2, f(n) cn
- T(n) ? 2T(n/2) cn , T(2) ? c
-
- T(n) ? 2T(n/2) cn
- ? 2(2T(n/4) cn/2) cn 4T(n/4) 2cn
- ? 4(2T(n/8) cn/4) 2cn 8T(n/8) 3cn
-
- ? 2i-1 (2T(n/2i) cn/2i-1) (i-1) . cn
- 2i T(n/2i) i . cn
-
- ? 2log n - 1 T(2) (log n - 1) . cn ? cn log
n
25Solution for timeformal analysis
- T(n) ? 2 T(n/2) c?n
- T(2) ? c
- Solution T(n) ? c?n?log n
- Proof by induction.
- For n 2 straightforward T(2) ? c ? c?2?log
2 - Suppose T(n/2) ? c?(n/2)?log (n/2) then
- T(n) ? 2T(n/2) c?n ? 2c?(n/2)?log (n/2)
c?n - 2c?(n/2)?log n 2c?n/2 c?n c?n?log n
26Example - sorting by merging
- Input list L of n numbers
- Problem Sort list L
- Algorithm MergeSort(L)
- If L has at most two elements then sort them by
comparison and exit - Else
- Split L into two lists prefix and suffix, each of
size n/2 - Sort, by merging, the prefix and the suffix
separately - MergeSort(prefix) and MergeSort(suffix)
- Merge sorted prefix with sorted suffix as
follows - Initialize final list as empty
- Repeat until either prefix or suffix is empty
- Compare the first elements on the lists prefix
and suffix and then move the smaller one to the
end of the final list - Concatenate final list with the remaining
non-empty list (prefix or suffix)
27MergeSort procedure
1
2
4
7
8
3
5
6
9
10
1
2
3
4
5
6
7
8
9
10
28Tree structure of sorting
root
merging operations
height 3
2
4
1
7
8
3
6
5
8 leaves
29Time and memory consumption
- Each recursive call
- needs additional constant memory, and
- the number of inputs increases by factor 2, and
- the size of each input decreases by factor 2
- Formula for time ( comparisons and list
operations) - T(n) ? n/2 2T(n/2) 2n 2T(n/2) 5n/2
- Additional memory size M(n) ? 2 M(n/2) 2
- M(n) ? 2 M(n/2) 2 ? 4 M(n/4) 4 2 ? 8 M(n/8)
8 4 2 - ? ? n/2 n/4 4 2 n - 2 O(n)
- In other words divide and conquer algorithm must
store the binary tree of pointers during the
computation
30Quick sort - algorithmic scheme
- Generic Quick Sort
- Select one element x from the input
- Partition the input into part containing elements
not greater than x and part containing elements
bigger than x - Sort each part separately
- Concatenate these two sorted parts
- Problem how to choose element x to balance the
sizes of these two parts? (to get the similar
recursive equations as for MergeSort)
31Why parts should be balanced?
- Suppose they are not balanced, but, say, the
smallest element is chosen - T(n) ? T(1) T(n-1) c n
- T(1) ? c
- Solution T(n) ? (c/2) . n2
- Proof by induction.
- For n 1 straightforward T(1) ? c ? (c/2) .
12 - Suppose T(n-1) ? (c/2) . (n-1)2 then
- T(n) ? T(n-1) c cn ? (c/2) . (n-1)2 c
(n1) - ? (c/2) . (n-1)2 (c/2) . 2n ? (c/2) . n2
32Two approaches to be fast
- Deterministic
- Additional tree structure of size O(n) needed for
identifying the median value - Time O(n log n), additional memory O(n)
- Randomized
- Select separation element x uniformly at random
- Time (expected) O(n log n), additional memory
O(n)
33Randomized approach - analysis
- Let T(n) denote the expected time sum of all
possible values of time weighted by the
probabilities of these values - T(n) ? 1/n (T(n-1)T(1) T(n-2)T(2)
T(0)T(n)) cn - T(0) T(1) 1, T(2) ? c
- Solution T(n) ? d n log n, for some constant d
? 8c - Proof by induction.
- For n 2 straightforward T(2) ? c ? d . 2
. log 2 - Suppose T(m) ? d m log m, for every m lt n then
- (1-1/n)?T(n) ? (2/n)?(T(0) T(n-1)) c n
- ? (2d/n)?(1 1 2log 2
(n-1)log(n-1)) c n - ? (2d/n)?((n2/2) log n n2/8) c n
- ? d n log n - d (n/4) c n ? d n
log n - d n/2 - T(n) ? (n/(n-1))?(d n log n - d n/2) ? d n log n
34Tree structure of random execution
root
1
6
5
7
height 5
3
2
4
1
2
3
4
5
6
7
8
8 leaves
35Textbook and Exercises
- READING Section 5.3
- EXERCISES
- Solved exercises 1, 2 from the textbook, chapter
5 Divide and Conquer - Prove that
- 1 1 2 log 2 (n-1)log(n-1) ? n(n/2)
log(4n) , and - (n/(n-1))?(d n log n - d n/2) ? d n log n , for n
gt 16
36Example 4Finding closest pair in time O(n log
n)
37Problem
- Closest pair of points
- Input n points on the plane
- Output two points having the closest distance
- Remarks
- Exhaustive search algorithm gives time O(n2)
- Similar approach gives ALL pairs of closest points
38Solution
- Preprocessing
- Sort points according to the first coordinate
(obtain horizontal list H) and according to the
second coordinate (obtain vertical list V) - Main Algorithm
- Partition input list H into halves (H1,H2) in
linear time (draw vertical line L containing
medium point) split list V accordingly into
V1,V2, where Vi contains the same elements as Hi
and inherits the initial order from V (for i
1,2) - Solve the problem for each half of the input
separately, obtaining two pairs of closest
points let ? be the smallest distance from the
obtained ones - Find if there is a pair which has distance
smaller than ? - if yes then find the smallest
distance pair
39Main difficulty
- Find if there is a pair which has distance
smaller than ? - if yes then find the smallest distance pair
- How to do it in linear time?
?
?
?
L
40Closest pair in the strip
- Find a sub-list P of V containing all points with
a distance at most ? from the line L - go through V and remove elements not ?close to L
- 2. For each element in P check its distance to
the next 8 elements and remember the closest pair
ever found
?
?
?
P
L
41Why to check only 8 points ahead?
- 1. Partition the strip into squares of length ?/2
each, as shown in the picture - 2. Each square contains at most 1 point - by
definition of ? - 3. If there is at least 2 full squares between
points then they can not be the closest points - 4. There are at most 8 squares to check
?
?/2
?/2
?/2
?/2
?/2
?/2
L
42Time complexity
- Preprocessing sorting in time O(n log n)
- Main Algorithm recursion in time O(n log n)
- T(n) ? 3n/2 2T(n/2) 8n 2T(n/2) 9.5n
- T(2) 1
43Example 5Integer multiplication in time
O(n1.59)
44Beyond ?(n log n) integers multiplication
- Input two integers x and y, each consisting of
at most n bits - Output multiply these integers
- Naïve Algorithm
- Multiply each bit i of x by y, add (i-1) zeroes
at the end - Add the obtained at most n numbers
- Time ?(n2) of bit operations
45Divide and Conquer approach
- Let x x1 2n/2 x2 and y y1 2n/2 y2
- Let p x1 x2 and q y1 y2. Then
- x y (x1 2n/2 x2) (y1 2n/2 y2)
- x1 y1 2n/2 (x2 y1 x1 y2) 2n x2 y2
- x1 y1 2n/2 (pq - x1 y1- x2 y2) 2n
x2 y2 - Algorithm
- Compute p and q (in linear time)
- Compute recursively x1 y1 , x2 y2 , pq
- Perform x1 y1 2n/2 (pq - x1 y1- x2 y2) 2n x2
y2 in linear time
46Time complexity
- T(n) ? 3T(n/2) c n
- T(1) ? c
- Solution T(n) ? d nlog 3, for some constant d ?
3c - Proof For n 1 straightforward T(1) ? c ? d
nlog 3 - General case
- T(n) ? 3T(n/2) c n ? 9T(n/4) 3c(n/2) c
n - ? ? c n (1 3/2 (3/2)2 (3/2)log n)
- c n 2(3/2)log n 1 ? 3c n? nlog 3 - 1
- 3c nlog 3 ? d? nlog 3 O(n1.59)
47Textbook and Exercises
- READING Chapter 5 from Section 5.4
- OBLIGATORY EXERCISES
- How to add two n-bit integers in linear number of
bit operations? - ADDITIONAL EXERCISES
- Solved Exercise 1 from the textbook, chapter 5
Divide and Conquer - Exercises 1, 6, 7 from the textbook, chapter 5
Divide and Conquer
48Conclusions
- Searching in time O(log n)
- Finding majority value in time O(n)
- Divide and conquer in time O(n log n)
- sorting algorithms (MergeSort, RandQuickSort)
- closest points algorithm
- Beyond ?(n log n) O(nlog 3) time algorithm for
integer multiplication