CS 450 Design

1
CS 450 Design Analysis of AlgorithmsNotes 4

• Fall 2009Sarah Mocas

2
Chapter 4

• Outline Chapter 4
• Recurrences
• Induction - substitution method
• Trees - iteration method
• Master Method

3
Recurrences

• We introduced this idea before, with Merge-Sort.
  
• Recall Just like a recursive program, a
  recurrence equation is defined in terms of itself
  (but using smaller sub-problems).
• A recurrence equation is an equation or
  inequality that describes a function in terms of
  it's values on smaller output.

4
Recurrences

• One way to analyze recursive algorithms is by
  using recurrence equations.
• Example T(n) ?(1) if n 1
• 2T(n/2) ?(n) if n gt 1
• We only want to bound the time T(n), not
  calculate it exactly, so we relax some details.
• T(n) is only defined when n is an integer.
• A boundary condition (the bottom out case of the
  recursion) is usually defined as T(1) ?(1), so
  as a constant value.
• We skip writing the boundary condition.

5
Recurrences

• Recall MergeSort
• Break a list into two parts
• Merge Sort the first part in time T(n/2)
• Merge Sort the second part in time T(n/2)
• Merge the two parts to form a sorted list in c1n

T(n) 2T(n/2) c1n T(1) c2
T(n) 2T(n/2) T(n) T(1) T(1)
or
6
Solving Recurrences

• Three main methods well use
• Induction the Substitution method
• Trees Iteration method
• Master method

7
Complexity of Merge Sort

• Substitution method Example 1 Begin with the
  merge sort recurrence
• Simplified a bit, but it works with all generic
  constants too
• T(n) 2T(n/2) n, T(1) T(1)
• Try (guess) T(n) cn lg(n)
• T(1) 0 no! try T(2) T(3) ok!
• Assume true up to and including n-1 T(n)
  2c(n/2)lg(n/2) n
• cnlg(n) cn n
• If c gt 1 then n cn lt 0 so
• T(n) cn lg(n) cn n cn lg(n) ok!

Log rules lg(n/2) lgn lg 2 and lg 2 1
8
Induction Substitution Method

• Substitution Method
• Basic Idea
• Step 1 Make an educated guess as to the
  solution then
• Step 2 use induction to prove that your guess is
  right.

9
Induction Substitution Method

• Substitution method Example 2
• T(n) 4T(n/2) n, T(1) T(1)
• I claim that T(n) O(n2) (educated guess).
• There is a problem in showing T(n) c n2 by
  induction directly.
• Consider the induction step where we substitute
  (n1) for n T((n1)) 4T((n1)/2) (n1)
• Assume n is odd. By the Induction Hypothesis T(n)
  cn2
• T((n1)) 4T((n1)/2) (n1)
• 4c((n1)/2)2 (n1)
• c(n1)2 (n1)
• But we need T(n) cn2 not the above.
• Instead we will show that T(n) cn2 - n.

10
Induction Substitution Method

• Substitution method Example 2
• T(n) 4T(n/2) n, T(1) T(1)
• Since cn2 n is in O(n2) I still claim that
  T(n) O(n2)
• Base Case Solve T(n) for n 4 (show T(4) c42
  - 4)
• T(4) 4T(4/2) 4
• 4T(2) 4
• 4(4 T(2/2) 2) 4
• 4(T(1) 2) 4
• 4(c 2) 4
• c4 12
• cn2 - n for n4 since
• c 42 - 4 if c lt 2 Okay!
• Note the base case for n1 does not work.

11
Induction Substitution Method

• Substitution method Example 2
• T(n) 4T(n/2) n, T(1) T(1). Proving T(n)
  cn2 - n.
• Induction Hypothesis Assume T(n) cn2 - n is
  true up to some arbitrary value n.
• Induction Step Show that given the assumption in
  the induction hypothesis then T(n1) c(n1)2 -
  (n1) is true.
• T((n1)) 4T((n1)/2) (n1)
• Assume n is odd, then by the Induction
  Hypothesis,
• T((n1)/2) c((n1)/2)2 - (n1)/2 so
  substitute in T(n1)
• T(n1) 4T((n1)/2) (n1)
• 4(c((n1)/2)2 - (n1)/2) (n1)
• c(n1)2 - 2(n1) (n1)
• c(n1)2 - (n1)
• But since cn2 - n O(n2) then T(n) O(n2).

12
Trees Iteration Method

• Iteration Method
• Basic Idea Expand the recurrence until we have a
  sum of terms and then change from the recurrence
  to a summation equation.

13
Trees Iteration Method

• Interation method Example 1
• T(n) 4T(n/2) n, T(1) T(1)
• T(n) 4T(n/2) n
• 4T( 4T(n/4) n/2) n
• 4T( 4T( 4T(n/8) n/4) n/2) n
• 4T( 4T( 4T( 4T(n/16) n/8) n/4) n/2)
  n
• repeat until the boundary condition is met
• So we get (in reverse order)
• T(n) n 4n/2 16n/4 64n/8
  something T(1)
• T(n) n 2n 4n 8n ... something T(1)
• Eventually we reached T(n/x) T(1) so we get
  T(1) something T(1)

14
Trees Iteration Method

• T(n) 4T( 4T( 4T( 4T(n/16) n/8) n/4) n/2)
  n
• Can be rewritten as
• T(n) n 2n 4n 8n ... something T(1)
• But this is
• T(n) 20n 21n 22n 23n ... something
  T(1)
• So the summation (without the bound) is
• T(n) ?i0 to ? 2in something T(1)
• How far do we have to go to get the boundary
  condition?
• If n is a power of 2 then this works out nicely
  so we will assume this first.
• Otherwise we could use T(n) 4T(floor(n/2)) n
  to get a bound.

15
Trees Iteration Method

• Summation (without the bound) is
• T(n) ?i0 to ? 2in something T(1)
• How far do we have to go to get the boundary
  condition?
• We want T(n/x) T(1) so we want n/x 1.
• Since x is growing as a power of 2 then in lgn
  iterations of the recursion we will get that x
  n and n/x 1.
• T(n) 20n 21n 22n 23n ... something
  T(1)
• ?i0 to lg n - 1 2in something
  T(1)
• Since we reach the boundary condition in lgn
  steps then we have to multiply T(1) by 4
  multiplied lgn times.
• T(n) ?i0 to lg n -1 2in 4lgnT(1)
• Now we have to bound the summation.

16

• Bounding the summation.
• T(n) ?i0 to lg n -1 2in 4lgnT(1)
• n?i0 to lg n -1 2i 4lgnT(1)
• We have an equation for
• ?i0 to n xi (xn1 1)/(x - 1)
• We can use this
• ?i0 to lg n -1 2i (2lgn 1)/(2 -
  1)
• so we get
• T(n) n(2lgn - 1) 4lgnT(1)
• We know
• 2lgn n since blog(base b) n n and
• 4lgn n2 since alog(base b) n nlog(base b) a so
  4lgn nlg4
• Combining and since T(1) is a constant value.
• T(n) n(n-1) cn2
• n2 n cn2
• T(n2) polynomial time DONE !!!

• This method is very messy but probably more
  direct then substitution.

17
Trees
Work done at each call

• Picture T(n) 4T(n/2) n, T(1) T(1)

Tree branching factor is number of recursive calls
At any level the cost is (number of nodes)(work
done at a node) Determined by branching factor
and work at each call
Tree height is the depth of the recursion
At the leaves the cost is (number of leaves)T(1)
18

• Interation method Example 2
• Bounding T(n) 3T(n/3) lgn
• 3T(3T(3T(3T(n/81) lg(n/27)) lg(n/9))
  lg(n/3)) lgn
• lgn 3lg(n/3) 9lg(n/9) 27lg(n/27)
  ... 3? T(1)
• We can rewrite this as
• T(n) 30lg(n/30) 31lg(n/31) 32lg(n/32)
  33lg(n/33) ... 3?T(1)
• So the summation (without the boundary) is
• T(n) ?i0 to ? 3ilg(n/3i) 3?T(1)
• Since x in T(n/x) is growing as a power of 3 then
  in log3 n iterations of the recursion we get that
  x n and n/x 1.
• T(n) ?i0 to log(base3) n-1 3ilg(n/3i)
  3?T(1)
• Since we reach the boundary condition in log3 n
  steps then we have to multiply T(1) by 3
  multiplied log3 n times.
• T(n) ?i0 to log(base3) n-1 3ilg(n/3i)
  3log(base 3)nT(1)

19

• Bounding ?i0 to log(base3) n-1 3ilg(n/3i)
  3log(base 3)nT(1).
• We know
• 3log(base 3)n n since blog(base b)n n
• logb(x/y) logb x - logb y
• logban nlogba
• so we can rewrite
• ?i0 to log(base3) n-1 3i(lgn - ilg3) nT(1)
• Separating the summation
• ? 3ilgn - ? i3ilg3 nT(1)
• lgn? 3i - lg3? i3i nT(1)
• Since ?i0 to m m3i ?i0 to m i3i we get
• lgn? 3i - lg3 log(base3)n? 3i nT(1)
• (lgn - lg3 log(base3)n) ?i0 to log(base3) n-1
  3i nT(1)

Summations go from i0 to log(base3) n-1
20
Trees Iteration Method

• Finish bounding
• ?i0 to log(base3) n-1 3ilg(n/3i) 3log(base
  3)nT(1).
• We know ?i0 to n xi (xn1 1)/(x - 1) so
• (lgn - lg3 log(base3)n) (3log(base3) n 1)/(3
  - 1) nT(1)
• ½(n-1)(lgn - lg3 log(base3)n) nT(1)
• Since lg3 lt lg4 and lg4 2 we can bound by
• ½(n-1)(lgn - 2log(base3)n) nT(1)
• n(lgn - 2log(base3)n) nT(1) after some
  small n
• O(nlgn) nTheta(1)
• and so T(n) O(nlgn) time. DONE!

21
Trees
Work done at each call

• Picture T(n) 3T(n/3) lgn, T(1) T(1)

Tree branching factor is number of recursive calls
At any level the cost is ( of nodes)(work done
at a node) Determined by branching factor and
work at each call
Tree height is the depth of the recursion
At the leaves the cost is (number of leaves)T(1)
22
Master Method

• Master Method is a formula that works for certain
  types of recurrences
• Works on many divide and conquer algorithms
• Requires
• T(n) aT(n/b) f(n) and a 0, b gt 1
• plus a few other caveats (coming up)

23
Basic Idea of Master Method

• Figure out which part of the recurrence dominates
  the complexity
• The divide part a T(n/b)
• Or the other work f(n)
• We do this finding out which (if any) of three
  cases are upheld.

24
Basic Idea of Master Method

• T(n) aT(n/b) f(n)
• a is the number of subproblems that are solved
  recursively (the number of recursive calls).
• b is the size of each subproblem relative to n
• (n/b is the size of the input to the recursive
  call).
• f(n) is the cost of dividing and combining
  subproblems.
• In a recursion tree the leaves are at depth logb
  n from the root.

25
Basic Idea of Master Method

• Let i be the level in a recursion tree. Then at
  level i (not including leaves) the cost is
  ai(f(n/bi))
• Cost of the leaves is the number of leaves x T(1)
• The leaves are at depth logbn from the root
  because that is when T(n/blogbn) T(1).
• Since the leaves are at depth logbn then the
  number of leaves is alogbn nlogba
• Since T(n) is the sum of the times at all of the
  nodes and leaves then T(n) ?i0 to log(base
  b)n-1 aif(n/bi) nlogbaT(1)
• The time T(n) might be dominated by
• 1. the cost of the leaves
• 2. the cost of the divide/combine or root
• 3. or evenly distributed at all levels.
• Master method says what the asymptotic running
  time will be depending on which cost is highest
  (dominates).

26
Master Method 3 Cases
T(n) aT(n/b) f(n) and a 0, b gt 1

• All logs are base b
• 1) f(n) O(nlogba-?) for some ? gt0
• 2) f(n) T(nlogba)
• 3) f(n) ?(nlogba? ) for some ?gt0
• and af(n/b) ? cf(n) for constant c lt 1, n gt n0
• Note this doesnt cover all recurrences, not
  even all with a 0, b gt 1

? T(n) T(nlogba) ? T(n) T(nlogba lg n) ?
T(n) T( f(n) )
27
Master Method 3 Cases
T(n) aT(n/b) f(n) and a 0, b gt 1

• 1) f(n) O(nlogba-?) for some ? gt0
• 2) f(n) T(nlogba)
• 3) f(n) ?(nlogba? ) for some ?gt0
• and af(n/b) ? cf(n) for constant c lt 1, n gt n0

? T(n) T(nlogba) ? T(n) T(nlogba lg n) ?
T(n) T( f(n) )

• In each case, what function is larger f(n) or
  nlogba?
• f(n) must be polynomially smaller or larger
• nlogba/n? or n?nlogba where e is a small constant

28
Master Method 1

• Consider T(n) 9T(n/3) n
• Therefore, we have a9, b3 and f(n) n
• nlogb a nlog39 which is T(n2) because 32 9.
• Since f(n) n and nlog39 n2 and so f(n)
  O(n2) we get that f(n) O(nlog39 - ?) where ?
  1. Now we can apply case 1 so the solution is
  T(n) T(n2)

29
Master Method 2

• Let T(n) T(2n/3) 1
• Clearly a 1, b 3/2, f(n) 1, and nlogba is
  nlog3/2 1 (and since log(1) is 0 by definition)
  this means that we have n0 1. Note f(n) T(1)
  
• Which case applies?
• Since f(n) 1 and nlog3/2 1 n0 1 then f(n)
  T(1) and we get that f(n) T(nlogba) so we have
  T(n)T(nlogba lg n) or T(1 lg n) T(lg n). (Case
  2 applies).

30
Master Method 3

• Now let T(n) 3T(n/4) n lg n
• Then a3, b4, and f(n) n lg n.
• nlogba is nlog43
• log4 3 is ln(3)/ln 4 or 0.792 so we have n0.792
• We have f(n) ??(nlog43 ?) where ? is 0.2 (so
  n0.792 ? is n0.8 2 n1) to give f(n)
  ?(n) case 3 applies.
• Second condition For n large enough and for c
  ¾,
• af(n/b) 3(n/4)lg(n/4) ? ¾ n lg n cf(n)
• By case 3, the solution is
• T(n) n lg n.

31
Master Method Merge Sort

• From Merge Sort T(n) 2T(n/2) T(n)
• nlogba nlog22 n1 n
• For case 2, if f(n) T(nlogba), then
• T(n) is T(nlogba lg n) or in this case,
• T(n) T(n lg n)

32
Master Method Binary Search

• From Binary Search, we have
• T(n) T(n/2) T(1)
• nlogba nlog21 n0 1
• Which case applies?
• We have f(n) T(1) so then for case 2
• T(n) T(1 lg n) T(lg n)

33
Master Method 3 Cases
T(n) aT(n/b) f(n) and a 0, b gt 1

• All logs are base b
• 1) f(n) O(nlogba-?) for some ? gt0
• 2) f(n) T(nlogba)
• 3) f(n) ?(nlogba? ) for some ?gt0
• and af(n/b) ? cf(n) for constant c lt 1, n gt n0
• Note this doesnt cover all recurrences, not
  even all with a 0, b gt 1

? T(n) T(nlogba) ? T(n) T(nlogba lg n) ?
T(n) T( f(n) )
34
Master Method What I Do

• T(n) 4 T(n/2) n2
• Find MM parameters a4, b2, f(n) n2
• Calculate (roughly) logba logba 2
• We always want to compare nlogba to f(n)
• So do that.

35
Master Method What I Do

• T(n) 4 T(n/2) n2
• Find MM parameters a4, b2, f(n) n2
• Calculate (roughly) logba logba 2
• Here nlogb a n2 f(n)
• So f(n) T(nlogb a) (case 2)
• And T(n) T(n2 lg n)

36
Master Method What I Do

• Now say T(n) 4 T(n/2) n
• Find MM parameters a4, b2, f(n) n
• Calculate (roughly) logba logba 2
• Here nlogb a n2 gt f(n)
• In fact f(n) O(n2 - ?) for 0 lt e lt 1 (case
  1)
• And T(n) T(n2)

37
Master Method What I Do

• Now say T(n) 4 T(n/2) n3
• Find MM parameters a4, b2, f(n) n3
• Calculate (roughly) logba logba 2
• Here nlogba n2 lt f(n)
• In fact f(n) ?(n2 ?) for 0 lt e lt 1 (case
  3?)
• Check 4(n/2)3 ? c n3 for some c lt 1 ? Yes
• And T(n) T(n3)

38
Key Points

• Solve recurrences with
• Induction
• Interation
• Master method
• Be careful on inductive proofs!