Instructor: Shengyu Zhang

1
CSC3160 Design and Analysis of Algorithms
Week 11 Divide and Conquer

• Instructor Shengyu Zhang

2
Starting example

• Merge sort.
• We have a list of numbers x1, , xn.
• We want to sort them in the increasing order.
• Merge sort
• Cut it into two halves with equal size.
• Suppose 2 divides n for simplicity.
• Sort each of the two halves
• Recursively.
• Merge the two sorted halves to get the final
  answer.
• Use two pointers, one for each half, to scan
  them, during which course do the appropriate
  merge.

3
Complexity

• Suppose this algorithm takes T(n) time for an
  input with n numbers.
• Thus each of the two halves takes T(n/2) time.
• The merging? O(n)
• Scanning 2n elements, an O(1) time operation
  needed for each.
• Total amount of time T(n) 2T(n/2) cn.

4
How to solve/bound this recurrence relation?

• T(n) 2T(n/2) cn
• 2 minuts.
• T(n) 2T(n/2) cn
• 2T(n/4) cn/2
• 4T(n/4) 2cn
• 2T(n/8) cn/4
• 8T(n/8) 3cn
• nT(n/n) __?__ cn
• O(1) O(log n)
• O(n log n).

5
Divide and conquer

• Breaking the problem into subproblems
• that are themselves smaller instances of the same
  type of problem
• Recursively solving these subproblems
• Appropriately combining their answers

6
Complexity

• Suppose the algorithm takes T(n) time for an
  input of size n.
• We break it into a subproblems each of the same
  size n/b.
• In general, a is not necessarily equal to b.
• Recursively solving each of these subproblems
  thus takes time T(n/b)
• Suppose combining the answers takes time O(nd).

7
General recurrence relation

• T(n) aT(n/b) O(nd)
• a gt 0, b gt 1, and d 0 are all constants.
• Then
• Proof in textbook. Not required.
• But you need to know how to apply it.
• For example, in the mergesort, a b 2, d1, so
  its O(n log n).

8
An very important example FFT
9
Multiplication of polynomials

• In many applications, we need to multiply two
  polynomials.
• Typical examples include signal processing.
• A degree-d polynomial is
• A(x) adxd ad-1xd-1 a1x a0
• The summation of two polynomials is easy
• A(x) adxd ad-1xd-1 a1x a0
• ) B(x) bdxd bd-1xd-1 b1x b0
• A(x)B(x) (adbd)xd (ad-1bd-1)xd-1
  (a1b1)x (a0b0).
• which still has degree d.

10
multiplication

• The multiplication of two polynomials is a
  different story.
• A(x) adxd ad-1xd-1 a1x a0
• ?) B(x) bdxd bd-1xd-1 b1x b0
• A(x)B(x) cdxd cd-1xd-1 c1x c0
• where ck a0bk a1bk-1 akb0
• The multiplication can have degree 2d.
• If we directly use this formula to multiply two
  polynomials, it takes O(d2) time.

11
Better? YES!

• By FFT We can do it in time O(d log d)!
• FFT Fast Fourier Transform
• Lets take this step-by-step.

12
What determines/represents a polynomial?

• Lets switch to degree-(d-1) for later
  simplicity.
• We already used coefficients (ad-1, ad-2, , a0)
  to represent a polynomial.
• Coefficient representation
• Other ways?
• Yes By giving values on d (distinct) points.
• Point-value representation.
• FACT A degree-(d-1) polynomial is uniquely
  determined by values on d distinct points.

13
Reason

• We have d coefficients to determine
• (ad-1, ad-2, , a0)
• We know values on d distinct points.
• (x0,y0), , (xd-1,yd-1)
• How to get the coefficients? Just solve the
  system of equations
• ad-1x0d-1 a1x0 a0 y0ad-1x1d-1
  a1x1 a0 y1 ... ad-1xd-1d-1 a1xd-1
  a0 yd-1

14

• In matrix form
• ? x0d-1, , x0, 1 ? ? a0 ? ? y0 ?
  x1d-1, , x1, 1 a1 y1
  ... ?
  xd-1d-1,,xd-1,1? ?ad-1 ? ?yd-1?
• The matrix is van der monde matrix, which is
  invertible. Thus it has a unique solution for the
  coefficients (ad-1, ad-2, , a0)

15
Whats good for the point-value representation

• It makes the multiplication extremely easy
• Though A(x)B(x) is hard
• For any fixed point x0, A(x0)B(x0) is simply
  multiplication of two numbers.
• So given (x0,A(x0)), , (xd-1,A(xd-1)) and
  (x0,B(x0)), , (xd-1,B(xd-1)), its really easy
  to get (x0,A(x0)B(x0)), , (xd-1,A(xd-1)B(xd-1)).
• O(n) time.
• Thus we have the following interesting idea for
  polynomial multiplication

16
Go to an easy world and come back

• HK used to have many industries.
• They later found that mainland has less expensive
  labor.
• So they
• moved the companies to mainland,
• did the work there,
• and then shipped the products back to HK
• This is worth doing if its really cheaper in
  mainland, and the traveling/shipping is not
  expensive either.
• which turned out to be true.

17
In our case

• We need to investigate the cost of traveling.
• Both way.

18
Traveling

• From coefficient representation to point-value
  representation
• Evaluate of polynomial on a number of points.
  --- evaluation.
• From point-value representation to coefficient
  representation
• Compute the polynomial back from the point
  values. --- interpolation.
• Both can be done in O(n log n) time.

19
Evaluation

• One point?
• A(x0) a0 a1x0 ad-1x0d-1
• Directly by the above d2 multiplications
• Horners rule A(x0) a0 x0(a1 x0(a2
  x0(ad-2 x0ad-1)))
• --- O(d) multiplications.

20

• How many points we need to evaluate on?
• Since we later want to reconstruct the product
  polynomial, which has degree 2d.
• We need to have values on 2d points for both A
  and B.

21

• Now evaluation on one point needs O(d), so
  evaluations on 2d points need O(d2).
• Too bad. We want O(d log d).
• Important fact The cost of evaluations on 2d
  points can be cheaper than 2d times the cost of
  one evaluation.
• If we choose the 2d points carefully.
• And it turns out that the 2d chosen points also
  make the interpolation (later) easier.
• This powerful tool is called Fourier Transform.

22
Complex roots of unity

• 1 has complex roots wn ei2p/n, which satisfies
  wnn 1.
• DFT (Discrete Fourier Transform) (a0, , an-1)
  ? (y0, , yn-1)where yk a0 a1wnk
  an-1wnk(n-1)
• Try n3 now.
• FFT a fast algorithm for implementing DFT.

23
All the essences are here

• For A(x) a0 a1x a2x2 an-1xn-1
• Suppose n is a power of 2.
• Define two new polynomials
• A0(x) a0 a2x a4x2 an-2xn/2-1
• A1(x) a1 a3x a5x2 an-1xn/2-1
• Then A(x) A0(x2) xA1(x2)
• Divide and Conquer!
• Evaluating A(x) on (wn0)2, (wn1)2, , (wnn-1)2
  for A0(x) and A1(x).
• Note that (wn0)2, (wn1)2, , (wnn-1)2 are not all
  distinct, they only has n/2 values each repeating
  twice!
• See the circle picture in last slide to convince
  yourself.

24

• Thus we only evaluate n/2 points.
• T(n) 2T(n/2) O(n)
• So T(n) O(n log n), as desired.

25
Interpolation

• How about interpolation?
• i.e., the process to get the coefficients by
  point values.
• Almost the same process!

26

• DFT
• DFT inverse?
• Take the same matrix, but use wn-jk for the
  (j,k)th entry
• As oppose to wnjk as in DFT.
• and then divide the whole matrix by n

27

• Why?
• You can directly check by multiplying the two
  matrices and get I (the identity matrix).
• Or
• The added minus sign makes the matrix to be the
  complex conjugate of the that of DFT.
• And the matrix of DFT is symmetric.
• Actually its unitary.
• Thus conjugate transpose inverse.

28
Two more examples

• Were gonna give two more examples
• Selection
• Matrix multiplication
• Simpler than FFT
• less math involved
• I put FFT in the long class because I dont want
  to cut it into two.

29
Selection

• Problem Given a list of n numbers, find the k-th
  largest.
• Idea?
• We can sort the list, which needs O(n log n).
• Can we do better, say, linear time?
• After all, sorting gives a lot more information
  than we requested.
• Not always a waste consider dynamic programming
  where solutions to subproblems are also produced.

30
pivot

• Suppose we use a number v in the list as a pivot.
• Then we can divide the list into three parts.
• SL Those numbers smaller than v
• Sv Those numbers equal to v
• SR Those numbers larger than v

31
After the partition

• The division is simple just scan the list and
  put elements into the corresponding part.
• O(n) time.
• To select the k-th largest value, it becomes
• Complexity? (Recurrence?)

32
Divide and conquer

• Note that though the problem is reduced to two
  subproblems (of sizes SL and SR), but we only
  need to solve one of them.
• Compare in quicksort, we need to sort both
  substrings!
• Complexity T(n) maxT(SL),T(SR) O(n)
• Not the same as mergesort because now SL and
  SR are not determined
• Depends on the pivot.

33

• If the pivot is the median
• T(n) T(n/2) O(n)
• Thus finally T(n) O(n), as desired.
• If the pivot is at one end
• T(n) T(n-1) O(n)
• Finally T(n) O(n2). Too bad.
• The similarity to quicksort tells us a random
  pivot performs well
• Its away from either end by cn with const. prob.

34

• To be more precise, its in (cn, n-cn) with prob.
  1-2c.
• And the recursion becomes
• T(n) T( (1-2c)n ) O(n)
• We shrink the range by a constant fraction
• Though this constant is less than ½
• But as long as its a constant, it only takes
  O(log n) steps to finish the recursion.
• Repeatedly times half vs. repeatedly times 0.9.

35

• Thus we can use the following simple strategy
• Pick a random pivot,
• If its in (n/4, 3n/4) do the recursion
• Else, go back to pick another random pivot
• Each random picked pivot falls in (n/4, 3n/4)
  with prob. ½. Thus the expected number of trials
  to get a pivot in the range is 2.
• Thus the expected running time is 2O(n) O(n)

36
Matrix multiplication

• Recall the product of two n?n matrices is
  another n?n matrix.
• Question how fast we can multiply the matrices?
• Recall zij ?k1,,n xikykj

37

• This takes O(n3) multiplications (of numbers).
• For a long time, people thought this was the best
  possible.
• Until Straussen came up with the following.

38

• If we break the matrix into blocks
• Then the product is just block multiplication
• 8 matrix multiplications of dimension n/2
• Plus O(n2) additions.

39

• Thus the recurrence is
• T(n) 8T(n/2) O(n2)
• This gives exactly the same O(n3), not
  interesting.
• However, Straussen observed that we can actually
  use only 7 (instead of 8) multiplications of
  matrices with dim n/2.

40
God knows how he came up with it.

• And here is how
• where

41

• Thus the recurrence becomes
• T(n) 7T(n/2) O(n2)
• And solving this gives
• T(n) O(nlog_2 7) O(n2.81).
• Best in theory? O(n2.38).
• Conjecture O(n2).
• In practice? People still use the O(n3) algorithm
  most of the time.
• Its simple.