Complexity Analysis

1
Complexity Analysis
2
Complexity

• The complexity of an algorithm quantifies the
  resources needed as a function of the amount of
  input data size.
• The resource measured is usually time (cpu
  cycles), or sometimes space (memory).
• Complexity is not an absolute measure, but a
  bounding function characterizing the behavior of
  the algorithm as the size of the data set
  increases.

3
Usefulness

• It allows the comparison of algorithms for
  efficiency, and predicts their behavior as data
  size increases.
• A particular algorithm may take years to compute.
  If you know beforehand, you wont just wait for
  it to finish

4
Complexity defined

• Big-Oh notation indicates the family to which an
  algorithm belongs.
• Formally, T(n) is O(f(n)) if there exist positive
  constants k and n0 such that
• T(n) ? k f(n)
  for all n gt n0.
• Examples

linear
quadratic
5
Complexity Classes

• There is a hierarchy of complexities.
• Heres some of the hierarchy
• O(1) lt O(log n) lt O(n) lt O(n logn) lt O(n2) lt
  O(n3) lt O(2n) lt O(n!)
• An algorithm with smaller complexity is normally
  preferable to one with larger complexity.
• Tuning an algorithm without affecting its
  complexity is usually not as good as finding an
  algorithm with better complexity.

6
Importance of Complexity

• consider each operation (step) takes 1ms
• n log2n n log2n n2 2n
• --------------------------------------------------
  --------------------
• 2 1ms 2ms 4ms 4ms
• 16 4ms 64ms 256ms 65ms
• 256 8ms 2ms 65ms 4 x 1063 years
• 10ms 10ms 1sec 6 x 10294 years

7
But computers are getting faster, right ?

• CPU speed 1.5 faster every 18 months
• (True fact re circuit density Moores law
  1965)

8
Computers are getting faster, but

• consider each operation (step) takes 1ms
• n log2n n log2n n2 2n
• --------------------------------------------------
  --------------------
• 2 1ms 2ms 4ms 4ms
• 16 4ms 64ms 256ms 65ms
• 256 8ms 2ms 65ms 4 x 1063 years
• 10ms 10ms 1sec 6 x 10294 years
• NP-complete class of problems best known alg.
  O(2n)

9
Importance of Complexity

• consider each operation (step) takes 1ms
• n 1 million
• Bubble sort will take O(n2) or 1012 operations
  or 277 hours.
• Quicksort, mergesort will take O(n logn) or 107
  operations or 10 sec.
• Sequential search will take O(n) or 500000
  comparisons (average).
• Binary search will take O(log n) or 20
  comparisons on average.

10
Importance of Complexity

• Trivia first binary search algo published in
  1946
• first correct binary search algo published
  in 1962

First tune the algorithm, then tune the code !
11
Deriving f(n) from T(n)

• Reduce to the dominant term As n becomes very
  large, what is the approximation of T(n)?
• E.g. O(n2 106n) O(n2).
• Eliminate multiplicative constants Its the
  characteristic shape of f(n) that matters.
• E.g. O(3n) O(n) O(n/3).

12
BubbleSort (non-recursive)

• for (i n i gt 1 i--) / Line 1 /
• for (j 1 j lt i j) / 2 /
• if (aj-1 gt aj) / 3 /
• temp aj-1 / 4 /
• aj-1 aj / 5 /
• aj temp / 6 /

13
BubbleSort

• for (i n i gt 1 i--) / Line 1 /
• for (j 1 j lt i j) / 2 /
• if (aj-1 gt aj) / 3 /
• temp aj-1 / 4 /
• aj-1 aj / 5 /
• aj temp / 6 /
• Line 1 Executed n times.
• Lines 2 3 Executed n (n-1) (n-2) 2
  1 n(n 1) / 2 times.
• O((n2 n)/2) O(n2)

14
Factorial (recursive functions)

• int fact (int n)
• if (n 0) return 1
• return n fact (n 1)
• T(n) T(n - 1) 1
• T(1) 0
• T(n) 1 T(n - 1) 1 1 T(n - 2) 1 1
  1 n
• O(n)

15
More Formally Solutions for Recursion

• Method 1 Induction
• - guess solution (constant k and function f(n)
  such that
• T(n) ? k f(n) for all n gt n0
• - prove by induction
• Method 2 Iterate T(n) recurrence, then prove by
  induction
• Method 3 Recursion trees

16
Recurrence Equations (Method 2)

• MergeSort
• T(n) - time for merge sort ?
• Calls itself recursively on the two halves (so
  2T(n/2)
• Merges the two halves in n steps
• T(n) n 2T(n/2)
• T(1) 1

17
Recurrence Equations (Method 2)

• T(n) time for merge sort
• T(n) n 2T(n/2) n 2 (n/2 2T(n/4))
• n n 4T(n/4) n n 4(n/4 2T(n/8))
  
• n n n 8(n/8 2T(n/16)) n logn

18
Recurrence Trees Method 3

• int fibonacci (int n)
• if (n lt 1) return 1
• return fibonacci(n 1) fibonacci (n - 2)
• 1 1 2 3 5 8 13