Big-O and Friends

1
Big-O and Friends
2
Formal definition of Big-O

• A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) lt
  cg(n) for all n gt N
• Example Let f(n) 2n2 3n 1
• n 1 2 3 4 5 6
  7 8
• f(n) 6 15 28 45 66 91 120
  153
• Let g(n) n2 and let c 3
• cg(n) 3 12 27 48 75 108 147 192
• So f(n) lt cg(n) whenever n gt N, wherec 3,
  g(n) n2, and N 4

• Hence, f(n) 2n2 3n 1 is O(n2)

3
Informal explanation

• A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) lt
  cg(n) for all n gt N
• The part about for all n gt N means we can
  ignore small values of n

• The c lets us adjust for functions that vary only
  by a constant

4
Big-O is an upper bound

• Big-O is an upper bound, not a lower bound
• Here is the same example as before
• n 1 2 3 4 5 6
  7 8
• f(n) 6 15 28 45 66 91 120
  153
• Let g(n) n3 and let c 1
• cg(n) 1 8 27 64 125 216 343
  512
• So f(n) lt cg(n) whenever n gt N, wherec 1,
  g(n) n3, and N 4

• Hence, f(n) 2n2 3n 1 is O(n3)

5
One more time

• Again, A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) lt
  cg(n) for all n gt N
• n 1 2 3 4 5 6
  7 8
• f(n) 6 15 28 45 66 91 120
  153
• Let g(n) 2n and let c 1
• cg(n) 2 4 8 16 32 64 128
  256
• So f(n) lt cg(n) whenever n gt N, wherec 1,
  g(n) 2n, and N 7
• That is, 2n2 3n 1 lt 1 2n whenever n gt 7

• Hence, f(n) 2n2 3n 1 is O(2n)

6
Big-O is a sloppy bound

• Formally, Big-O is just an upper bound
• For example, all the sorting algorithms we have
  studied can be said to be O(n2), or O(n3),
  orO(n3 log n), or O(2n), etc.
• However, these sorting algorithms are not O(1),
  or O(log n), or O(n)
• If you are asked on a test a question such as Is
  insertion sort an O(n3) algorithm? the correct
  answer is Yes
• Informally, however, we use Big-O to mean the
  least upper bound we can find

7
Big-O is formally defined

• A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) lt
  cg(n) for all n gt N
• Note that this does not put any restrictions on
  the form of the function g(n)
• Hence, it is reasonable and legal to say that,
  for example, an algorithm is O(2n2 3n 1)
• We usually want to simplify the function g(n)
• Since Big-O notation is formally defined, we can
  prove certain properties of Big-O that allow us
  to simplify the g(n) function

8
Big-O notation is transitive

• If f(n) is O(g(n)), and g(n) is O(h(n)),then
  f(n) is O(h(n))
• By definition, there exist c1 and N1 such
  thatf(n) lt c1g(n) for all n gt N1
• By definition, there exist c2 and N2 such
  thatg(n) lt c2h(n) for all n gt N2
• Hence, f(n) lt c1g(n) lt c1c2h(n) for N max(N1,
  N2)
• So if we take c c1c2 and N max(N1, N2)then
  f(n) lt ch(n) for all n gt N
• So what?
• This allows us to simplify O(O(g(n))) to O(g(n))

9
More theorems about Big-O

• If f(n) is O(h(n)) and g(n) is also O(h(n)),
  thenf(n) g(n) is O(h(n))
• The function ank is O(nk)
• The function nk is O(nkj) for any positive j
• From the above, we conclude that the Big-O of any
  polynomial is its highest power
• Example 3n4 5n3 8n
• By 2 O(3n4) is O(n4), O(5n3) is O(n3), and O(8n)
  is O(n)
• By 3 O(n4) is O(n4), O(n3) is O(n4), and O(n) is
  O(n4)
• By 1 O(n4) O(n4) O(n4) is O(n4)

10
Is is not equals

• We have been making statements such as,O(n) is
  O(n4)
• It would be wrong to say O(n) O(n4)
• We are using X is Y to mean Any X is also a Y
• We might say Fido is a dog
• This is an asymmetric relation It is not true
  that any dog is Fido
• When we say O(n) is O(n4), we mean that
  anything which is O(n) is also O(n4), but not
  necessarily the reverse

11
Still more theorems about Big-O

• If f(n) cg(n), then f(n) is O(g(n))
• O(logAn) is O(logBn) for any positive numbers A
  ? 1 and B ? 1
• Changing the base of a logarithm just changes the
  result by a constant multiplier
• logAX (logAB)logBX
• O(logAn) is O(log2n) for any positive A ? 1
• Hence, we can always use logarithms base 2

12
Big-O

• Big-O is an upper bound
• Big-O is a lower bound
• A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) lt
  cg(n) for all n gt N
• A function f(n) is O(g(n)) if there exist
  positive numbers c and N such that f(n) gt
  cg(n) for all n gt N
• If a function is O(n2), it is at least as fast as
  n2
• If a function is O(n2), it is at least as slow as
  n2

13
Big-?

• A function f(n) is ?(g(n)) if there exist
  positive numbers c1, c2 and N such that
  c1g(n) lt f(n) lt c2g(n) for all n gt N
• Big-? sets both a lower and an upper bound
• A function f(n) is ?(g(n)) if it is both O(g(n))
  and ?(g(n))
• This does not make ?(g(n)) uniquef(n) 2n2
  3n 1 is ?(n2) but it is also ?(2n2), ?(3n2),
  ?(86n2), etc.
• Why not always use Big-? instead of Big-O?
• Not every algorithm has a Big-? value
• Quicksort, for example, can be O(n) or O(n2)

14
What you need to know

• Memorize and understand the definitions of Big-O,
  Big-O, and Big-?, particularly Big-O
• Expect to see these on tests
• More importantly, you need to be able to examine
  an algorithm and determine its running time
• Usually this is not difficult
• Sometimes it can range from quite difficult to
  nearly impossible
• We will come back to this in a later lecture

15
Simple analysis

• For series of loops, just add
• for (int i 0 i lt n i) ...some O(1)
  stuff...for (int j 0 j lt m j) ...some
  more O(1) stuff...
• This is O(n) O(m) time
• If n is proportional to m, this simplifies to
  O(n) time
• For nested loops, just multiply
• for (int i 0 i lt n i) ...some O(1)
  stuff... for (int j 0 j lt m j)
  ...some more O(1) stuff
• This is O(nm) time
• If m is proportional to n, this is the same as
  O(n2) time

16
More complex analysis

• Recursion can add some complications
• static int depth(BinaryTree tree) if (tree
  null) return 0 int leftDepth
  depth(tree.getLeftChild()) int rightDepth
  depth(tree.getRightChild()) return
  Math.max(leftDepth, rightDepth) 1
• Formal analysis is a bit tricky, but...
• This algorithm visits every node in the binary
  tree exactly once
• Therefore, the algorithm is O(n), where n is the
  number of nodes in the binary tree

17
The End