Counting the bits Analysis of Algorithms

1
Counting the bitsAnalysis of Algorithms

• Will it run on a larger problem?
• When will it fail?

2
Software Concerns

• Correctness
• test
• understand
• prove
• Maintainability
• modify
• enhance
• Decomposability
• cohesion and coupling
• multi-programmer, multi-year
• Efficiency
• memory
• cycles

3
Sample Questions

• Task sorting n elements
• How fast can you do it?
• in worst case
• on average
• How much more memory is needed?
• Data assumptions?
• in memory or on disk?
• nearly sorted?
• duplicates retained?
• ....
• Different algorithms for different conditions

4
How can we tell how well an algorithm works?

• 1. Experimentally, I.E.
• Run algorithm on data, report cpu time
• depends on coding
• depends on machine
• depends on compiler
• little generality
• can do it
• expensive
• run multiple times, show average performance
• guess relevant variables and vary them
• Whats relevant for sorting?

5
2. Theoretical Analysis

• Give upper and lower bounds for complexity
• demonstrates an understanding of algorithm
• shows relevant variables
• shows importance of variables
• independent of machine, language, coding
• - analysis is sometimes difficult
• - usually only works for simple algorithms
• - often doesnt include important constants
• Same questions for time can be done for memory
  usage

6
Language of Analysis

• big O notation
• A function f(n) is O(g(n)) iff there are positive
  constants c and N such than f(n) lt cg(n) when
  ngtN.
• Intuitively this says that f is eventually less
  than a constant times g.
• Examples
• 17n2 1000n15 is O(n2). generalize
• 10nnlog10n is O(nlg(n)) where lg log2
• It turns out that sorting is O(n log(n)).
• What about finding the maximum in an array?
• What about finding the k-th largest in an array?

7
Squeezing the complexity lower bounds

• When upper bounds equal lower bounds, then you
  know you have the best possible algorithm.
• A function f(n) is big-omega(g(n)) iff there are
  positive constants c and N such that f(n)gt cg(n)
  for n gt N.
• Intuitively, this says f is eventually larger
  than g.
• example n3 -1000n2 is big-omega of n3.
• Theoreticians work on closing the gap between the
  upper and lower bounds.
• It turns out that sorting is big-omega(nlog(n)),
  hence sorting is solved.
• caveat this assumes no special properties of the
  data.
• sometimes constants count
• sometimes eventually is not good enough

8
Useful Theorems

• Let f and g be function of n
• If f lt g, then O(fg) O(g).
• O(constantf) O(f).
• O(ff) O(f)2.
• O(Polynomial(n) ) O(nmax degree)
• O(f/g) .. Can be anything.
• Style write the simplest and smallest
  O-notation.
• eg. do not write O(n2 3n), but O(n2).
• Do not write O(3n3), but O(n3).

9
Proving O theorems

• To prove if f1 and f2 are O(g(n)) then f1f2 is
  O(g(n)g(n)).
• Proof by your nose.
• Ask what do I need to do?
• By definition find a constant c and integer N
  such that f1(n)f2(n) lt c g(n)g(n) for ngtN
• Ask what do I know?
• There are constants c1,c2, N1,N2 such that
• f1(n) lt c1g(n) for n gtN1
• f2(n) ltc2g(n) for n gtN2
• Arent c and N obvious now?
• Can you do this?

10
Another Proof

• To prove if f1 and f2 are O(g(n)) then f1f2 is
  O(g(n)).
• By your nose
• Since f1 is O(g(n)), f1(n) ltC1g(n) for ngt N1.
• Since f2 is O(g(n)), f2(n) lt C2g(n) for ngt N2.
• Then f1(n)f2(n) lt (C1C2) for ngt max(N1,N2).
• Hint go back to these proofs in a few days and
  see if you can prove them without looking.

11
More notation

• A function f(n) is big-theta of g(n) iff it is
  both big-O and big-omega of g(n).
• Intuitively, they have same growth rate.
• A function f(n) is little-o of g(n) iff it is
  big-O of g(n), but not big-omega of g(n).
• TSP is big-O(2n). Why?
• No one knows if it is big-omega(2n).
• Intuitively this means there is some doubt.
• These results require proofs. One can also get
  estimates by running careful experiments.
• But experiments cant prove anything.

12
Traveling Salesman Problem

• Given n cities and distances between them
• Find tour of minimum length.
• A tour is path that visits every city
• How many tours are possible?
• n(n-1)...1 n!
• n! gt 221 2(n-1)
• So n! is omega of 2n. ( lower bound)
• No one knows if there is a polynomial time
  algorithm to find minimum length tour.
• There are many very good heuristic algorithms.

13
Boolean Satisfiability

• Suppose we have a boolean formula on n-variables
• e.g. (x1 or not x2 or x3)(x2 or not x5 or
  x7)(...)
• This is a conjunct of disjuncts. If restricted to
  at most 3 literals per disjunct then 3-sat.
• Theorem 3-Sat and general Boolean satisfiability
  are equivalently difficult.
• Does this have a solution?
• i.e. assignment of true or false to the variables
  so that the entire formula is true.
• When the formula has at most 3 literals (variable
  or its negation) in each clause, the problem is
  called 3-sat.
• If we just checked every possibility, how many
  cases?
• 2n , so exhaustive search is O(2n)

14
Satisfiability, the value

• No one knows if this can be done in polynomial
  time.
• Nobel prize level problem, but none given in CS.
• One Million prize
• Application Given 2 circuits, are they
  equivalent?
• Suppose they each have n inputs.
• Let the two circuits, or formulas, be f1 and f2.
• Consider XOR(f1, f2).
• This is a boolean formula on n-variables.
• If it has a solution S, then f1(S) ! f2(S) and
  the two circuits are different.

15
A problem we can doLinear Lookup/Search

• Task determine if value is in array (n items)
• Algorithm
• For i 1 to n
• if (ai element) return true
• else return false
• In worst case, time O(n).
• In best case, O(1).
• What about space? O(1)

16
Space Analysis

• If program has a bad time complexity, you can
  wait.
• If program has a bad space complexity, the
  program bombs.
• SPACE is critical.
• Space utilization occurs in two places
• 1. Every time you call new.
• Analysis is like time complexity
• 2. Hidden usage function calls
• Depth of function calls is critical here.
• Not a problem except when using recursion.

17
Find most similar student records

• Assume we have n records RecordI
• Each record has k grades
• Algorithm Sketch
• for j 1n
• for h j1n
• compute similarity of records j and h
• if similarity better than best, save j,h and
  score
• end h
• end j
• return j and h.
• Time complexity O(n2)
• Space complexity O(1).

18
Divide and Conquer

• General Problem Solving method
• Top-down division of labor
• General Algorithm
• Solution Solve(Problem)
• Solve(Problem) if Problem easy,
• 
  return basicSolve(Problem)
• else
• 
  Pieces split(Problem)
• 
  PartSolutions Solve(Pieces)
• 
  return merge(PartSolutions)
• Examples mergesort, quicksort, binary search

19
General analysis of Divide and Conquer

• Suppose splitting yields a subproblems
• Suppose size of subproblems is N/b
• Suppose merging cost is O(nk)
• Then T(N)
• if agtbk O(N(log b a))
• if a bk O(Nk log(N) )
• if a lt bk O(Nk)
• For example, in mergesort where N bm.
• A2, b2, k 1 Therefore O(Nlog(N))
• You should know how to use this theorem (you
  dont need to memorize result).

20
Binary Search

• Binary search (assumes ordered)
• BinarySearch(a, l, u)
• mid (lu)/2
• if ( lu ) return amid element
• if (element amid) return mid
• else if (element ltamid) return
  BinarySearch(a,l,mid)
• else if (element gtamid) return
  BinarySearch(a,mid,u)
• Time Analysis
• f(n) cf(n/2)
• f(1) c.
• Hence f(n) clg(n).
• Expand out to see this, with n a power of 2.

21
Binary Search

• Space Analysis
• Only new memory counts
• New memory occurs when
• constructors are called
• recursion occurs
• Back to code
• What is max depth of stack?
• O(log n)

22
Maximum contiguous subsequence problem

• Given a1an are integers, find i, j such that
  sum of ak from i to j is maximum.
• Algorithmic Approaches
• Exhaustive, dumb but a start
• optimize via problem constraints
• Divide and Conquer
• requires breaking problem in pieces whose
  solutions can be merged
• Dynamic Programming
• requires merging subsolutions of smaller problems
  into full solution
• Be Clever this is for the theoreticians
• Transform a known problem solution

23
Exhaustive

• Best 0, begin 0, end 0
• For i 1n
• for j i1 n
• temp 0
• for k i j
• temp ak
• if (temp gt best)
• best temp
• update begin, end
• Time complexity O(N3)
• Space complexity O(1)
• see text for Java code

24
Partition trick

• For i 1n/3 gt
  n/3 times
• for j 2n/3 n
  gt n/3 times
• temp 0
• for k i j
  gt n/3
• temp ak
• if (temp gt best)
• best temp
• update begin, end
• Time complexity O(N3) N3/9
• More painful analysis in the text.

25
Optimize (exhaustive)

• Best 0, begin 0, end 0
• For i 1n
• temp 0
• for j i1 n
• temp ak
• if (temp gt best)
• best temp
• update begin, end
• Time O(n2)
• Space O(1)

26
Dynamic Programming

• General Idea use results of smaller problem
• Decomposition finding relationship between
  problem and smaller ones
• Idea
• Compute Sij sum ai aj
• Keep track of best
• Note
• Sii ai, boundary condition
• Sik Sik-1 ak
• See Matrix
• Time O(n2)
• Space O(n2)

27
Fibonacci via DP

• Definition of f(k)
• if (k lt2) return 1 else return f(k-1)f(k-2
• Expand f(4) f(3) f(2) f(2)f(1)
  f(1)f(0)
• f(1)f(0)
  f(1)f(1)f(0)
• Redundant computation exponential! (see text)
• Dynnamic programming bottom-up instead of
  top-down.
• StraightForward use array f(k) depends on two
  previous values.
• Optimise dont really need array, just last 2
  values.
• T(N) T(N-1)T(N-2) 2

28
Dynamic Programming (example 2)

• Define Comb(k, n) n! / (k!(n-k)!)
• Why doesnt this compute well?
• Goal find definition in terms of smaller k,n
• Note Comb(k,n) Comb(k-1, n-1) Comb(k,n-1)
• Ok, so dynamic programming will work where
• Skn is determine by element above and
  to left
• time analysis O(n2)
• space analysis O(n2)
• Why is this useful?

29
Divide and Conquer

• Previous examples
• MergeSort, QuickSort, BinarySearch
• Divide array in half, say left and right
• Suppose solutions to left and right
• How to combine?
• From each half need two solutions
• interior solution (away from boundary)
• boundary solution
• Solution is
• best of
• interior left, interior right, sum of (boundary
  solutions)
• See text for code.

30
Clever

• Look at properties of answer
• Note if aiaj is a solution then
• aiap must be positive.
• Best 0, temp 0
• For j 1n
• temp aj
• if (temp gt best)
• best temp
• else if (temp lt0)
• temp 0
• Time O(n)

31
Transform Problem

• Problem what is minimum subsequence sum?
• Replace each ai by -ai. Done.
• Problem. Assume each ai is positive. What is
  maximum product?
• Replace each ai by log(ai).
• Note xy gt uv iff log(xy) gtlog(uv)
• But log(xy) log(x) log(y)
• Transforms not always easy to find.

32
Concrete classes in Collections
33
Summary

• Memory and time complexity measures
• Critical for some applications
• Sometimes the constants count
• Be aware of your computational expenses
• Document non-O(1) memory or time costs
• Consider problem decomposition, either dynamic
  programming or divide and conquer