ALGORITHM ANALYSIS

1
ALGORITHM ANALYSIS

• Analysis of Resource consumption by processor
• Time
• Memory
• Number of processors
• Power
• Proof that the algorithm works correctly

2
ALGORITHM ANALYSIS

• Time is measured with Step Counts in an Algorithm
• Why?
• Not a constant number, but as a function of Input
  Size
• Why?
• We are interested primarily in Worst Case
  Scenario
• Why?

3
WHY STEP-COUNTED?

• Code run time varies depending on
• Processor speed
• Language
• I/O
• What is the number of steps by the following
  fragment?
• For i1100 for j 1 2 print i

4
Why time-complexity as function of input-size n?

• What is the number of steps by the following
  fragment?
• For i1n for j 1 m print i
• Algorithms are to work on different problem sizes
• What is the input size
• Number of elements in input, e.g. array size in a
  search problem

5
Why Worst-case scenario?

• What is the time-complexity (steps) search alg
  below?
• Input array A1..n 3, 9, 5, 7, search key k
  3
• For i1n
• if k Ai print i
• return
• Best case time complexity is normally
  meaningless
• Sometimes Average-case analysis is done, assuming
  some distribution of input types

6
Reminding of Complexity Functions

• Upper bound (Big-Oh) t(N) O(f(N)) if there
  exists a positive constant c (real) and a
  corresponding integer n0 such that t(N) ? c.f(N)
  when N ? n0, for some integer n0.
• Related definitions
• Lower bound t(N) ?(g(N)) if there exists a
  positive constant c (real) and a corresponding
  integer n0 such that t(N) ? c.g(N) when N? n0.
• Tight bound T(N) ?(h(N)) if and only if t(N)
  O(h(N)) and t(N) ?(h(N)).

7
A short-cut way of comparing two functions f(n)
and g(n)

• Lim n-gtinf f(n) / g(n) constant, implies both
  f(n) and g(n) have the same order of terms in n,
  or, f(n) ?(g(n))
• Example f(n) 3n2 5n, g(n) 7n2, the
  limit is 3/7
•  
• Lim n-gtinf f(n) / g(n) 0, implies g(n) has
  higher order term in n than that in f(n), or,
  f(n) O(g(n))
• Example f(n) 3n2 5n, g(n) 2n3, the
  limit is 0
•  
• Lim n-gtinf f(n) / g(n) inf, implies f(n)
  has higher order term in n than that in g(n), or,
  f(n) ?(g(n))
• Example f(n) 3n2 5n, g(n) 22n, the
  limit is infinity

8
Points to Note

• Order complexities are typically monotonically
  increasing functions with respect to
  problem-input size n.
• We typically consider only dominant term ignoring
  the other ones
• 2(n2) 3n we ignore the 3n, and the constant
  multiplicand 2 (absorbed in c).
• So, 2(n2) 3n O(n2). It is actually ?( n2).
• O is often loosely (and confusingly) used for ?
  in the literature, even in this class!
• n2 3n O(n2), also O(n3), and O(n4),
  
• However, we will prefer the closest one O(n2) as
  the acceptable order notation for the function
  actually we are referring to ?( n2).
•  
• Increasing order of functions
• Constant or O(1), O(log N), O(log2 N), ,
  O(N), O(N logN), O(N2), O(N2log N), O(N3), ,
  O(2N), O(3N),

9
GENERAL RULES

• Loops number of iterations/ recursions with
  respect to the input problem instance size N
•  
• Nested loops multiplied
• for I 1 through N do
• for j 1 through N do
• -whatever-
•  
• Analysis O(NN)
•  
• Consecutive statements add steps, which leads to
  the max
• for I 1 through N do
• -whatever-
• for I 1 through N do
• for j 1 through N do
• -moreover-
•  
• Analysis O(N) O(N2) --gt O(N2)
•  
• Conditional statements max of the alternative
  paths (worst-case)

10
Why Study Complexity?
Code Fibonacci Series recursive iterative
algorithms and time against input sizes
11
Background Needed

• Data Structures Big-O notations, Linked list,
  Sorting, Searching, recursion,,,
• Discrete Mathematics Set theory, Logic,
  Recurrence equation, Matrix operations,,,
• Programming experience

12
Syllabus overview

• Introduction
• Four algorithm types divide-conquer, greedy,
  dynamic programming, branch-bound
• Graph algorithms (a few only!)
• Complexity theory NP-completeness
• An advanced topic, e.g. linear programming
• Undergrad GPU coding project
• Grad An advanced topic self-study report

13
Syllabus objective

• More conscious problem solver
• Exposure to proving correctness of algorithm (and
  its connection to both solving a problem, and
  computing its complexity)
• Fundamental understanding of computing problem
  complexity
• Undergrad project GPU computing
• Grad project introduction to a current topic

14
Example MAXIMUM SUBSEQUENCE SUM (MSQS)

• Problem MSSQ Given an array of numbers find a
  subsequence whose sum is maximum out of all such
  subsequences.
• Example 3, 4, 7, 1, 9, -2, 3, -1
• (e.g. rise and fall in stock-market)
• Answer 11 (for subsequence 1 9 -2 3 11)
• Note for all positive integers the answer is
  sum of the whole array.

15
MSQS Algorithm 1

• Input Array AN, e.g. 3, 4, 7, 1, 9, -2,
  3, -1
• Output Maximum subsequence sum, e.g., 11
• Algorithm 1
• maxSum 0 // We expect
  non-negative value here at the end
• for (i 0 through N-1) do
• for (j i through N-1) do // choice of
  subsequence i through j
• thisSum 0
• for (k i through j) do // addition loop
• thisSum thisSum ak // O(N3)
•  
• if (thisSum gt maxSum) then
• maxSum thisSum // O(N2)
• return maxSum
• End Algorithm 1.
• Analysis of Algorithm 1 ?i0N-1 ?jiN-1
  ?kij 1 O(N3)

16
MSQS Algorithm 2

• Input Array AN, e.g. 3, 4, 7, 1, 9, -2,
  3, -1
• Output Maximum subsequence sum, e.g., 11
• Algorithm 2
• maxSum 0 // We expect non-negative value
  here at the end
• for (i 0 through N-1) do
• thisSum 0
• for (j i through N-1) do
• thisSum thisSum aj // reuse the
  partial sum from
• // the previous iteration
• if (thisSum gt maxSum) then
• maxSum thisSum
• return maxSum
• End Algorithm 2.
•  
• Analysis of Algorithm 2 ?i0N-1 ?jiN-1 1
  O(N2)

17
MSQS Algorithm 3
Complexity T(n) 2T(n/2) O(N) T(1)
1 Solve T(n) O(n log n) Inductive
Proof (base) n1 (hypothesis) lines 15, 16
18-32 work correctly for n2k (step) These are
only three options So, lines 34-35 returns
correct sum.
Weiss, textbook, 1999
18
MSQS Algorithm 4

• Input Array AN, e.g. 3, 4, 7, 1, 9, -2,
  3, -1
• Output Maximum subsequence sum, e.g., 11
• Algorithm 4
• maxSum 0 thisSum 0
• for (j 0 through N-1) do
• thisSum thisSum aj // reuse the
  partial sum from
• // the previous iteration
• if (thisSum gt maxSum) then
• maxSum thisSum
• else if (thisSum lt 0) then
• thisSum 0 // ignore computations so
  far
• return maxSum
• End Algorithm 4.
•  
• Analysis of Algorithm 4 ?i0N-1 O(1)
  O(N)
• Exercise (1) How to track the start-end indices
  for the maxSum solution?
• (2) Prove the algorithm 4, stating any underlying
  assumption/hypothesis.

19
Proof of MSQS Algo-4

• Lemma 1 Maximum-sub-sequence-sum is non
  negative. Proof trivial. (the problem is defined
  as such)
• Lemma 2 No pre-fix of a non-negative
  maximum-sub-sequence-sum (MSQS) can be negative.
• Proof by contradiction Presume such a MSQS P
  (W Q), W is prefix and Q is suffix parts.
• We know, ?P ?W ?Q if ?W lt0 then ?P lt ?Q
  Hence, ?P is not MSQS.
• Thm MSQS Algo-4 finds the MSQS correctly.
• Proof sketch by induction on loop invariant
  maxSum
• Base for null input maxSum0 is MSQS
• Hypothesis at every iteration j k, maxSum is
  MSQS up to (a1, a2, , ak-1)
• Step if maxSumak lt0 then by Lemma 2 it cannot
  be prefix in any MSQS,
• and step 8 rejects the sequence built so far.
• Input Array AN, e.g. 3, 4, 7, 1, 9, -2,
  3, -1
• Output Maximum subsequence sum, e.g., 11
• Algorithm 4
• maxSum 0 thisSum 0
• for (j 0 through N-1) do
• thisSum thisSum aj // reuse the
  partial sum from
• // the previous iteration
• if (thisSum gt maxSum) then

20
Summary

• What do we analyze in an algorithm, why, and how
  do we analyze?
• Basic O-notations
• How complexity matters
• Exercise Code the following two fibonacci series
  algorithms and increase input values, check time.
  What are the step/time-complexities?
  Space-complexities?
• Input an integer n Output Fibonacci number for
  n

Recursive Algorithm Fib(n) if n 0 or n 1 return 1 else return (Fib(n-1) Fib(n-2)) End Fib. Iterative version FibIt(n) temp1 1 temp21 for i3n do temp3 temp1 temp2 temp1 temp2 temp2 temp3 end for return temp2 End FibIt.