Algorithm Efficiency

1
Algorithm Efficiency

• There are often many approaches (algorithms) to
  solve a problem. How do we choose between them?
• At the heart of a computer program design are two
  (sometimes conflicting) goals
• 1. To design an algorithm that is easy to
  understand, code, and debug.
• 2. To design an algorithm that makes efficient
  use of the computers resources.
• Goal 1 is the concern of Software Engineering.
• Goal 2 is the concern of data structures and
  algorithm analysis.
• When goal 2 is important, how do we measure an
  algorithms cost?

2
How to Measure Efficiency?

• Empirical comparison (run the programs).
• Only valid for that machine.
• Only valid for that compiler.
• Only valid for that coding of the algorithm.
• Asymptotic Algorithm Analysis
• Must identify critical resources
• time - where we will concentrate
• space
• Identify factors affecting that resource
• For most algorithms, running time depends on
  size of the input.
• Running time is expressed as T(n) for some
  function T on input size n.

3
Examples of Growth Rate

• Example 1
• int largest (int array, int n)
• int currlarge 0
• for (int I0 Iltn I)
• if (arrayIgtcurrlare)
• currlargearrayI
• return currlarge
• Example 2
• sum 0
• for (I1 Iltn I)
• for (j1 jltn j)
• sum

4
Growth Rate Graphs
5
Expanded View
6
Best, Worst and Average Cases

• Not all inputs of a given size take the same
  time.
• Sequential search for K in an array of n
  integers
• Begin at the first element in array and look at
  each element in turn until K is found.
• Best Case
• Worst Case
• Average Case
• While average time seems to be the fairest
  measure, it may be difficult to determine.
• When is the worst case time important?
• Time critical events (real time processing).

7
Faster Computer or Faster Algorithm?

• What happens when we buy a computer 10 times
  faster?
• f(n) n n change
  n/n
• 10n 1,000 10,000 n10n
  10
• 20n 500 5,000 n10n
  10
• 5n log n 250 1,842 sqrt(10)nltnlt10n 7.37
• 2n2 70 223 nsqrt(10)n
  3.16
• 2n 13 16 nn3
  gt1
• nSize of input that can be processed in 1 hour
  (10,000 steps).
• n Size of input that can be processed in one
  hour on the new machine (100,000 steps).

8
Asymptotic Analysis Big-oh

• Definition T(n) is in the set O(f(n)) if there
  exist two positive constants c and n0 such that
  T(n)lt cf(n) for all ngtn0.
• Usage the algorithm is in O(n2) in best,
  average, worst case.
• Meaning for all data sets big enough (i.e.,
  ngtn0), the algorithm always executes in less than
  cf(n) steps in best, average, worst case.
• Upper Bound
• Example if T(n)3n2 then T(n) is in O(n2).
• Tightest upper bound
• T(n)3n2 is in O(n3), we prefer O(n2).

9
Big-oh Example

• Example 1. Finding the value X in an array.
• T(n)csn/2.
• For all values of ngt1, csn/2ltcsn.
  Therefore, by the definition, T(n) is in O(n) for
  n01 and ccs.
• Example 2. T(n)c1n2c2n in the average case
• c1n2c2n lt c1n2c2n2 lt(c1c2)n2 for all
  ngt1.
• Therefore, T(n) is in O(n2).
• Example 3 T(n)c. This is in O(1).

10
Big-Omega

• Definition T(n) is in the set ?(g(n)) if there
  exist two positive constants c and n0 such that
  T(n) gt cg(n) for all ngtn0.
• Meaning For all data sets big enough (i.e., ngtn0
  ), the algorithm always executes in more than
  cg(n) steps.
• It is a LOWER bound.
• Example T(n)c1n2c2n
• c1n2c2n gt c1n2 for all ngt1.
• T(n)gt cn2 for cc1 and n01.
• Therefore, T(n) is in ?(n2) by the definition
• We want the greatest lower bound.

11
Theta Notation

• When big-Oh and ? are the same for an algorithm,
  we indicate this by using ? (big-Theta) notation.
• Definition an algorithm is said to be ?(h(n)) if
  it is in O(h(n)) and it is in ? (h(n)).
• Simplifying rules
• if f(n) is in O(g(n)) and g(n) is in O(h(n)) then
  f(n) is in O(h(n)).
• if f(n) is in O(kg(n)) for any constant kgt0, then
  f(n) is in O(g(n)).
• if f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)),
  then (f1f2)(n) is in O(max(g1(n),g2(n))).
• if f1(n) is in O(g1(n)) and f2(n) is in O(g2(n)),
  then f1(n)f2(n) is in O(g1(n)g2(n)).

12
Big O rules

• If T1(n)O(f(n)) and T2(n)O(g(n)) then
• T1(n)T2(n)max( O(f(n)), O(g(n)) )
• T1(n)T2(n)O(f(n)) O(g(n))
• If T(n) is a polynomial of degree k then
  T(n)?(nk).
• logkn O(n) for any constant k. Logarithms grow
  very slowly.

13
General Algorithm Analysis Rules

• The running time of a for loop is at most the
  running time of the statements inside the for
  loop times the number of iterations.
• Analyze nested loops inside out. Then apply the
  previous rule.
• Consecutive statements just add (so apply the max
  rule).
• The running time of a if/else statement is never
  more than the running time of the test plus the
  larger of the times of the true and false case.

14
Running Time of a Program

• Example 1 ab
• this assignment statement takes constant time, so
  it is ?(1)
• Example 2
• sum0
• for (I1 Iltn I)
• sumn
• Example 3
• sum0
• for (j1 jltn j)
• for (I1 Iltj I)
• sum
• for (k1 kltn k)
• akk-1

15
More Examples

• Example 4
• sum10
• for (I1 Iltn I)
• for (j1 jltn j)
• sum1
• sum20
• for (I1 Iltn I)
• for (j1 jltI j)
• sum2
• Example 5
• sum10
• for (k1 kltn k2)
• for (j1 jltn j)
• sum1
• sum20
• for (k1 kltn k2)
• for (j1 jltk j)
• sum2

16
Binary Search

• int binary (int value, int array, int size)
• int left-1
• int rightsize
• while (left1! right)
• int mid(leftright)/2
• if (value lt arraymid) rightmid
• else if (valuegtarraymid) leftmid
• else return mid
• return -1

17
Binary Search Example

• Position Key
• 0 11
• 1 13
• 2 21
• 3 26
• 4 29
• 5 36
• 6 40
• 7 41
• 8 45
• 9 51
• 10 54
• 11 56
• 12 65
• 13 72
• 14 77
• 15 83

Now lets search for the value 45.
18
Unsuccessful Search

• Position Key
• 0 11
• 1 13
• 2 21
• 3 26
• 4 29
• 5 36
• 6 40
• 7 41
• 8 45
• 9 51
• 10 54
• 11 56
• 12 65
• 13 72
• 14 77
• 15 83
• How many elements are examined in the worse case?

Now lets search for the value 24
19
Case Study Maximum Subsequence

• Given a sequence of integers a1, a2,, an, find
  the subsequence that gives you the largest sum.
• Since there is no size limit on the subsequence,
  then if the sequence is all positive or all
  negative then the solution is trivial.

20
Simple Solution

• Look at all possible combinations of start and
  stop positions of the subsequence.
• for (i0 iltn i)
• for (ji jltn j)
• thissum0
• for (ki kltjk)
• thissumthissumak
• if (thissumgtmaxsum) maxsumthissum

21
Analysis of Simple Solution

• Inner loop is executed j-i1 times.
• The middle loop changes j from i to n-1.
• Looking at the j-I1 when j goes from I to n-1,
  we have 12(n-i).
• So this is done (n-i1)(n-i)/2 times.
• (n2-2nii2n-i)/2

22
More Analysis

• The outer loop changes i from 0 to n-1.
• n2 summed n times is n3
• The sum of 2ni when i changes from 0 to n-1 is
  2n(n-1)(n)/2 n3-n2
• The sum of i2 when i changes from 0 to n-1 is
  (n-1)(n)(2n-1)/6 (2n3-3n2-n)/6
• The sum of n when i changes is n2.
• The sum of i when i changes from 0 to n-1 is
  (n-1)(n)/2 (n2-n)/2.
• Total is (n3 n3-n2 (2n3-3n2-n)/6 n2
  (n2-n)/2)/6
• This is O(n3).

23
An improved Algorithm

• Start at position i and find the sum of all
  subsequences that start at position i. Then
  repeat for all starting positions.
• for (i0 iltn i)
• thissum0
• for (ji jltn j)
• thissumthissumaj
• if (thissumgtmaxsum) maxsumthissum

24
Analysis of Improved Algorithm

• The inner loop goes from i to n-1.
• When i is 0, this is n times
• When i is 1 it is n-1 times
• Until i is n-1 then 1 time
• Summing this up backwards, we get 12n
  n(n1)/2(n2n)/2 O(n2)

25
Final great algorithm

• for (j0 jltn j)
• thissumthissumaj
• if (thissumgtmaxsum)maxsumthissum
• else
• if(thissumlt0) thissum0
• O(n)

26
Analyzing Problems

• Upper bound The upper bound of best known
  algorithm to solve the problem.
• Lower bound The lower bound for every possible
  algorithm to solve that problem, even unknown
  algorithms.
• Example Sorting
• Cost of I/O ?(n)
• Bubble or insertion sort O(n2)
• A better sort (Quicksort Mergesort, Heapsort) O(n
  log n)
• We prove in chapter 8 that sorting is ?(n log n)

27
Multiple Parameters

• Compute the rank ordering for all C pixel values
  in a picture of P pixels.
• Monitors have a fixed number of colors (256, 16M,
  64M).
• Need to count the number of each color and
  determine the most used and least used colors.
• for (i0 iltC i)
• counti0
• for (i0 iltP i)
• countvaluei
• sort(count)
• If we use P as the measure, then time is O(P
  logP)
• Which is bigger, C or P? 600x4002400
  1024x10241M
• More accurate is O(P C log C)

28
Space Bounds

• Space bounds can also be analyzed with asymptotic
  complexity analysis.
• Time Algorithm
• Space Data Structure
• Space/Time Tradeoff Principle
• One can often achieve a reduction in time if one
  is willing to sacrifice space, or vice versa.
• Encoding or packing information
• Boolean flags- takes one bit, but a byte is the
  smallest storage, so pack 8 booleans into 1 byte.
  Takes more time, less space.
• Table Lookup
• Factorials - Compute once, use many times
• Disk based Space/Time Tradeoff Principle
• The smaller you can make your disk storage
  requirements, the faster your program will run.
  Disk is slow.