Analysis of Algorithms

1
Analysis of Algorithms

• Issues
• Correctness
• Time efficiency
• Space efficiency
• Optimality
• Approaches
• Theoretical analysis
• Empirical analysis

2
Theoretical analysis of time efficiency

• Time efficiency is analyzed by determining the
  number of repetitions of the basic operation as a
  function of input size
• Basic operation the operation that contributes
  most towards the running time of the algorithm.
• T(n) copC(n)

3
Input size and basic operation examples
4
Empirical analysis of time efficiency

• Select a specific (typical) sample of inputs
• Use physical unit of time (e.g., milliseconds)
• OR
• Count actual number of basic operations
• Analyze the empirical data

5
Best-case, average-case, worst-case

• For some algorithms efficiency depends on type of
  input
• Worst case W(n) maximum over inputs of size
  n
• Best case B(n) minimum over inputs of
  size n
• Average case A(n) average over inputs of
  size n
• Number of times the basic operation will be
  executed on typical input
• NOT the average of worst and best case
• Expected number of basic operations repetitions
  considered as a random variable under some
  assumption about the probability distribution of
  all possible inputs of size n

6
Example Sequential search

• Problem Given a list of n elements and a search
  key K, find an element equal to K, if any.
• Algorithm Scan the list and compare its
  successive elements with K until either a
  matching element is found (successful search) of
  the list is exhausted (unsuccessful search)
• Worst case C_w (n) n
• Best case C_b (n) 1
• Average case C_a (n) (n1) / 2

7
Types of formulas for basic operation count

• Exact formula
• e.g., C(n) n(n-1)/2
• Formula indicating order of growth with specific
  multiplicative constant
• e.g., C(n) 0.5 n2
• Formula indicating order of growth with unknown
  multiplicative constant
• e.g., C(n) cn2

8
Order of growth

• Most important Order of growth within a constant
  multiple as n?8
• Example
• How much faster will algorithm run on computer
  that is twice as fast?
• How much longer does it take to solve problem of
  double input size?
• See table 2.1

9
Table 2.1
10
Sec 2.2 Asymptotic growth rate Notations

• A way of comparing functions that ignores
  constant factors and small input sizes
• O(g(n)) class of functions f(n) that grow no
  faster than g(n)
• T (g(n)) class of functions f(n) that grow at
  same rate as g(n)
• O(g(n)) class of functions f(n) that grow at
  least as fast as g(n)
• see figures 2.1, 2.2, 2.3

11
Big-oh
12
Big-omega
13
Big-theta
14
Establishing rate of growth Method 1 using
limits
lt

• limn?8 T(n)/g(n)

gt

• Examples
• 10n vs. 2n2 (10n/2n2) 5/n
  limn?8 5/n 0
• n(n1)/2 vs. n2 (n(n1)/2 / n2) (1
  1/n) / 2
• logb n vs. logc n log b n log b
  c log c n

15
LHôpitals rule

• If
• limn?8 f(n) limn?8 g(n) 8
• The derivatives f, g exist,
• Then

• Example logn vs. n

16
Establishing rate of growth Method 2 using
definition

• f(n) is O(g(n)) if order of growth of f(n)
  order of growth of g(n) (within constant
  multiple)
• There exist positive constant c and non-negative
  integer n0 such that
• f(n) c g(n) for every n n0
• Examples
• 10n is O(2n2) O(n2) 10n 10 n2
  for all n 1
• 5n20 is O(10n) O(n) 5n20 5n20n 25n for
  all n 1

17
Establishing rate of growth Method 2 using
definition

• f(n) is O(g(n)) if order of growth of f(n)
  order of growth of g(n) (within constant
  multiple)
• There exist positive constant c and non-negative
  integer n0 such that
• f(n) c g(n) for every n n0
• Examples
• 10n is O(n) 10n 10 n for all n 1
• 5n20 is O(n) 5n20 5n for all n 1

18
Establishing rate of growth Method 2 using
definition

• f(n) is T(g(n)) if order of growth of f(n)
  order of growth of g(n) (within two constant
  multiples)
• There exist positive constants c1, c2 and
  non-negative integer n0 such that
• c1 g(n) f(n) c2 g(n) for
  every n n0
• Examples
• 10n is T(n) 9n 10n 11n for all n 1
• 5n20 is T(n) 5n 5n20 25n for all n 1

19
Basic Asymptotic Efficiency classes
20
Sec 2.3 Time efficiency of nonrecursive
algorithms

• Steps in mathematical analysis of nonrecursive
  algorithms
• Decide on parameter n indicating input size
• Identify algorithms basic operation
• Determine worst, average, and best case for input
  of size n
• Set up summation for C(n) reflecting algorithms
  loop structure
• Simplify summation using standard formulas (see
  Appendix A)

21
Examples

• Matrix multiplication
• Selection sort
• Insertion sort
• Mystery Algorithm

22
Matrix multipliacation
See p. 65
-
-
-
-
n-1 ? i 0

-
n-1 ? j 0
-
n-1 ? k 0
-
c
n-1 ? i 0
n-1 ? j 0
n-1 ? k 0
c
c n
3
23
Selection sort
See p. 99
-
-
-
-
n-2 ? i 0
n-1 ? j i 1
-
c
n-2 ? i 0
n-1 ? j i 1
n-2 ? c (n-i-1) i 0
c
c n (n-1) / 2
24
Insertion sort
See p. 160 - 161
-
-
-
-
-
-
n-1 ? c c (n-1) i 1
n-1 ? i 1
i-1 ? j 0
(best case)
c
(worst case)
25
Mystery algorithm
Gaussian Elimination with Partial Pivoting (p.
207)

• for i 1 to n - 1 do
• max i
• for j i 1 to n do
• if A j, i gt A max, i then max j
• for k i to n 1 do
• swap A i, k with A max, k
• for j i 1 to n do
• for k n 1 downto i do
• A j, k A j, k - A i, k A j, i
  / A i, i

See p. 208 for summations
26
Sec 2.4 Example Recursive evaluation of n !

• Definition n ! 12(n-1)n
• Recursive definition of n! n ! n (n-1) !
  , 0 ! 1
• Algorithm
• if n0 then F(n) 1
• else F(n) F(n-1) n
• return F(n)
• Recurrence for number of multiplications to
  compute n!

27
Time efficiency of recursive algorithms

• Steps in mathematical analysis of recursive
  algorithms
• Decide on parameter n indicating input size
• Identify algorithms basic operation
• Determine worst, average, and best case for input
  of size n
• Set up a recurrence relation and initial
  condition(s) for C(n)-the number of times the
  basic operation will be executed for an input of
  size n (alternatively count recursive calls).
• Solve the recurrence to obtain a closed form or
  estimate the order of magnitude of the solution
  (see Appendix B)

28
Important recurrence types

• One (constant) operation reduces problem size by
  one.
• T(n) T(n-1) c T(1) d
• Solution T(n) (n-1)c d
  linear
• A pass through input reduces problem size by one.
• T(n) T(n-1) cn T(1) d
• Solution T(n) n(n1)/2 1 c d
  quadratic
• One (constant) operation reduces problem size by
  half.
• T(n) T(n/2) c T(1) d
• Solution T(n) c lg n d
  logarithmic
• A pass through input reduces problem size by
  half.
• T(n) 2T(n/2) cn T(1) d
• Solution T(n) cn lg n d n
  n log n

29
A general divide-and-conquer recurrence

• T(n) a T(n/b) f (n) where f (n) ?
  T(nk)
• a lt bk T(n) ? T(nk)
• a bk T(n) ? T(nk lg n )
• a gt bk T(n) ? T(nlog b a)
• Note the same results hold with O instead of T.

30
Fibonacci numbers

• The Fibonacci sequence
• 0, 1, 1, 2, 3, 5, 8, 13, 21,
• Fibonacci recurrence
• F(n) F(n-1) F(n-2)
• F(0) 0
• F(1) 1
• Another example
• A(n) 3A(n-1) 2A(n-2) A(0) 1 A(1) 3

2nd order linear homogeneous recurrence relation
with constant coefficients
31
Solving linear homogeneous recurrence relations
with constant coefficients

• Easy first 1st order LHRRCCs
• C(n) a C(n -1) C(0) t
  Solution C(n) t an
• Extrapolate to 2nd order
• L(n) a L(n-1) b L(n-2)
  A solution? L(n) r n
• Characteristic equation (quadratic)
• Solve to obtain roots r1 and r2e.g. A(n)
  3A(n-1) - 2(n-2)
• xn 3 x(n-1) 2 x(n-2) ? x2 3x
  2 ? x2 3x 2 0
• (x-1)(x-2) 0 ?
  r1 1, r2 2
• General solution to RR linear combination of r1n
  and r2n
• A(n) C1 r1n C2 r2n C1 1 n C2
  2 n C1 C2 2 n
• Particular solution use initial
  conditionse.g.A(0) 1 A(1) 3
• A(0) C1 C2 1 A(1) C1
  2C2 3 ? C2 2 , C1 - 1

32
Computing Fibonacci numbers

• Definition based recursive algorithm
• F(n) F(n-1) F(n-2) for n gt 1, F(0)
  0, F(1) 1
• Nonrecursive brute-force algorithm
• Explicit formula algorithm
• See Eq. 2.11 on P. 80
• Logarithmic algorithm based on formula

• for n1, assuming an efficient way of computing
  matrix powers.