Title: Design and Analysis of Computer Algorithm Lecture 21
1Design and Analysis of Computer AlgorithmLecture
2-1
- Pradondet Nilagupta
- Department of Computer Engineering
2Acknowledgement
- This lecture note has been summarized from
lecture note on Data Structure and Algorithm,
Design and Analysis of Computer Algorithm all
over the world. I cant remember where those
slide come from. However, Id like to thank all
professors who create such a good work on those
lecture notes. Without those lectures, this slide
cant be finished.
3Algorithm 1
- 1 int count_1( int N)
- 2
- 3 sum 0
- 4 for i1 to N
- 5 for ji to N
- 6 sum
- 7
- 8
- 9 return sum
- 10
1
1
The running time is
4Algorithm 2
- 1 int count_2( int N)
- 2
- 3 sum 0
- 4 for i1 to N
- 5 sum N1-j
- 6
- 7 return sum
- 8
1
1
The running time is
5Algorithm 3
- 1 int count_3( int N)
- 2
- 3 sum N(N1)/2
- 4 return sum
- 5
4
1
The running time is 5 time unit
6Algorithm 4
- 1 int count_0( int N)
- 2
- 3 sum 0
- 4 for i1 to N
- 5 for j1 to N
- 6 If iltj then
- 7 sum
- 8
- 9
- 10 return sum
- 11
O(1)
O(N)
O(N2)
O(N2)
O(N2)
O(1)
The running time is O(N2)
7Summary of Running Times
8Asymptotic Running Times
9Example Sorting
- (approximate number of comparisons)
Question How about other operations such as
additions, and other instructions? ---gt Focus
on Growth rate
10Running Times and Big O
11Time Complexity of Algorithm
12Asymptotic Analysis
13Asymptotic Notation
- Think of n as the number of records we wish to
sort with an algorithm that takes f(n) to run.
How long will it take to sort n records? - What if n is big?
- We are interested in the range of a function as n
gets large. - Will f stay bounded?
- Will f grow linearly?
- Will f grow exponentially?
- Our goal is to find out just how fast f grows
with respect to n.
14Classifying functions by theirAsymptotic Growth
Rates (1/2)
- asymptotic growth rate, asymptotic order, or
order of functions - Comparing and classifying functions that ignores
constant factors and small inputs. - The Sets big oh O(g), big theta ?(g), big omega
?(g)
?(g) functions that grow at least as fast as g
g
?(g) functions that grow at the same rate as g
O(g) functions that grow no faster as g
15Classifying functions by theirAsymptotic Growth
Rates (2/2)
- O(g(n)), Big-Oh of g of n, the Asymptotic Upper
Bound - Q(g(n)), Theta of g of n, the Asymptotic Tight
Bound and - W(g(n)), Omega of g of n, the Asymptotic Lower
Bound.
16Example
- Example f(n) n2 - 5n 13.
- The constant 13 doesn't change as n grows, so it
is not crucial. The low order term, -5n, doesn't
have much effect on f compared to the quadratic
term, n2. - We will show that f(n) Q(n2) .
- Q What does it mean to say f(n) Q(g(n)) ?
- A Intuitively, it means that function f is the
same order of magnitude as g.
17Example (cont.)
- Q What does it mean to say f1(n) Q(1)?
- A f1(n) Q(1) means after a few n, f1 is
bounded above below by a constant. - Q What does it mean to say f2(n) Q(n lg n)?
- A f2(n) Q(n lg n) means that after a few n, f2
is bounded above and below by a constant times
nlg n. In other words, f2 is the same order of
magnitude as nlg n. - More generally, f(n) Q(g(n)) means that f(n) is
a member of Q(g(n)) where Q(g(n)) is a set of
functions of the same order of magnitude.
18Big-Oh
- The O symbol was introduced in 1927 to indicate
relative growth of two functions based on
asymptotic behavior of the functions now used to
classify functions and families of functions
19Upper Bound Notation
- We say Insertion Sorts run time is O(n2)
- Properly we should say run time is in O(n2)
- Read O as Big-O (youll also hear it as
order) - In general a function
- f(n) is O(g(n)) if ? positive constants c and n0
such that f(n) ? c ? g(n) ? n ? n0 - e.g. if f(n)1000n and g(n)n2, n0 gt 1000 and c
1 then f(n0) lt 1.g(n0) and we say that f(n)
O(g(n)) - The O notation indicates 'bounded above by a
constant multiple of.'
20Asymptotic Upper Bound
c g(n)
- f(n) ? c g(n) for all n ? n0
- g(n) is called an
- asymptotic upper bound of f(n).
- We write f(n)O(g(n))
- It reads f(n) is big oh of g(n).
f(n)
g(n)
n0
21Big-Oh, the Asymptotic Upper Bound
- This is the most popular notation for run time
since we're usually looking for worst case time. - If Running Time of Algorithm X is O(n2) , then
for any input the running time of algorithm X is
at most a quadratic function, for sufficiently
large n. - Some upper bounds are not very tight but they are
still upper bounds. - e.g. 2n2 O(n3) .
- From the definition using c 1 and n0 2. O(n2)
is tighter.
22Example 1
for all ngt6, g(n) gt 1 f(n). Thus the
function f is in the big-O of g. that is, f(n)
in O(g(n)).
23Example 2
There exists a n0 s.t. for all ngtn0, f(n) lt 1
g(n). Thus, f(n) is in O(g(n)).
24Example 3
There exists a n05, c3.5, s.t. for all ngtn0,
f(n) lt c h(n). Thus, f(n) is in O(h(n)).
25Example of Asymptotic Upper Bound
4g(n)4n2
- 4 g(n) 4n2
- 3n2 n2
- ? 3n2 9 for all n ? 3
- gt 3n2 5
- f(n)
- Thus, f(n)O(g(n)).
f(n)3n25
g(n)n2
3
26Exercise on O-notation
- Show that 3n22n5 O(n2)
- 10 n2 3n2 2n2 5n2
- ? 3n2 2n 5 for n ? 1
- c 10, n0 1
27Usage
- We usually use the simplest formula when we use
the O-notation. - We write
- 3n22n5 O(n2)
- The followings are all correct but we dont
usually use them - 3n22n5 O(3n22n5)
- 3n22n5 O(n2n)
- 3n22n5 O(3n2)
28Exercise on O-notation
- O(n2)
- O(n log n)
- O(n2)
- O(n log n)
- f1(n) 10 n 25 n2
- f2(n) 20 n log n 5 n
- f3(n) 12 n log n 0.05 n2
- f4(n) n1/2 3 n log n
29Classification of Function BIG O (1/2)
- A function f(n) is said to be of at most
logarithmic growth if f(n) O(log n) - A function f(n) is said to be of at most
quadratic growth if f(n) O(n2) - A function f(n) is said to be of at most
polynomial growth if f(n) O(nk), for some
natural number k gt 1 - A function f(n) is said to be of at most
exponential growth if there is a constant c, such
that f(n) O(cn), and c gt 1 - A function f(n) is said to be of at most
factorial growth if f(n) O(n!).
30Classification of Function BIG O (2/2)
- A function f(n) is said to have constant running
time if the size of the input n has no effect on
the running time of the algorithm (e.g.,
assignment of a value to a variable). The
equation for this algorithm is f(n) c - Other logarithmic classifications f(n) O(n
log n) -
f(n) O(log log n)
31Big O Fact
- A polynomial of degree k is O(nk)
- Proof
- Suppose f(n) bknk bk-1nk-1 b1n b0
- Let ai bi
- f(n) ? aknk ak-1nk-1 a1n a0
32Some Rules
- Transitivity
- f(n) O(g(n)) and g(n) O(h(n)) gt f(n)
O(h(n)) - Addition
- f(n) g(n) O(max f(n) ,g(n))
- Polynomials
- a0 a1n adnd O(nd)
- Heirachy of functions
- n log n O(n) 2n n3 O(2n)
33Some Rules
- Base of Logs ignored
- logan O(logbn)
- Power inside logs ignored
- log(n2) O(log n)
- Base and powers in exponents not ignored
- 3n is not O(2n)
- 2
- a(n ) is not O(an)
34OMEGA and LITTLE O
- Big O and Big Theta represent the asymptotic
behavior of functions by using the lt and
relationships. - gt and lt relationships are represented by Omega
and little o. - If f W(g), then f is at least as big as g (or g
is a lower bound for f) - e.g.f(n) n3 and g(n) n2
35Lower Bound Notation
- We say InsertionSorts run time is ?(n)
- In general a function
- f(n) is ?(g(n)) if ? positive constants c and n0
such that 0 ? c?g(n) ? f(n) ? n ? n0 - Proof
- Suppose run time is an b
- Assume a and b are positive (what if b is
negative?) - an ? an b
36Asymptotic Lower Bound
- f(n) ? c g(n) for all n ? n0
- g(n) is called an
- asymptotic lower bound of f(n).
- We write f(n)?(g(n))
- It reads f(n) is omega of g(n).
f(n)
c g(n)
n0
37Example of Asymptotic Lower Bound
g(n)n2
- g(n)/4 n2/4
- n2/2 n2/4
- ? n2/2 9 for all n ? 6
- lt n2/2 7
- Thus, f(n) ?(g(n)).
f(n)n2/2-7
c g(n)n2/4
4
38Example Omega
- Example n 1/2 W( lg n) .
- Use the definition with c 1 and n0 16.
Checks OK. - Let n gt 16 n 1/2 (1) lg n if and only if n
( lg n )2 - by squaring both sides.
- This is an example of polynomial vs. log.
39BIG THETA
- A function f which is at least linear growth, is
also at least quadratic growth and also at least
cubic growth, etc. However, we want to find the
"slowest" class of functions whose growth rate is
the same as that of function f.
40Big Theta Notation
- Definition Two functions f and g are said to be
of equal growth, f Big Theta(g) if and only if
both - fQ(g) and g Q(f).
- Definition f(n) O(g(n)) means positive
constants c1, c2, and n0 such that - c1 g(n) ? f(n) ? c2 g(n) ? n ? n0
- If f(n) O(g(n)) and f(n) W(g(n)) then f(n)
Q(g(n)) - (e.g. f(n) n2 and g(n) 2n2)
41Theta, the Asymptotic Tight Bound
- Theta means that f is bounded above and below by
g BigTheta implies the "best fit". - f(n) does not have to be linear itself in order
to be of linear growth it just has to be between
two linear functions,
42Asymptotically Tight Bound
- f(n) O(g(n)) and f(n) ?(g(n))
- g(n) is called an
- asymptotically tight bound of f(n).
- We write f(n)?(g(n))
- It reads f(n) is theta of g(n).
c2 g(n)
f(n)
c1 g(n)
n0
43USING LIMIT OF RATIO TO DEFINE ASYMPTOTICS
- Definition Let f and g be functions such that
the limit of the ration f(n)/g(n) exists (but is
possibly infinite) as n approaches infinity. Call
this limit a. Then - f o(g) if a 0
- f O(g) if a is finite (a gt 0)
- f Q(g) if a is strictly positive and finite
- f W(g) if a is strictly positive and possibly
infinite - Note L'Hopital's Rule may be needed
-
44Other Asymptotic Notations
- A function f(n) is o(g(n)) if ? positive
constants c and n0 such that f(n) lt c g(n) ? n
? n0 - A function f(n) is ?(g(n)) if ? positive
constants c and n0 such that c g(n) lt f(n) ? n
? n0 - Intuitively,
- o() is like lt
- O() is like ?
- ?() is like gt
- ?() is like ?
45THEOREMS USING BIG O
- 1. If T1 O(f), T2 O(g) where all functions
are from N to the positive reals, then - (a) T1 T2 max(O(f),O(g))
- (b) T1 T2 O(f g)
- 2. If T(x) is a polynomial of degree n, then T(x)
O(xn) - 3. Logkn O(n) for any constant k
- 4. If f(n) is in O(g(n)) and g(n) is in O(h(n))
then f(n) is in O(h(n)) - 5. If f(n) is in O(k g(n)) for any constant kgt0
then f(n) is in O(g(n))
46Asymptotic Notation Properties
- Transitivity
- Reflexivity
- Symmetry
- Transpose symmetry
47Transitivity
- f(n) ?(g(n)) and g(n) ?(h(n)) imply f(n)
?(h(n)) - f(n) O(g(n)) and g(n) O(h(n)) imply f(n)
O(h(n)) - f(n) ?(g(n)) and g(n) ?(h(n)) imply f(n)
?(h(n)) - f(n) o(g(n)) and g(n) o(h(n)) imply f(n)
o(h(n)) - f(n) ?(g(n)) and g(n) ?(h(n)) imply f(n)
?(h(n))
48Reflexivity
- f(n) ?(f(n))
- f(n) O(f(n))
- f(n) ?(f(n))
49Symmetry and Transpose Symmetry
- f(n) ?(g(n)) iff g(n) ?(f(n))
- f(n) O(g(n)) iff g(n) ?(f(n))
- f(n) o(g(n)) iff g(n) ?(f(n))
50Manipulating Asymptotic Notation
- cO(f(n)) O(f(n))
- O(O(f(n))) O(f(n))
- O(f(n))O(g(n)) O(f(n)g(n))
- O(f(n)g(n)) f(n)O(g(n))
- O(f(n)g(n)) ( max ( f(n),g(n))
51USING BIG O
- O(g(n)) denotes the set of all functions f(n)
such that - f(n) lt c g(n), for some constant c gt 0.
- Thus, "f(n) is O(g(n))" means that f(n) is a
member of the set of functions O(g(n)). With this
meaning, we can write such statements as n2 O(
n ) is O( n2 ) - That is, for all c gt 0, there is a constant d gt 0
such that - n2 cn lt dn2 , for all n gt 0
52Examples
- 1. 2n3 3n2 n 2n3 3n2 O(n)
- 2n3 O( n2
n) 2n3 O( n2 ) - O(n3 ) O(n4)
- 2. 2n3 3n2 n 2n3 3n2 O(n)
- 2n3 Q(n2 n)
- 2n3 Q(n2)
Q(n3)
53Examples (cont.)
- 3. Suppose a program P is O(n3), and a program Q
is O(3n), and that currently both can solve
problems of size 50 in 1 hour. If the programs
are run on another system that executes exactly
729 times as fast as the original system, what
size problems will they be able to solve?
54Example (cont.)
- n3 503 729 3n
350 729 - n n log3 (729 350)
- n n log3(729) log3 350
- n 50 9 n 6 log3
350 - n 50 9 450 n 6
50 56 - Improvement problem size increased by 9 times
for n3 algorithm but only a slight improvement in
problem size (6) for exponential algorithm.
55More Examples
- (a) 0.5n2 - 5n 2 W( n2). Let c 0.25 and n0
25. - 0.5 n2 - 5n 2 0.25( n2) for
all n 25 - (b) 0.5 n2 - 5n 2 O( n2). Let c 0.5 and n0
1. - 0.5( n2) 0.5 n2 - 5n 2 for all n 1
- (c) 0.5 n2 - 5n 2 Q( n2) from (a) and (b)
above. - Use n0 25, c1 0.25, c2 0.5 in the
definition.
56More Examples
- (d) log 100/97 n O(n) . (Notation lg n log2
n) - Try this one on your own.
- (e) It is okay to say that 2 n2 3n - 1 2 n2
Q(n). - Recall the mergesort work with T(n)
2T(n/2) Q(n). - (f) 6 2n n2 W(2n). Let c 1 and n0 1.
- (g) 3n 3 Q(n2). Why? It is not W( lg n)
- There exists no positive c and n0 such that
3n 3 c n2 for n n0
57More Examples
- (h) 6 2n n2 O(2n). Let c 7 and n0 4.
- Note that 2n n2 for n 4. Not a tight
upper bound, but it's true. - (i) 10 n2 2 O(n4).
- There's nothing wrong with this, but usually
we try to get the closest g(n). Better is to use
O(n2 ).
58More examples
- 2n gt n2 is an example of exponential vs.
polynomial complexity. - Claim that this is true for all n gt 16. Proof by
induction - base 216 65,536 162
- induction hypothesis Assume 2n - 1 (n -
1)2 . WTS 2n n2 - 2n 2(2n - 1) 2 (n - 1)2 by the induction
hypothesis - 2 n2 - 4n 2 n2 (n2 - 4n 2) gt
n2 if n2 - 4n 2 gt 0 - n2 - 4n 2 (n - 2)2 - 2 gt 0 when (n -
2)2 gt 2, true since n gt 16.
59How to show a function f(n) is in/not in O(g(n)).
- f(n) is in O(g(n))
- find a constant c and large n0 such
that - for all n gt n0, f(n) lt c g(n).
- f(n) is not in O(g(n))
- for any constant c and any large n0,
- we can find a m such that
- f(m) gt c g(m).
- Usually it is more difficult to proof that
- a function f(n) is not in the big-O of another
- function g(n).
c,n0 ngtn0 s.t. f(n)lt c g(n) c,n0
ngtn0 s.t. f(n)gt c g(n)
60Example
- Example n2 ? O(nlogn)
- Proof
- For any c and n, Let n1 c.2n
- n1. n2 n1 . c.2n
- gt n1 . cn
- cn1 log n1
61Example n log n ? O(log n!)
- n log n log n log n log n .
log n - log n! log n log (n-1) log (n-2)
log 1. - (1/2) n log n lt log n log n
log n - 2 log n! 2 log n 2 log (n-1) 2
log n/2 2 log 1.
62Theorem 1
- Theorem 1 log n lt n for all n gt 1
- Proof by induction
- n 1 , log 1 0 lt 1 TRUE
- Assume for all k lt (n 1) that log k lt k
- WTS (Want To Show) log(n 1) lt n 1
- log (n 1) lt log( n n) log 2n log2
log n 1 log n lt n 1 - Corollary log n O(n) ( i.e., n dominates
lg n)
63Theorem 2
- Theorem 2 The base of a logarithm doesn't
matter for - asymptotics. (logan and logbn differ
only by a constant (not dependent on
n). - Proof Recall x ab iff logax b. These are
inverse functions. - Let y loga n. Then ay n and logb n
logb ay y logb a. - Thus logb n (loga n) ( logb a)
constant loga n - Corollary logan O(n)
- Proof logan O(lg n) from Theorem 2 and
O(n) from - Theorem 1.
64Theorem 3
- Theorem 3 Exponentials dominate (grow faster)
than polynomials - Proof Given any polynomial of degree a, we wish
to compare - na/cn where c gt1
- Note limit as n ? infinity of na/cn is 8/8 . Use
L'Hospitals rule a times and the top goes to 0
while the bottom keeps the term cn . Since the
limit is 0, use the definition of limit to get an
n0 such that n? n0 na/cn lt 1. or na lt cn .
65Big O Again!!!!
- O(1) The cost of applying the algorithm can be
bounded independently of the value of n. This is
called constant complexity. - O(log n) The cost of applying the algorithm to
problems of sufficiently large size n can be
bounded by a function of the form k log n, where
k is a fixed constant. This is called logarithmic
complexity. - O(n) linear complexity
- O(n log n) n lg n complexity
- O(n2) quadratic complexity
66Big O Again!!!!
- O(n3) cubic complexity
- O(n4) quartic complexity
- O(n32) polynomial complexity
- O(cn) If constant c 1, then this is called
exponential complexity - O(2n) exponential complexity
- O(en) exponential complexity
- O(n!) factorial complexity
- O(nn)
67Practical Complexity t lt 250
68Practical Complexity t lt 500
69Practical Complexity t lt 1000
70Practical Complexity t lt 5000
71Practical Complexity
72Other Logarithmic Running Time Algorithms
- An algorithm is O(log n) if it takes constant
(O(1)) time to cut the problem size by a fraction
(usually 1/2). - If constant time is needed to reduce the problem
by a constant amount (i.e. make smaller by 1),
then algorithm is O(n).
73THINGS TO REMEMBER IN ANALYSIS
- Constants or low-order terms are ignore
- e.g. if f(n) 2n2 then f(n) O(n2)
- Most important resource to analyze is running
time other factors are algorithm used and input
to the algorithm - Parameter N, usually referring to number of data
items processed, affects running time most
significantly. - N may be degree of polynomial, size of file
to be sorted or searched, number of nodes in a
graph, etc.
74THINGS TO REMEMBER IN ANALYSIS
- Worst case is amount of time program would take
with worst possible input configuration - worst case is bound for input and easier to find
usually the metric chosen - Average case is amount of time a program is
expected to take using "typical" input data - definition of "average" can affect results
- average case is much harder to compute
75GENERAL RULES FOR ANALYSIS(1/5)
- 1. Consecutive statements
- Maximum statement is the one counted
- e.g. a fragment with single for-loop followed by
double for- loop is O(n2).
t1t2 max(t1,t2)
76GENERAL RULES FOR ANALYSIS(2/5)
- 2. If/Else
- if cond then S1
- else
- S2
Max(t1,t2)
77GENERAL RULES FOR ANALYSIS(3/5)
- 3. For Loops
- Running time of a for-loop is at most the running
time of the statements inside the for-loop times
number of iterations - for (i sum 0 i lt n
i) - sum ai
- for loop iterates n times, executes 2 assignment
statements each iteration gt asymptotic
complexity of O(n)
78GENERAL RULES FOR ANALYSIS(4/5)
- 4. Nested For-Loops
- Analyze inside-out. Total running time is running
time of the statement multiplied by product of
the sizes of all the for-loops - e.g. for (i 0 i lt n i)
- for (j 0, sum a0 j lt i j)
- sum aj
- printf("sum for subarray - through d is d\n",
i, sum)
79GENERAL RULES FOR ANALYSIS(5/5)
80GENERAL RULES FOR ANALYSIS
- Strategy for analysis
- analyze from inside out
- analyze function calls first
- if recursion behaves like a for-loop, analysis is
trivial otherwise, use recurrence relation to
solve