Introduction in Computer Science 2 Asymptotic Complexity

1
Introduction in Computer Science 2Asymptotic
Complexity
DEEDS Group - TU Darmstadt Prof. Neeraj
Suri Constantin Sarbu Brahim Ayari Dan
Dobre Abdelmajid Khelil
2
Remember Sequential Search
Given Array A of integers and a constant
c. Question Is c in A?

• Memory Complexity (in Java)
• int 4 bytes,
• boolean 1 byte
• Memory used
• size(A) size(c) size(n)
• size(i) size(found)
• n413

boolean contains (int A, int c) int n
A.length boolean found false for(int i
0, i lt n i) if (A i c) found
true return (found)
Time Complexity counting operations
Input A c n Assignment Comparisons
Array access Increments 1,4,2,7 6 4
1110 44 4 4 2,7,6,1 2
4 1111 44 4
4 2,1,8,4,19,7,16,3 5 8 1110 88
8 8 4,4,4,4,4,4 4 6
1116 66 6 6
3
Why Asymptotic Complexity?

• Time complexity
• gives a simple characterization of an algorithms
  efficiency
• allows to compare it to alternative algorithms
• In the last lecture we determined exact running
  time, but extra precision usually doesnt worth
  the effort of computing it
• Large input sizes constants and lower order
  terms are ruled out
• This means we are studying asymptotic complexity
  of algorithms
• ? we are interested in how the running time
  increases with the size of the input in the limit
• Usually, an algorithm which is asymptotically
  more efficient is not the best choice for very
  small inputs )

4
Today Efficiency Metrics - Complexity

• Upper Bounds O (big O) - Notation
• Properties, proof f ? O(g), sum and product rules
• Loops, conditional statements, conditions,
  procedures
• Examples Sequential search, selection sort
• Lower Bounds ? (Omega) Notation
• Bands ? (Theta) - Notation

5
Asymptotic Time Complexity Upper Bound
c g(n)
T(n)
f(n)
n
n0
c gt 0 ? n0 ? ? n gt n0 cg(n) gt f(n)
6
O-Notation (pronounce big-Oh)

• Given f N ? R g N ? R
• Definition
• O(g) f ?n0?N, c?R, c gt 0 ?n ? n0 f(n) ?
  cg(n)
• Intuitively
• O(g) the set of all functions f, that grow,
  at most, as fast as g
• One says
• If f ? O(g), then g is an asymptotical upper
  bound for f

7
Example

• O(n4) , n, n2, nlogn, n3, n4, 3n4, cn3,
• n3 ? O(n4)
• nlogn ? O(n4)
• n4 ? O(n4)
• Generally slower growth ? O (faster growths)

8
O-Notation

• Often shortened as f O(g) instead of f ? O(g)
• But f O(g) is no equality in the common
  meaning, only interpretable from left to right!
• Normally, for analysis of algorithms
• f N ? N and g N ? N,
• since the input is the size of the input data and
  the value is the amount of elementary operations
• For average case analysis the set R is also
  used
• f N ? R and g N ? R

9
Example O-Notation

• T1(n) n 3 ? O(n) because n 3 ? 2n ?n ? 3
• T2(n) 3n 7 ? O(n)
• T3(n) 1000n ? O(n)
• T4(n) 695n2 397n 6148 ? O(n2)
• Functions are mostly monotonically increasing and
  ? 0.
• Criteria for finding f ? O(g)
• If f(n) / g(n) ? c for some n ? n0 then f
  O(g)
• Example

n0
c
lim f(n) / g(n) ? c
n ??
n2
10
Proving that f ? O(g)

• The proof has two parts
• Finding the closed form
• Solving the inequality f(n) ? c.g(n) from the
  definition
• Illustration using an example
• A is an algorithm, which sorts a set of numbers
  in increasing order
• Assumption A performs according to f(n) 3 6
  9 ... 3n
• Proposition A has the complexity O(n2)
• Closed form for f(n) 3 6 9 ... 3n
• f(n) 3(123...n) 3n(n1)/2

11
Proving that f ? O(g)

• Task Find a value c, for which 3n(n1)/2 ?
  cn2 (for n gt one n0)
• Try c3 3n(n1)/2 ? 3n2 n2 n ? 2n2 n ?
  n2 1 ? n for all n ? 1 Q.E.D.

12
Consequences of the O-Notation

• O-Notation is a simplification
• It eliminates constants O(n) O(n/2) O(17n)
• It forms an upper bound, i.e.
• from f(n) ? O(n log2n) follows that f(n) ? O(n2)
• For O-Notation the basis for logarithms is
  irrelevant, as

13
Properties of O-Notation

• Inclusion relations of the O-Notation O(1) ?
  O(log n) ? O(n) ? O(n log n) ? O(n2) ? O(n3)
  ? O(2n) ? O(10n)
• ? We try to set the bounds as tight as possible
• Rule

14
Pronunciation
15
Calculating the Time Complexity

• The time complexity of a program comes from the
  complexity of its parts
• The complexity of the elementary operations is
  O(1) (elementary operation e.g. assignment,
  comparison, arithmetic operations, array
  access, )
• A defined sequence of elementary operations
  (independent of the input size n) also has the
  complexity O(1)

16
Sum and Product Rules

• Given the time complexities of two algorithms T1
  and T2
• Summation rule For the execution of T1 followed
  by T2
• Product rule For the nested execution of T1 and
  T2

17
Loops in Series

• Loops in series (n and m are the problem
  sizes)for (int i 0 i lt n i) operation
  
• for (int j 0 j lt m j) operation
• Complexity O(nm) O(max(n,m)) (sum rule)

18
Nested Loops

• Nested loops (n is the problem size)
• When inner loop execution is not dependent on the
  problem size, e.g.for (int i 0 i lt n i)
  for (int j 0 j lt 17 j)
  operationComplexity O(17n) O(n) (Product
  rule)
• Otherwise for (int i 0 i lt n i) for
  (int j 0 j lt n j) operationComplexity
  
• (Product rule)
• Ex read the data from a n x n matrix -gt very
  expensive (O(n2))!

19
Conditional Statement

• Conditional Statement if B then T1
• else T2
• Cost of if is constant, therefore negligible
• T(n)T1(n) or T(n)T2(n)
• Good (if decidable) Longer sequences are chosen,
  i.e., the dominant operation should be used
• Upper boundary assessment also possible
• T(n) lt T1(n) T2(n) ? O(g1(n)g2(n))

20
Condition Example

• Loop with condition (n is the problem size)for
  (int i 0 i lt n i) if (i
  0) block1 else block2
• block1 is executed only once gt not relevant
• (when not T(block2) gtgt n.T(block1) )
• block2 is dominant
• Complexity O(n.T(block2))

21
Procedure Calls

• Procedures are analyzed separately, and their
  execution times inserted for each call
• For recursive procedure calls a recurrence
  relation for T(n) must be found
• Once again Find a closed form for the
  recursive relation (example follows shortly)

22
Analysis of simple Algorithms

• Iterative Algorithms (today)
• Composed of smaller parts ? sum rule
• Consider loops ? multiplication rule
• Recursive Algorithms (next lecture)
• Time factors
• Breaking a problem in several smaller ones
• Solving the sub-problems
• Recursive call of the method for solving the
  problem
• Combining the solutions for the sub-problems

23
Example 1 Sequential Search
boolean contains (int A, int c) int n
A.length boolean found false for(int i
0, i lt n i) if (A i c) found
true return (found)

• Cost consists of part a, and a part b which is
  repeated n times
• T(n) abn
• T(n) ? O(n)

24
Example 2 Selection Sort
void SelectionSort (int A) int
MinPosition, temp, i, j for (in-1 igt0 i--)
MinPosition i for (j0 jlti j) if (
Aj lt AMinPosition ) MinPosition j temp
Ai Ai AMinPosition AMinPosition
temp
c
b

• Inner loop is executed i times, i lt n gt upper
  boundary c.n
• Outer loop is executed n times, constant costs b
  
• Costs n.(bcn) bn cn2 gt O(n2)

25
? (Omega) - Notation

• Analog to O(f) we have
• ?(g) h ? cgt0 ? ngt0 ?ngtn h(n) ? c g(n)
  
• Intuitively
• ?(g) is the set of all functions that grow at
  least as strong as g
• One says
• if f ? ?(g), then g sets a lower bound for f.
• Note f ? O(g) ? g ? ?(f)

26
Example ?-Notation
T(n)
f(n)
c2 g(n)
n
n0
c2, n0 gt 0 such that f(n) ? (g(n))
g(n) sets an lower bound for f(n)
27
? (Theta) - Notation

• With the sets O(g) and ?(g) we can define
• ?(g) O(g) ? ?(g)
• Intuitively
• ?(g) is the set of functions that grow exactly
  as strong as g
• Meaning if f ? O(g) and f ? ?(g) then f?? ?(g)
• In this case one talks about an exact bound

28
Example ?-Notation
T(n)
c1 g(n)
f(n)
c2 g(n)
n
n0
c1, c2, n0 gt 0 such that f(n) ? (g(n))
g(n) sets an exact bound for f(n)
29
Non-Asymptotic Execution Time

• Algorithms with a higher asymptotic complexity
  can be more efficient for smaller problem sizes
• Asymptotic execution time only holds for certain
  values of n
• The constants do make a difference for smaller
  input sets

30
Complexity and Recursion

• Up till now, weve seen only iterative algorithms
• What about recursive algorithms?
• Following week Refreshing recursion
• Then Complexity Analysis with recurrence
  relation