EE441 Data Structures Chapter V Complexity of Algorithms

1
EE441 Data StructuresChapter VComplexity of
Algorithms

• Özgür B. Akan
• Department of Electrical Electronics
  Engineering
• Middle East Technical University
• akan_at_eee.metu.edu.tr
• www.eee.metu.edu.tr/akan

2
Algorithms

• An algorithm is a computable set of steps to
  achieve a desired result.
• Not sufficient that algorithms perform the
  required tasks.
• They should do so efficiently, making the best
  use of
• Space
• Time

Efficiency is important!!!
3
Efficiency in Time and Space

• Time
• Instructions take time
• How fast does the algorithm perform?
• What affects its run-time?
• Space
• Data structures take space
• What kind of data structures can be used?
• How does the choice of data structure affect the
  runtime?

4
Objectives of Complexity Analysis

• Two algorithms that accomplish the same task
• Which one is better???
• Given an algorithm, is it possible to determine
  how long it will take to run?
• Input is unknown
• Do not want to trace all possible execution paths
• For different input, is it possible to determine
  how an algorithms runtime changes?

5
Measuring Growth of Work

• While it is possible to measure the work done by
  an algorithm for a given set of input, we need a
  way to
• Measure the rate of growth of an algorithm based
  upon the size of the input
• Compare algorithms to determine which is better
  for the situation

6
Sequential Search Algorithm

• e.g. Search an n-element array for a match with
  a given key return the array index of the
  matching element or -1 if no match is found
• int SeqSearch(DataType list , int n, DataType
  key)
• // note DataType must be defined earlier
• // e.g., typedef int DataType
• // or typedef float DataType etc.
• for (int i0 iltn i)
• if (listikey)
• return i
• return 1
• Here, at worst case n comparisons
  (operations) performed
• expected (average) n/2 comparisons
• expected computation time ? n

7
Sequential Search Algorithm

• e.g., if the algorithm takes 1 ms with 100
  elements
• it takes 5 ms with 500 elements 200ms
  with 20000 elements etc.
• Another approach Binary Search Algorithm
• (Using a sorted array)

8
Binary Search Algorithm

• int BinarySearch(DataType list, int low, int
  high, DataType key)
• int mid
• DataType midvalue
• while (lowlthigh)
• mid(lowhigh)/2 // note integer division,
  middle of array
• midvaluelistmid
• if (keymidvalue) return mid
• else if (keyltmidvalue) highmid-1
• else lowmid1
• return 1

9
Binary Search Algorithm

• e.g. int list55,17,36,37,45
• low0, high4 key44
• 1) mid(04)/22 3) mid(44)/24
• midvaluelist236 midvaluelist445
• keygtmidvalue keyltmidvalue
• lowmid13 highmid-13
• 2) mid(34)/23 4) since high3ltlow4, exit the
  loop
• midvaluelist337 return -1 (not found)
• keygtmidvalue
• lowmid14

10
Binary Search Algorithm

• e.g.
• int list55,17,36,37,45
• low0, high4 key5
• (the same example with different key)
• 1) mid(04)/22 2) mid(01)/20
• midvaluelist236 midvaluelist05
• keyltmidvalue keymidvalue
• highmid-11 return 0 (found)

11
Binary Search Algorithm

• In the worst case, Binary Search makes ?log2n ?
  comparisons
• e.g. n log2n
• 8 3
• 20 5
• 32 5
• 100 7
• 128 7
• 1000 10
• 1024 10
• 64000 16
• 65536 16

(ceil) Smallest integer larger than or equal to
e.g. if Binary Search takes 1msec for 100
elements, it takes tk ?log2n ? 1mseck
?log2 100? k1/7 msec/comparison Hence,
t(1/7) ?log2n ? t500(1/7)
?log2500?9/7?1.29msec t20000(1/7)
?log220000?15/7?2.1msec
12
Binary Search Algorithm

• Therefore,

13
Computational Complexity

• Compares growth of two functions
• Independent of constant multipliers and
  lower-order effects
• Metrics
• Big-O Notation O( )
• Big-Omega Notation ?( )
• Big-Theta Notation ?( )
• Allow us to evaluate algorithms
• Has precise mathematical definition
• Used in a sense to put algorithms into families
• May often be determined by inspection of an
  algorithm

14
Definition Big-O Notation

• Function f(n) is O(g(n)) if there exists a
  constant K and some n0 such that
• f(n)Kg(n) for all nn0
• i.e., as n??, f(n) is upper-bounded by a
  constant times g(n).

• Usually, g(n) is selected among
• log n (note loganklogbn for any a,b??)
• n, nk (polynomial)
• kn (exponential)

15
Big-O Notation
Kg(n)
Upper Bound
f(n)
f(n) is O(g(n))
Work done
Our Algorithm
n0
Size of input
16
Comparing Algorithms

• The O() of algorithms determined using the formal
  definition of O() notation
• Establishes the worst they perform
• Helps compare and see which has better
  performance

17
Big-O (Examples)

• e.g. f(n)n2250n106 is O(n2)
• because
• f(n)n2n2n2 for n 103
• 3n2

n0
K
e.g. f(n)2n1023n?n is O(2n) because
1023nlt2n for ngtn0 and ?n lt2n ?n f(n)32n
for ngtn0
K
18
Correct Interpretation of O()

• O(1) or Order One
• Does not mean that it takes only one operation
  
• Does mean that the work does not change as n
  changes
• Is notation for constant work
• O(n) or Order n
• Does not mean that it takes n operations
• Does mean that the work changes in a way that is
  proportional to n
• Is a notation for work grows at a linear rate

19
Big-Omega Notation

• Function f(n) is ?(g(n)) if there exists a
  constant K and some n0 such that
• Kg(n) f(n) for all nn0
• i.e., as n??, f(n) is lower-bounded by a
  constant times g(n).

20
Big-Theta Notation

• Function f(n) is ?(g(n)) if there exist
  constants K1 and K2 and some n0 such that
• K1g(n) f(n) K2g(n) for all nn0
• i.e., as n??, f(n) is upper and lower bounded by
  some constants times g(n).

K2g(n)
f(n)
f(n) is ?(g(n))
Work done
K1g(n)
n0
Size of input
21
Examples

• 3n2 17
• ?(1), ?(n), ?(n2)
• O(n2), O(n3),
• ?(n2)

22
Analogous to Real Numbers

• f(n) O(g(n)) (a lt b)
• f(n) ?(g(n)) (a gt b)
• f(n) ?(g(n)) (a b)