MS 101: Algorithms - PowerPoint PPT Presentation

About This Presentation
Title:

MS 101: Algorithms

Description:

Insertion Sort takes advantage if the list is already partially ... Needless to say those algorithms are not purely based on ... (arrays or linked list), ... – PowerPoint PPT presentation

Number of Views:80
Avg rating:3.0/5.0
Slides: 93
Provided by: Cs
Category:

less

Transcript and Presenter's Notes

Title: MS 101: Algorithms


1
MS 101 Algorithms
  • Instructor
  • Neelima Gupta
  • ngupta_at_cs.du.ac.in

2
Table Of Contents
  • Introduction to some tools to designing
    algorithms through Sorting
  • Iterative
  • Divide and Conquer

3
Iterative Algorithms Insertion Sort an example
  • x1,x2,........., xi-1,xi,.......,xn
  • For I 2 to n
  • Insert the ith element xi in the partially
    sorted list x1,x2,........., xi-1.

(at rth position)
4
An Example Insertion Sort
15
8
7
10
12
5
At Iteration 1 key 8
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
5
An Example Insertion Sort
8
15
7
10
12
5
At Iteration 2 key 7
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
6
An Example Insertion Sort
7
8
15
10
12
5
At Iteration 3 key 10
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
7
An Example Insertion Sort
7
8
10
15
12
5
At Iteration 4 key 12
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
8
An Example Insertion Sort
7
8
10
12
15
5
At Iteration 5 key 5
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
9
An Example Insertion Sort
5
7
8
10
12
15
Final Output
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
10
Analysis Insertion Sort
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j -
1 Aj1 key
11
Running Time Analysis
Statement C N
InsertionSort(A, n) for i 2 to n
c1 n key Ai c2 (n-1) j i -
1 c3 (n-1) while (j gt 0) and (Aj gt
key) c4 S(Ti 1) Aj1
Aj c5 S Ti j j - 1 c6 S
Ti Aj1 key c7 (n-1)
where Ti is number of while expression
evaluations for the ith for loop iteration Ci is
the constant time required for 1 execution of the
statement N is the number of times the statement
is executed
12
Total time
  • T(n) (c1 c2 c3 c7 )n (c2 c3 c7)
    (c4 c5 c6) Ti c4

n
?
i2
13
Worst Case
Worst case Ti i 1 i.e. Ti
(i 1) n(n-1)/2 hence, T(n) (c1 c2
c3 c4 c7 )n (c2 c3 c4 c7) (c4 c5
c6)n(n-1/2) an2 bn c where, a
1/2 (c4 c5 c6) b -1/2 (c4 c5
c6) (c1 c2 c3 c4 c7 ) c -(c2 c3
c4 c7)
n
n
?
?
i2
i2
14
Best Case
Best case Ti 1 i.e. Ti
1 (n 1) hence, T(n) (c1 c2 c3 c4
c7 )n (c2 c3 c4 c7) (c4 c5
c6)(n-1) an b where, a c1
c2 c3 2c4 c5 c6 c7 b -(c2 c3
2c4 c5 c6 c7 )
n
n
?
?
i2
i2
15
Analysis of Algorithms
  • Before we move ahead, let us define the notion of
    analysis of algorithms more formally

16
Input Size
  • Time and space complexity
  • This is generally a function of the input size
  • How we characterize input size depends
  • Sorting number of input items
  • Multiplication total number of bits
  • Graph algorithms number of nodes edges
  • Etc

17
Lower Bounds
  • Please understand the following statements
    carefully.
  • Any algorithm that sorts by removing at most one
    inversion per comparison does at least n(n-1)/2
    comparisons in the worst case.
  • Hence Insertion Sort is optimal in this category
    of algorithms.

18
Optimal?
  • What do you mean by the term Optimal?
  • Answer If an algorithm runs as fast in the worst
    case as it is possible (in the best case), we say
    that the algorithm is optimal. i.e if the worst
    case performance of an algorithm matches the
    lower bound, the algorithm is said to be optimal

19
Inversion -
  • Example 4, 2, 3
  • No. of pairs 3C2 , out of which
  • (4,2) is out of order i.e. inversion
  • (2,3) is in order
  • (4,3) inversion

20
  • In n elements there will be n(n-1)/2 inversions
    in worst case.
  • Thus, if an algorithm sorts by removing at most
    one inversion per comparison then it must do at
    least n(n-1)/2 comparisons in the worst case.

21
  • Insertion Sort
  • x1,x2,...,xk-1, xk,.............,xi-1,xi
  • Let xi is inserted after xk-1
  • No. of comparisons (i-1) (k 1) 1 i k
    1
  • No. of inversions removed (i-1) (k 1) i
    k
  • No. of inversions removed/comparison
  • (1-k1)/(i-k)
  • lt 1

22
  • Thus Insertion sort falls in the category of
    comparison algorithms that remove at most one
    inversion per comparison.
  • Insertion sort is optimal in this category of
    algorithms .

23
SELECTION SORT
  • The algorithm works as follows
  • Find the maximum value in the array.
  • Swap it with the value in the last position
  • Repeat the steps above for the remainder of the
    array.

24
Selection Sort An Example
  • For i n to 2
  • Select the maximum in a1.......ai.
  • b) Swap it with ai.

25
  • Selection Sort
  • x1,x2,, xk,..........., xn
  • Let the maximum is located at xk
  • Swap it with xn
  • Continue

26
  • Selection Sort
  • x1,x2,, xk,..........., xn-i1,xn
  • Suppose we are at the ith iteration
  • Let the maximum is located at xk
  • Swap it with xn-i1

27
An Example Selection Sort
5 20 6 4 15 3 10 2
28
An Example Selection Sort
5 2 6 4 15 3 10 20

29
An Example Selection Sort
5 2 6 4 10 3 15 20
30
An Example Selection Sort
5 2 6 4 3 10 15 20
31
An Example Selection Sort
5 2 3 4 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
32
An Example Selection Sort
4 2 3 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
33
An Example Selection Sort
3 2 4 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
34
An Example Selection Sort
2 3 4 5 6 10 15 20
DONE!!!!
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
35
ANALYSISSELECTION SORT
  • SelectionSort(A, n)
  • for i n to 2 maxi
    for ji-1 to 1
  • If ajgtamax
  • Maxj
  • If(max!i)
  • Swap aiamax

36
ANALYSIS SELECTION SORT
  • Selecting the largest element requires scanning
    all n elements (this takes n - 1 comparisons)
  • and then swapping it into the last position.
    Finding the next largest element requires
    scanning the remaining n - 1 elements and so
    on...
  • (n - 1)  (n - 2)  ...  2  1  n(n - 1) / 2
  • i.e T(n2) comparisons
  • T(n) T(n2)

37
  • Selection Sort
  • x1,x2,, xk,..........., xn-i1,xn
  • Suppose we are at the ith iteration
  • Let the maximum is located at xk
  • No. of comparisons (n i 1) - 1 n - i
  • No. of inversions removed (n i 1) 1
    (k-1) n i k 1
  • No. of inversions removed/comparison
  • ( n i k 1)/ (n i)
  • lt 1 (since k gt 1)

38
  • Thus Selection sort also falls in category of
    comparison algorithms that remove at most one
    inversion per comparison.
  • Selection sort is also optimal in this category
    of algorithms .

39
Merge Sort(Divide and Conquer Technique)
  • Divide the list into nearly equal halves
  • Sort each half recursively
  • Merge these lists

40
18 9 15 13 20 10 7 5
Divide into 2 Subsequence
18 9 15 13
20 10 7 5
18 9
15 13
20 10
7 5
18
9
15
13
20
10
7
5
41
Next Sort the 2 subsequences and merge them.
42
Sorted Sequence
5 7 9 10 13 15 18 20
Sort merge 2 Subsequence
9 13 15 18
5 7 10 20
9 18
13 15
10 20
5 7
18
9
15
13
20
10
7
5
Initial Subsequence
43
Merging
  • Let we have two sorted lists
  • A
  • B
  • Compare a1 with b1 and put smaller one in new
    array C and increase the index of array having
    smaller element.

a1
a2
a3
.
an
b1
b2
b3
.
bm
44
  • Let a1 lt b1
  • A
  • B
  • Sorted Array C
  • C

a1
a2
a3
.
an
b1
b2
b3
.
bm
a1
45
  • Example 1
  • A
  • B
  • Sorted Array C
  • C

20
40
60
80
10
30
35
38
45
50
52
10
20
46
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
Thanks toGaurav Gulzar (MCA -11)
47
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
Thanks toGaurav Gulzar (MCA -11)
48
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
Thanks toGaurav Gulzar (MCA -11)
49
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
Thanks toGaurav Gulzar (MCA -11)
50
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
Thanks toGaurav Gulzar (MCA -11)
51
Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
60
80
Thanks toGaurav Gulzar (MCA -11)
52
Worst Case Analysis
  • If no. of elements in first list is n in
    second list is m then
  • Total No. of comparisons n-1m
  • i.e. O(m n) in worst case and

53
What is the best case?
  • Number of Comparisons in the best case
  • min(n , m), i.e. O(min(n , m))

54
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
55
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
56
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
57
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
58
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
59
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
60
  • Example 2
  • A
  • B
  • Sorted Array C
  • C

5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
52
61
What is the best case?
  • Arrays
  • Total number of steps O(mn)copying the rest
    of the elements of the bigger array.
  • Linked List
  • Total number of steps
  • min(n , m), i.e. O(min(n , m)) in best case.

62
Analysis of Merge Sort
  • Since size of both the lists to be merged is
    n/2, in either case (arrays or linked list), time
    to merge the two lists is T(n).
  • If T(n) no. of comparisons performed on an
    Input of size n then
  • T(n) T(n/2) T(n/2) T(n)
  • 2T(n/2) cn ? ngt n0
  • ? T(n) O(nlogn)

63
  • If T(n) no. of comparisons performed on an
    Input of size n then
  • T(n) T(n/2) T(n/2) T(n)
  • 2T(n/2) cn ? ngt n0
  • ? T(n) O(nlogn)

64
Final Points
  • Merging is T(m n) in case of an Array
  • If we use link list then it is O(min(m , n))
  • Time for merging is O(n) and O(n/2) i.e. T(n)

65
Conclusion
  • Merge Sort T(nlogn)
  • Have we beaten the lower bound?
  • No,
  • It just means that merge sort does not fall in
    the previous category of algorithms. It removes gt
    1 inversions per comparisons.

66
  • What is the
  • Worst case
  • Best Case
  • for Merge Sort?

67
Merge Sort Vs Insertion Sort
  • What is the advantage of merge sort?
  • What is the advantage of insertion sort?

68
Merge Sort Vs Insertion Sort contd..
  • Merge Sort is faster but does not take advantage
    if the list is already partially sorted.
  • Insertion Sort takes advantage if the list is
    already partially sorted.

69
Lower Bound
  • Any algorithm that sorts by comparison only does
    at least ?(n lg n) comparisons in the worst case.

70
Decision Trees
  • Provides an abstraction of comparison sorts. In
    a decision tree, each node represents a
    comparison.
  • Insertion sort applied on x1, x2, x3

x1x2
x1ltx2
x1gtx2
x2x3
x1x3
x2gtx3
x3gtx1
x3gtx2
x1gtx3
x2x3
x3x1
x1ltx2ltx3
x3gtx1gtx2
x2gtx3
x3gtx2
x3gtx1
x3ltx1
x2gtx1gtx3
x2gtx3gtx1
x1gtx2gtx3
x1gtx3gtx2
71
Decision Trees
  • What is the minimum number of leaves in a
    decision tree?
  • Longest path of the tree gives us the height of
    the tree, and actually represents the worst case
    scenario for the algorithm.
  • height of a decision tree O (n log n)
  • i.e. any comparison sort will perform at least
    (n logn) comparisons in the worst case.

72
  • Proof
  • Let h denotes the height of the tree.
  • Whats the maximum of leaves of a binary tree
    of height h? 2h
  • 2h gt number of leaves gt n!
  • (where n no. of elements and n! is a lower
    bound on the no. of leaves in the decision tree)
  • gt hgt log(n!)
  • gt n log n ( By Stirlings Approximation)
  • ( SA n! v (2.p.n) .(n/e)n
  • gt (n/e)n
  • Thus, log(n!) gt n log n n log e gt n logn. )

73
Merge Sort is Optimal
  • Thus the time to comparison sort n elements is
    ?(n lg n)
  • Corollary Merge-sort is asymptotically optimal
    comparison sorts.
  • Later well see another sorting algorithm in this
    category namely heap-sort that is also optimal.
  • Well also see some algorithms which beat this
    bound. Needless to say those algorithms are not
    purely based on comparisons. They do something
    extra.

74
Quick Sort(Divide and Conquer Technique)
  • Pick a pivot element x
  • Partition the array into two subarrays around the
    pivot x such that elements in left subarray is
    less than equal to x and element in right
    subarray is greater than x
  • Recursively sort left subarray and right subarray

x
  x
gtx
75
Quick Sort (Algorithm)
  • QUICKSORT(A, p, q)
  • if p lt q
  • kPARTITION(A, p, q)
  • QUICKSORT(A, p, k-1)
  • QUICKSORT(A, k1, q)

76
Quick Sort (Example)
  • Let we have following elements in our array -
  • i j
  • i j
  • i j

27 14 9 22 8 41 56 31 15 53 99 11 30 24
14 27 9 22 8 41 56 31 15 53 99 11 30 24
14 9 27 22 8 41 56 31 15 53 99 11 30 24
77
Quick Sort (Example)
  • i j
  • i j
  • i j

14 9 22 27 8 41 56 31 15 53 99 11 30 24
14 9 22 8 27 41 56 31 15 53 99 11 30 24
14 9 22 8 27 24 56 31 15 53 99 11 30 41
78
Quick Sort (Example)
  • i j
  • i j
  • i j

14 9 22 8 24 27 56 31 15 53 99 11 30 41
14 9 22 8 24 27 30 31 15 53 99 11 56 41
14 9 22 8 24 27 11 31 15 53 99 30 56 41
79
Quick Sort (Example)
  • i j
  • i j
  • i j

14 9 22 8 24 11 27 31 15 53 99 30 56 41
14 9 22 8 24 11 27 99 15 53 31 30 56 41
14 9 22 8 24 11 27 53 15 99 31 30 56 41
80
Quick Sort (Example)
  • i j
  • i j (stop)
  • (recursive call on left array) (recursive call
    on right array)

14 9 22 8 24 11 27 15 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
81
  • i j i j
  • i j i j
  • i j i j
  • i j i
    j
  • i j
    i j
  • i j i j (stop)
  • i j (st op)

14 9 22 8 24 11 15
53 99 31 30 56 41
27
9 14 22 8 24 11 15
27
53 41 31 30 56 99
9 14 15 8 24 11 22
27
41 53 31 30 56 99
9 14 11 8 24 15 22
27
41 31 53 30 56 99
9 11 14 8 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
41 31 30 53 56 99
27
82
9 11 8 14 24 15 22 27 41 31 30 53 56 99
i j i j i j
i j
9 8 11 14 15 24 22 27 31 41 30 53 56 99
i j i j i
j i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
(stop) i j i j (stop) (stop) i j
i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
Sorted Array
83
Analyzing Quicksort
  • Worst Case?
  • Partition is always unbalanced
  • Worst Case input?
  • Already-sorted input, if the first element is
    always picked as the pivot
  • Best case?
  • Partition is perfectly balanced
  • Best Case input?
  • ?
  • Worst Case and Best Case input when the middle
    element is always picked as the pivot?

84
Worst Case of Quicksort
  • In the worst case
  • T(1) ?(1)
  • T(n) T(n - 1) ?(n)
  • Does the recursion look familiar?
  • T(n) ?(n2)

85
Best Case of Quicksort
  • In the best case
  • T(n) 2T(n/2) ?(n)
  • Does the recursion familiar?
  • T(n) ?(n lg n)

86
Why does Qsort works well in practice?
  • Suppose that partition() always produces a 9-to-1
    split. This looks quite unbalanced!
  • The recurrence is
  • T(n) T(9n/10) T(n/10) n
  • T(n) ? (n log n)
  • Such an imbalanced partition and ?(n log n) time?

87
Why does Qsort works well in practice?
  • Intuitively, a real-life run of quicksort will
    produce a mix of bad and good splits
  • Pretend for intuition that they alternate between
    best-case (n/2 n/2) and worst-case (n-1 1)
  • What happens if we bad-split root node, then
    good-split the resulting size (n-1) node?

88
Why does Qsort works well in practice?
  • Intuitively, a real-life run of quicksort will
    produce a mix of bad and good splits
  • Pretend for intuition that they alternate between
    best-case (n/2 n/2) and worst-case (n-1 1)
  • What happens if we bad-split root node, then
    good-split the resulting size (n-1) node?
  • We end up with three subarrays, size 1, (n-1)/2,
    (n-1)/2
  • Combined cost of splits n n -1 2n -1 O(n)
  • No worse than if we had good-split the root node!

89
Why does Qsort works well in practice?
  • Intuitively, the O(n) cost of a bad split (or 2
    or 3 bad splits) can be absorbed into the O(n)
    cost of each good split
  • Thus running time of alternating bad and good
    splits is still O(n lg n), with slightly higher
    constants
  • How can we be more rigorous? well do average
    analysis of Qsort later while doing randomized
    algorithms.

90
Quicksort Vs Merge Sort
  • Merge Sort takes O(n lg n) in the worst case
  • Quick Sort takes O(n2) in the worst case
  • So why would anybody use Qsort instead of merge
    sort?
  • Because in practice, Qsort is quick as the worst
    case doesnt happen often.

91
Up Next
  • Linear-Time Sorting Algorithms

92
  • The End
Write a Comment
User Comments (0)
About PowerShow.com