Title: MS 101: Algorithms
1MS 101 Algorithms
- Instructor
- Neelima Gupta
- ngupta_at_cs.du.ac.in
2Table Of Contents
- Introduction to some tools to designing
algorithms through Sorting - Iterative
- Divide and Conquer
3Iterative Algorithms Insertion Sort an example
- x1,x2,........., xi-1,xi,.......,xn
-
- For I 2 to n
- Insert the ith element xi in the partially
sorted list x1,x2,........., xi-1.
(at rth position)
4An Example Insertion Sort
15
8
7
10
12
5
At Iteration 1 key 8
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
5An Example Insertion Sort
8
15
7
10
12
5
At Iteration 2 key 7
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
6An Example Insertion Sort
7
8
15
10
12
5
At Iteration 3 key 10
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
7An Example Insertion Sort
7
8
10
15
12
5
At Iteration 4 key 12
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
8An Example Insertion Sort
7
8
10
12
15
5
At Iteration 5 key 5
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
9An Example Insertion Sort
5
7
8
10
12
15
Final Output
1
2
3
4
5
6
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j - 1
10Analysis Insertion Sort
InsertionSort(A, n) for i 2 to n key
Ai j i - 1 while (j gt 0) and (Aj gt
key) Aj1 Aj j j -
1 Aj1 key
11Running Time Analysis
Statement C N
InsertionSort(A, n) for i 2 to n
c1 n key Ai c2 (n-1) j i -
1 c3 (n-1) while (j gt 0) and (Aj gt
key) c4 S(Ti 1) Aj1
Aj c5 S Ti j j - 1 c6 S
Ti Aj1 key c7 (n-1)
where Ti is number of while expression
evaluations for the ith for loop iteration Ci is
the constant time required for 1 execution of the
statement N is the number of times the statement
is executed
12Total time
- T(n) (c1 c2 c3 c7 )n (c2 c3 c7)
(c4 c5 c6) Ti c4
n
?
i2
13Worst Case
Worst case Ti i 1 i.e. Ti
(i 1) n(n-1)/2 hence, T(n) (c1 c2
c3 c4 c7 )n (c2 c3 c4 c7) (c4 c5
c6)n(n-1/2) an2 bn c where, a
1/2 (c4 c5 c6) b -1/2 (c4 c5
c6) (c1 c2 c3 c4 c7 ) c -(c2 c3
c4 c7)
n
n
?
?
i2
i2
14Best Case
Best case Ti 1 i.e. Ti
1 (n 1) hence, T(n) (c1 c2 c3 c4
c7 )n (c2 c3 c4 c7) (c4 c5
c6)(n-1) an b where, a c1
c2 c3 2c4 c5 c6 c7 b -(c2 c3
2c4 c5 c6 c7 )
n
n
?
?
i2
i2
15Analysis of Algorithms
- Before we move ahead, let us define the notion of
analysis of algorithms more formally
16Input Size
- Time and space complexity
- This is generally a function of the input size
- How we characterize input size depends
- Sorting number of input items
- Multiplication total number of bits
- Graph algorithms number of nodes edges
- Etc
17Lower Bounds
- Please understand the following statements
carefully. - Any algorithm that sorts by removing at most one
inversion per comparison does at least n(n-1)/2
comparisons in the worst case. - Hence Insertion Sort is optimal in this category
of algorithms.
18Optimal?
- What do you mean by the term Optimal?
- Answer If an algorithm runs as fast in the worst
case as it is possible (in the best case), we say
that the algorithm is optimal. i.e if the worst
case performance of an algorithm matches the
lower bound, the algorithm is said to be optimal
19Inversion -
- Example 4, 2, 3
- No. of pairs 3C2 , out of which
- (4,2) is out of order i.e. inversion
- (2,3) is in order
- (4,3) inversion
-
20- In n elements there will be n(n-1)/2 inversions
in worst case. - Thus, if an algorithm sorts by removing at most
one inversion per comparison then it must do at
least n(n-1)/2 comparisons in the worst case.
21- Insertion Sort
-
- x1,x2,...,xk-1, xk,.............,xi-1,xi
-
- Let xi is inserted after xk-1
- No. of comparisons (i-1) (k 1) 1 i k
1 - No. of inversions removed (i-1) (k 1) i
k - No. of inversions removed/comparison
- (1-k1)/(i-k)
- lt 1
22- Thus Insertion sort falls in the category of
comparison algorithms that remove at most one
inversion per comparison. - Insertion sort is optimal in this category of
algorithms .
23SELECTION SORT
- The algorithm works as follows
- Find the maximum value in the array.
- Swap it with the value in the last position
- Repeat the steps above for the remainder of the
array.
24Selection Sort An Example
- For i n to 2
- Select the maximum in a1.......ai.
- b) Swap it with ai.
25- Selection Sort
-
- x1,x2,, xk,..........., xn
-
- Let the maximum is located at xk
- Swap it with xn
- Continue
26- Selection Sort
-
- x1,x2,, xk,..........., xn-i1,xn
- Suppose we are at the ith iteration
-
- Let the maximum is located at xk
- Swap it with xn-i1
27An Example Selection Sort
5 20 6 4 15 3 10 2
28An Example Selection Sort
5 2 6 4 15 3 10 20
29An Example Selection Sort
5 2 6 4 10 3 15 20
30An Example Selection Sort
5 2 6 4 3 10 15 20
31An Example Selection Sort
5 2 3 4 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
32An Example Selection Sort
4 2 3 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
33An Example Selection Sort
3 2 4 5 6 10 15 20
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
34An Example Selection Sort
2 3 4 5 6 10 15 20
DONE!!!!
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
Thanks to MCA 2012 Chhaya Nathani(9)
35ANALYSISSELECTION SORT
- SelectionSort(A, n)
- for i n to 2 maxi
for ji-1 to 1 - If ajgtamax
- Maxj
- If(max!i)
- Swap aiamax
36ANALYSIS SELECTION SORT
- Selecting the largest element requires scanning
all n elements (this takes n - 1 comparisons) - and then swapping it into the last position.
Finding the next largest element requires
scanning the remaining n - 1 elements and so
on... - (n - 1) (n - 2) ... 2 1 n(n - 1) / 2
- i.e T(n2) comparisons
- T(n) T(n2)
37- Selection Sort
-
- x1,x2,, xk,..........., xn-i1,xn
- Suppose we are at the ith iteration
-
- Let the maximum is located at xk
- No. of comparisons (n i 1) - 1 n - i
- No. of inversions removed (n i 1) 1
(k-1) n i k 1 - No. of inversions removed/comparison
- ( n i k 1)/ (n i)
- lt 1 (since k gt 1)
38- Thus Selection sort also falls in category of
comparison algorithms that remove at most one
inversion per comparison. - Selection sort is also optimal in this category
of algorithms .
39Merge Sort(Divide and Conquer Technique)
- Divide the list into nearly equal halves
- Sort each half recursively
- Merge these lists
4018 9 15 13 20 10 7 5
Divide into 2 Subsequence
18 9 15 13
20 10 7 5
18 9
15 13
20 10
7 5
18
9
15
13
20
10
7
5
41Next Sort the 2 subsequences and merge them.
42Sorted Sequence
5 7 9 10 13 15 18 20
Sort merge 2 Subsequence
9 13 15 18
5 7 10 20
9 18
13 15
10 20
5 7
18
9
15
13
20
10
7
5
Initial Subsequence
43Merging
- Let we have two sorted lists
- A
- B
- Compare a1 with b1 and put smaller one in new
array C and increase the index of array having
smaller element.
a1
a2
a3
.
an
b1
b2
b3
.
bm
44- Let a1 lt b1
- A
- B
- Sorted Array C
-
- C
a1
a2
a3
.
an
b1
b2
b3
.
bm
a1
45- Example 1
- A
- B
- Sorted Array C
-
- C
20
40
60
80
10
30
35
38
45
50
52
10
20
46Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
Thanks toGaurav Gulzar (MCA -11)
47Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
Thanks toGaurav Gulzar (MCA -11)
48Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
Thanks toGaurav Gulzar (MCA -11)
49Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
Thanks toGaurav Gulzar (MCA -11)
50Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
Thanks toGaurav Gulzar (MCA -11)
51Example 1 A B Sorted Array C C
20
40
60
80
10
30
35
38
45
50
52
10
20
30
35
38
40
45
50
52
60
80
Thanks toGaurav Gulzar (MCA -11)
52Worst Case Analysis
- If no. of elements in first list is n in
second list is m then - Total No. of comparisons n-1m
- i.e. O(m n) in worst case and
-
53What is the best case?
- Number of Comparisons in the best case
- min(n , m), i.e. O(min(n , m))
54- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
55- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
56- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
35
57- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
58- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
59- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
60- Example 2
- A
- B
- Sorted Array C
-
- C
5
8
10
30
35
38
45
50
52
5
8
10
30
35
38
45
50
52
61What is the best case?
- Arrays
- Total number of steps O(mn)copying the rest
of the elements of the bigger array. - Linked List
- Total number of steps
- min(n , m), i.e. O(min(n , m)) in best case.
62Analysis of Merge Sort
- Since size of both the lists to be merged is
n/2, in either case (arrays or linked list), time
to merge the two lists is T(n). - If T(n) no. of comparisons performed on an
Input of size n then - T(n) T(n/2) T(n/2) T(n)
- 2T(n/2) cn ? ngt n0
- ? T(n) O(nlogn)
-
63- If T(n) no. of comparisons performed on an
Input of size n then - T(n) T(n/2) T(n/2) T(n)
- 2T(n/2) cn ? ngt n0
- ? T(n) O(nlogn)
64Final Points
- Merging is T(m n) in case of an Array
- If we use link list then it is O(min(m , n))
- Time for merging is O(n) and O(n/2) i.e. T(n)
65Conclusion
- Merge Sort T(nlogn)
- Have we beaten the lower bound?
- No,
- It just means that merge sort does not fall in
the previous category of algorithms. It removes gt
1 inversions per comparisons.
66- What is the
- Worst case
- Best Case
- for Merge Sort?
67Merge Sort Vs Insertion Sort
- What is the advantage of merge sort?
- What is the advantage of insertion sort?
68Merge Sort Vs Insertion Sort contd..
- Merge Sort is faster but does not take advantage
if the list is already partially sorted. - Insertion Sort takes advantage if the list is
already partially sorted.
69Lower Bound
- Any algorithm that sorts by comparison only does
at least ?(n lg n) comparisons in the worst case.
70Decision Trees
- Provides an abstraction of comparison sorts. In
a decision tree, each node represents a
comparison. - Insertion sort applied on x1, x2, x3
x1x2
x1ltx2
x1gtx2
x2x3
x1x3
x2gtx3
x3gtx1
x3gtx2
x1gtx3
x2x3
x3x1
x1ltx2ltx3
x3gtx1gtx2
x2gtx3
x3gtx2
x3gtx1
x3ltx1
x2gtx1gtx3
x2gtx3gtx1
x1gtx2gtx3
x1gtx3gtx2
71Decision Trees
- What is the minimum number of leaves in a
decision tree? - Longest path of the tree gives us the height of
the tree, and actually represents the worst case
scenario for the algorithm. - height of a decision tree O (n log n)
- i.e. any comparison sort will perform at least
(n logn) comparisons in the worst case.
72- Proof
- Let h denotes the height of the tree.
- Whats the maximum of leaves of a binary tree
of height h? 2h - 2h gt number of leaves gt n!
- (where n no. of elements and n! is a lower
bound on the no. of leaves in the decision tree) - gt hgt log(n!)
- gt n log n ( By Stirlings Approximation)
- ( SA n! v (2.p.n) .(n/e)n
- gt (n/e)n
- Thus, log(n!) gt n log n n log e gt n logn. )
-
73Merge Sort is Optimal
- Thus the time to comparison sort n elements is
?(n lg n) - Corollary Merge-sort is asymptotically optimal
comparison sorts. - Later well see another sorting algorithm in this
category namely heap-sort that is also optimal. - Well also see some algorithms which beat this
bound. Needless to say those algorithms are not
purely based on comparisons. They do something
extra.
74Quick Sort(Divide and Conquer Technique)
- Pick a pivot element x
- Partition the array into two subarrays around the
pivot x such that elements in left subarray is
less than equal to x and element in right
subarray is greater than x - Recursively sort left subarray and right subarray
x
x
gtx
75Quick Sort (Algorithm)
- QUICKSORT(A, p, q)
- if p lt q
- kPARTITION(A, p, q)
- QUICKSORT(A, p, k-1)
- QUICKSORT(A, k1, q)
76Quick Sort (Example)
- Let we have following elements in our array -
- i j
-
- i j
-
- i j
27 14 9 22 8 41 56 31 15 53 99 11 30 24
14 27 9 22 8 41 56 31 15 53 99 11 30 24
14 9 27 22 8 41 56 31 15 53 99 11 30 24
77Quick Sort (Example)
14 9 22 27 8 41 56 31 15 53 99 11 30 24
14 9 22 8 27 41 56 31 15 53 99 11 30 24
14 9 22 8 27 24 56 31 15 53 99 11 30 41
78Quick Sort (Example)
14 9 22 8 24 27 56 31 15 53 99 11 30 41
14 9 22 8 24 27 30 31 15 53 99 11 56 41
14 9 22 8 24 27 11 31 15 53 99 30 56 41
79Quick Sort (Example)
14 9 22 8 24 11 27 31 15 53 99 30 56 41
14 9 22 8 24 11 27 99 15 53 31 30 56 41
14 9 22 8 24 11 27 53 15 99 31 30 56 41
80Quick Sort (Example)
- i j
-
- i j (stop)
-
- (recursive call on left array) (recursive call
on right array)
14 9 22 8 24 11 27 15 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
14 9 22 8 24 11 15 27 53 99 31 30 56 41
81-
- i j i j
- i j i j
- i j i j
- i j i
j - i j
i j - i j i j (stop)
- i j (st op)
14 9 22 8 24 11 15
53 99 31 30 56 41
27
9 14 22 8 24 11 15
27
53 41 31 30 56 99
9 14 15 8 24 11 22
27
41 53 31 30 56 99
9 14 11 8 24 15 22
27
41 31 53 30 56 99
9 11 14 8 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
27
41 31 30 53 56 99
9 11 8 14 24 15 22
41 31 30 53 56 99
27
829 11 8 14 24 15 22 27 41 31 30 53 56 99
i j i j i j
i j
9 8 11 14 15 24 22 27 31 41 30 53 56 99
i j i j i
j i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
(stop) i j i j (stop) (stop) i j
i j (stop)
8 9 11 14 15 22 24 27 31 30 41 53 56 99
Sorted Array
83Analyzing Quicksort
- Worst Case?
- Partition is always unbalanced
- Worst Case input?
- Already-sorted input, if the first element is
always picked as the pivot - Best case?
- Partition is perfectly balanced
- Best Case input?
- ?
- Worst Case and Best Case input when the middle
element is always picked as the pivot?
84Worst Case of Quicksort
- In the worst case
- T(1) ?(1)
- T(n) T(n - 1) ?(n)
- Does the recursion look familiar?
- T(n) ?(n2)
85 Best Case of Quicksort
- In the best case
- T(n) 2T(n/2) ?(n)
- Does the recursion familiar?
- T(n) ?(n lg n)
86Why does Qsort works well in practice?
- Suppose that partition() always produces a 9-to-1
split. This looks quite unbalanced! - The recurrence is
- T(n) T(9n/10) T(n/10) n
- T(n) ? (n log n)
- Such an imbalanced partition and ?(n log n) time?
87Why does Qsort works well in practice?
- Intuitively, a real-life run of quicksort will
produce a mix of bad and good splits - Pretend for intuition that they alternate between
best-case (n/2 n/2) and worst-case (n-1 1) - What happens if we bad-split root node, then
good-split the resulting size (n-1) node?
88Why does Qsort works well in practice?
- Intuitively, a real-life run of quicksort will
produce a mix of bad and good splits - Pretend for intuition that they alternate between
best-case (n/2 n/2) and worst-case (n-1 1) - What happens if we bad-split root node, then
good-split the resulting size (n-1) node? - We end up with three subarrays, size 1, (n-1)/2,
(n-1)/2 - Combined cost of splits n n -1 2n -1 O(n)
- No worse than if we had good-split the root node!
89Why does Qsort works well in practice?
- Intuitively, the O(n) cost of a bad split (or 2
or 3 bad splits) can be absorbed into the O(n)
cost of each good split - Thus running time of alternating bad and good
splits is still O(n lg n), with slightly higher
constants - How can we be more rigorous? well do average
analysis of Qsort later while doing randomized
algorithms.
90Quicksort Vs Merge Sort
- Merge Sort takes O(n lg n) in the worst case
- Quick Sort takes O(n2) in the worst case
- So why would anybody use Qsort instead of merge
sort? - Because in practice, Qsort is quick as the worst
case doesnt happen often.
91Up Next
- Linear-Time Sorting Algorithms
92