Greedy Algorithms

1
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems
Input Output Objective
2
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
3
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
4
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
5
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
6
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
7
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
8
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
(Greedy) algo
9
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
total time selected
10
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection 2
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
total time selected
Greedy algo ?
11
Greedy Algorithms

• make decisions greedily,
  previous decisions are
  never reconsidered

Optimization problems Activity selection 2
Input Output Objective
a set of time-intervals a subset of
non-overlapping intervals maximize of selected
intervals
total time selected
An algo ?
12
BackTracking

• brute force try all possibilities
• running time?

13
BackTracking

• brute force try all possibilities
• running time? O(n!)

Anything better for Activity selection 2?
14
BackTracking

• brute force try all possibilities
• running time? O(n!)

Anything better for Activity selection 2?
15
BackTracking

• brute force try all possibilities
• running time? O(n!)

Anything better for Activity selection 2?
- dynamic programming
16
Dynamic Programming

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

17
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Let Sk
considering only the first k intervals, what is
the optimal length?
18
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Let Sk
considering only the first k intervals, what is
the optimal length?
Which value are we ultimately looking for?
19
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Let Sk
considering only the first k intervals, what is
the optimal length?
Which value are we ultimately looking for?
Sn
How to compute Sk1, having computed S1, ,
Sk ?
20
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
Sn
How to compute Sk1, having computed S1, ,
Sk ?
21
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
Sn
How to compute Sk1, having computed S1, ,
Sk ?
22
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
maxj Sn,j
How to compute Sk1, having computed S1, ,
Sk ?
23
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
maxj Sn,j
How to compute Sk1, having computed S1, ,
Sk ?
?
24
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
maxj Sn,j
How to compute Sk,j, having computed Sk,j
for every (k,j) such that kltk ?
25
Dynamic Programming Activity selection 2

• idea remember values for smaller instances, use
  it for computing the value for a larger instance

Sk,j
Let Sk
considering only the first k intervals, what is
the optimal length ending with
interval j ?
Which value are we ultimately looking for?
maxj Sn,j
How to compute Sk,j, having computed Sk,j
for every (k,j) such that kltk ?
Sk,j maxj fj bj Sk-1,j
(fj-bj)
26
Dynamic Programming Activity selection 2
Sk,j maxj fj bj Sk-1,j
(fj-bj)
Can output the optimal time, how to find the
optimal selection?
27
Approaches to Problem Solving

• greedy algorithms
• dynamic programming
• backtracking
• divide-and-conquer

28
Approaches to Problem Solving

• greedy algorithms
• dynamic programming
• backtracking
• divide-and-conquer

our focus this week
29
Problem Huffman Coding
binary character code assignment
of binary strings to characters
e.g. ASCII code A 01000001 B
01000010 C 01000011
fixed-length code
How to decode ?
01000001010000100100001101000001
30
Problem Huffman Coding
binary character code assignment
of binary strings to characters
e.g. code A 0 B 10 C 11

variable-length code
How to decode ? 0101001111
31
Problem Huffman Coding
binary character code assignment
of binary strings to characters
e.g. code A 0 B 10 C 11

variable-length code
DEF A code is prefix-free if no codeword is a
prefix of another codeword.
How to decode ? 0101001111
32
Problem Huffman Coding
binary character code assignment
of binary strings to characters
e.g. another code A 1 B 10 C
11
variable-length code
DEF A code is prefix-free if no codeword is a
prefix of another codeword.
How to decode ? 10101111
33
Problem Huffman Coding
binary character code assignment
of binary strings to characters
e.g. another code A 1 B 10 C
11
not prefix-free
variable-length code
DEF A code is prefix-free if no codeword is a
prefix of another codeword.
How to decode ? 10101111
34
Problem Huffman Coding
- optimal prefix-free code
E 11.1607
A 8.4966
R 7.5809
I 7.5448
O 7.1635
T 6.9509
N 6.6544
S 5.7351
L 5.4893
C 4.5388
U 3.6308
D 3.3844
P 3.1671
M 3.0129
H 3.0034
G 2.4705
B 2.0720
F 1.8121
Y 1.7779
W 1.2899
K 1.1016
V 1.0074
X 0.2902
Z 0.2722
J 0.1965
Q 0.1962
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
35
Problem Huffman Coding
- optimal prefix-free code
A 60 B 20 C 10 D 10
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
36
Problem Huffman Coding
- optimal prefix-free code
00 01 10 11
A 60 B 20 C 10 D 10
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
37
Problem Huffman Coding
- optimal prefix-free code
00 01 10 11
Optimal ?
A 60 B 20 C 10 D 10
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
38
Problem Huffman Coding
- optimal prefix-free code
00 01 10 11
Optimal ? 2-bits per character Can do better ?
A 60 B 20 C 10 D 10
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
39
Problem Huffman Coding
- optimal prefix-free code
00 01 10 11
Optimal ? 2-bits per character Can do better
? YES, 1.6-bits per character
A 60 B 20 C 10 D 10
Input Output Objective
an alphabet with frequencies prefix
code expected number of bits per character
40
Problem Huffman Coding
- optimal prefix-free code
Huffman ( a1,f1,a2,f2,,an,fn) if n1
then codea1 ? else let fi,fj
be the 2 smallest fs Huffman (
ai,fifj,a1,f1,,an,fn ) codeaj ?
codeai 0 codeai ? codeai 1
41
Problem Huffman Coding
Let x,y be the symbols with frequencies fx lt
fy. Then in an optimal prefix code
length(Cx) ? length(Cy).
Lemma 1
42
Problem Huffman Coding
Let x,y be the symbols with frequencies fx lt
fy. Then in an optimal prefix code
length(Cx) ? length(Cy).
Lemma 1
Let C w0 be a longest codeword in an optimal
code. Then w1 is also a codeword.
Lemma 2
43
Problem Huffman Coding
Let x,y be the symbols with frequencies fx lt
fy. Then in an optimal prefix code
length(Cx) ? length(Cy).
Lemma 1
Let C w0 be a longest codeword in an optimal
code. Then w1 is also a codeword.
Lemma 2
Let x,y be the symbols with the smallest
frequencies. Then there exists an optimal prefix
code such that the codewords for x and y differ
only in the last bit.
Lemma 3
44
Problem Huffman Coding
Let x,y be the symbols with frequencies fx lt
fy. Then in an optimal prefix code
length(Cx) ? length(Cy).
Lemma 1
Let C w0 be a longest codeword in an optimal
code. Then w1 is also a codeword.
Lemma 2
Let x,y be the symbols with the smallest
frequencies. Then there exists an optimal prefix
code such that the codewords for x and y differ
only in the last bit.
Lemma 3
The prefix code output by the Huffman algorithm
is optimal.
Theorem