Review

1
Review
Thinking about Algorithms Abstractly
Jeff Edmonds York University
COSC 3101
2
Meta Algorithms

• I try to provide a unified way to think about
  each algorithm type.
• Follow my fixed set of steps.
• Even if you dont get the details of the
  algorithm correct, at least get the right
  structure.
• Try to match the parts of the new problem with
  the parts of the meta algorithm. eg instances,
  solutions, bird question,

3
Iterative Algorithms
4
Partial Correctness

• Proves that IF the program terminates then it
  works ltPreCondgt ltcodegt Þ ltPostCondgt

5
No Assumptions

• Suppose a Martian jumped
• into the top of the loop.
• All you would know is that ltloop-invariantgt was
  true.
• It alone must provide enough information
• so that, after going around the loop, you can
  establish that the loop invariant is again true.

6
Algorithm Termination
You need to define someMeasure of progressto
prove that your algorithmeventually terminates.
7
More of the Input Loop Invariant
The input consists of an array of objects
We have read in the first i objects. We will
pretend that this prefix is the entire
input. We have a solution for this prefix plus
some additional information
8
More of the OutputLoop Invariant
The output consists of an array of objects
I have produced the first i objects.
Produce the i1st output object.
Done when output n objects.
i to i1
9
Restricted Search Space Loop Invariant

• Maintain a sublist.
• If the key is contained in the original list,
  then the key is contained in the sublist.

key 25
10
Iterative Algorithms
For all major algorithms covered. Learn pre and
post conditions. Learn the Loop Invariant Learn
how to make progress while maintaining the LI.
11
Friends Strong Induction
The steps to proof the program is correct are
similar to those for an iterative algorithm.
12
Friends Strong Induction
Carefully write the specifications for the
problem.
13
Friends Strong Induction
Carefully write the specifications for the
problem.

• To be sure that we solve the problem for every
  legal instance.
• So that we know
• what we can give to a friend.

Set of legal instances (inputs)
ltpreCondgt

• So that we know
• what is expected of us.
• what we can expect from our friend.

Required output
ltpostCondgt
14
Friends Strong Induction
Focus on only one step. Do not trace out the
entire computation.

• Consider your input instance
• Remember that you know nothing other than what
  is given to you by the preconditions.
• Be sure each path in your code returns what is
  required by the postcondition.

15
Friends Strong Induction
Focus on only one step. Do not trace out the
entire computation.
Consider your input instance

• Do not worry about who your boss is
• or how your friends solve their instance.
• No global variable or effects

16
Friends Strong Induction
From your instance, construct one or more
subinstances.
17
Friends Strong Induction
If your instance meets the preconditions, then
each of your subinstances also meet the
preconditions.
18
Friends Strong Induction
Prove that the subinstances that you give to you
friends are in some way smaller'' than your
instance
19
Friends Strong Induction
Prove that the subinstances that you give to you
friends are in some way smaller'' than your
instance
And your instance has a finite size.
20
Friends Strong Induction
Assume by magic (strong induction) your friends
give you a correct solution for each of these.
21
Friends Strong Induction
Use their solutions for their subinstances to
help you find a solution to your own instance.
22
Friends Strong Induction
If your friend's solutions meet the
postconditions for their subinstances, then your
solution meets the postcondition for your
instance.
23
Friends Strong Induction

• If you want more from your friends,
• change the pre and/or postconditions
• Be sure to document it.
• Be sure you and all of your friends are
  using the same pre/postconditions.
• Be sure to handle these changes yourself.

24
Friends Strong Induction
If your instance is sufficiently small, solve it
yourself as a base case.
25
Friends Strong Induction
If your instance is sufficiently small, solve it
yourself as a base case.
Your solution for the base case meets the
postconditions.
26
Friends Strong Induction
27
Induction
Friends Strong Induction
Strong Induction
28
Friends Strong Induction

• Induction Hypothesis

The recursive algorithm works for every
instance of size n.''
The algorithm works for every instance of any
size.''
size i
29
Friends Strong Induction
Give and solve the recurrence relation for the
Running Time of this algorithm.
30
Friends Strong Induction
Done
31
Graph Search
We know found nodes are reachable from s because
we have traced out a path.
If a node has been handled, then all of its
neighbors have been found.
l
32
Graph Search

• Queue Handle in order found.
• Breadth-First Search
• Stack Handle most recently found
• Depth-First Search
• Priority Queue Handle node that seems to
  be closest to s.
• Shortest (Weighted) Paths

33
Dijkstra's
Handled Nodes
Found Nodes
Handled paths go through handled edges through
any number of handled nodes followed by last
edge to an unhandled node.
For handled w,d(w) is the length of the
shortest paths to w.
d(v) is the length of the shortest handled path
to v.
Handle node with smallest d(u).
34
DFS
FoundNot HandledStack ltnode, edgesgt
s
b
a
e
d
g
c
f
i,0
j
c,2
i
a,1
s,1
h
m
k
l
35
Linear Order
a
h
b
i
c
j
d
k
e
g
c
l
f
b
When take off stackadd to backwards order
i,j,k,d,e,g,l,f
36
Optimization Problems

• Ingredients
• Instances The possible inputs to the problem.
• Solutions for Instance Each instance has an
  exponentially large set of solutions.
• Cost of Solution Each solution has an easy to
  compute cost or value.
• Specification
• Preconditions The input is one instance.
• Postconditions An valid solution with optimal
  cost. (minimum or maximum)

37
Network Flow

• Solution
• The amount of flow Fltu,vgt through each edge.
• Flow Fltu,vgt cant exceed capacity cltu,vgt.
• No leaks, no extra flow.
• For each node v flow in flow out
• ?u Fltu,vgt ?w Fltv,wgt

38
Network Flow

• Value of Solution
• Flow from s into the network
• minus flow from the network back into s.
• rate(F) ?u Flts,ugt

- ?v Fltv,sgt
Goal Max Flow
39
Min Cut

• Value Solution CltU,Vgt
• cap(C) how much can flow from U to V
• ?u?U,v?V cltu,vgt

Goal Min Cut
s
t
40
Network Flow
cltv,ugt
Fltu,vgt
cltu,vgt
Flow Graph
Augmentation Graph
fltu,vgt/cltu,vgt
cltu,vgt-Fltu,vgt
u
v
u
v
Fltu,vgtcltv,ugt
0/cltv,ugt
41
Network Flow
cltu,vgt
Fltu,vgt
cltu,vgt
Flow Graph
Augmentation Graph
Fltu,vgt/cltu,vgt
cltu,vgt-Fltu,vgt
u
v
u
v
Fltu,vgtcltv,ugt
0/cltv.ugt
42
Max Flow Min Cut

• Theorem
• For all Networks MaxF rate(F) MinC cap(C)
• Prove ? F,C rate(F) ? cap(C)

• Prove ? flow F, alg either
• finds a better flow F
• or finds cut C such that rate(F) cap(C)
• Alg stops with an F and C for which rate(F)
  cap(C)
• F witnesses that the optimal flow cant be less
• C witnesses that it cant be more.

43
Max Flow Min Cut
-w

• Given Flow F
• Construct Augmenting Graph GF
• Find path P
• Let w be the max amount flowcan increase along
  path P.
• Increase flow along path P by w.
• i.e newF oldF w P

44
Max Flow Min Cut

• Given Flow F
• Construct Augmenting Graph GF
• Find path P using BFS, DFS, or generic search
  algorithm
• No path

45
Max Flow Min Cut

• Let Falg be this final flow.
• Let cut CalgltU,Vgt,
• where U are the nodes reachable from s in the
  augmented graph
• and V not.
• Claim rate(Falg) cap(Calg)

46
Abstract Out Essential Details
Cost 29, 8, 1, 2
47
Optimization Problems with a Greedy Algorithm
Instances A set of objects and a relationship
between them.
Cost of Solution The number of objects in
solution or the sum of the costs of objects
48
Fixed vs Adaptive Priority

• Fixed Priority
• Sort the objects from best to worst and loop
  through them.
• Adaptive Priority
• Greedy criteria depends on which objects have
  been committed to so far.
• At each step, the next best object is chosen
  according to the current greedy criteria.
• Searching or re-sorting takes too much time.
• Use a priority queue.

49
Fixed vs Adaptive Priority
50
We have not gone wrong. There is at least one
optimal solution consistent with the choices made
so far.
Initially no choices have been made and hence all
optimal solutions are consistent with these
choices.
51
Let optSLI denote one.
?
Proof changes optSLI into optSours and proves it

• is a valid solution
• is consistent both with previous and new choices.
• is optimal

52
Alg commit to or reject each object.
Has giving a solution S.
opt sol consistent with these choices.
S must be an optimal solution.
Alg returns S .
53
Proof Greedy Algorithm Works

• Looks hard,
• but they all have the same structure.
• Take my proof structure.
• Change what the instances, solutions, costs
  are.
• Change what the greedy criteria is.
• Change the change step.
• Change the proof that the changed solution is
• valid
• consistent
• optimal
• The rest is the same.

54
Optimization Problems

• Dont mix up the following
• What is an instance
• What are the objects in an instance
• What is a solution
• What are the objects in a solution
• What is the cost of a solution
• Greedy algorithm
• What does the algorithm do know
• What does the prover do know
• What does the fairy god mother do know
• Recursive Backtracking / Dynamic Programming
• What does the algorithm do know
• What does the little bird do know
• What does the friend do know

55
Review
Designing Recursive Back Tracking Algorithm

• What are instances, solutions, and costs?
• Given an instance I,
• What question do you ask the little bird?
• Given a bird answer k ? K,
• What instance subI do your give your friend?
• Assume he gives you optSubSol for subI.
• How do you produce an optSol for I from
• the birds k and
• the friends optSubSol?
• How do you determine the cost of optSol from
• the birds k and
• the cost of the friends optSubSol?
• Try all birds answers and take best of best.

56
Review
Recursive Back Tracking Algorithm
Dynamic Programming Algorithm

• Given an instance I,
• Imagine running the recursive alg on it.
• Determine the complete set of subI ever given
  to you, your friends, their friends,
• Build a table indexed by these subI
• Fill in the table in order so that nobody waits.
• the cost of its optimal solution
• advice given by the bird
• Run the recursive alg with birds advice to find
  the solution to your instance.

57
(No Transcript)
58
Dynamic Programmingdonts

• When looping over the subinstances
• be clear what the set of subinstances are
• which is currently being solved, i.e. which
  instance is cost(i,j)?
• If you know that the set of subinstances are the
  prefixes of the input, i.e. lta1,a2, , aigt, then
  dont have a two dimensional table.
  Table1..n,1..n.
• Dont loop over i and loop over j if j never gets
  mentioned again.

59
Dynamic Programmingdonts

• .When trying all bird answers
• be clear what the set of bird answers are,
• which is currently being tried,
• what it says about the solution being looked
  for.
• When getting help from your friend,
• be clear what the subinstance is that you are
  giving him
• How do you use the current instance and the birds
  answer to form his subinstance.
• Dont simply say cost(i-1,j-1)

60
Dynamic Programmingdonts

• .Think about what the base cases should be.
• Dont make an instances a base cases if they
  can be solved using the general method.
• is used to start a comment. Dont put it in
  front of code.

61
History of Classifying Problems
Halting
Non-Deterministic Polynomial Time

• There are lots of important search problems
• exponential time to search
• poly time to verify given witness

SAT
Matching
Circuit-Sat Problem Does a circuit have a
satisfying assignment.
GCD
62
Non-DeterministicPoly-Time Decision Problems (NP)

• Key Given
• an instance I ( ltG,kgt)
• and a solution S ( subset of nodes)
• there is a poly-time alg Valid(I,S) to test
  whether or not S is a valid solution for I.
• Poly-time in I not in S. S cant be too big.
  

Not Valid
Valid
Formal definition Prob ? NP iff ? poly time
Valid such that Prob(I) ? S Valid(I,S)
63
Non-DeterministicPoly-Time Decision Problems (NP)

• Key
• If the instance has a valid solution
• A non-deterministic (fairy god mother) could
  prove it to you by giving you
• such a solution as a witness.
• You could check that it is valid.
• You could convince your boss.

Valid
64
Non-DeterministicPoly-Time Decision Problems (NP)

• Key
• If the instance does not have a valid solution
• A non-deterministic (fairy god mother) could
  prove it to you by giving you
• ????
• You have no way to convince your boss.

k5
65
Reductions

• Reduction Design a fast algorithm for one
  computational problem, using a supposedly fast
  algorithm for another problem as a subroutine.
• Use to compare the two problems.
• Even if we dont know whether they can be solved
  in polynomial time or not,
• we can learn that either they both can or neither
  can.
• We can also learn that they have a similar
  structure.

66
Reductions
Design a fast algorithm for Palg using a
supposed fast algorithm for Poracle as a
subroutine.
We will only consider reductions of this simple
form.
67
Reductions
We give a fast algorithm for Palg using a
supposed fast algorithm for Poracle as a
subroutine.
If there is not a fast algorithm for Palg
If there is a fast algorithm for Palg
?
??
then there is not fast algorithm for Poracle
?
If there is a fast algorithm for Poracle
If there is not a fast algorithm for Poracle
then there is a fast algorithm for Palg
??
?
?
68
Reductions
We give a fast algorithm for Palg using a
supposed fast algorithm for Poracle as a
subroutine.
Conclusions
Palg is at least as easy as Poracle (Modulo
polynomial terms.)
Poracle is at least as hard as Palg (Modulo
polynomial terms.)
The problems have a similar underling structure.
Notation Palg poly Poracle
69
NP-Complete Problems
SAT
Lets compare the problems in NP
None are harder than SAT!
Circuit-Sat Problem Does a circuit have a
satisfying assignment.
GCD
70
NP-Complete Problems

• Problem Pnew is NP-Complete
• Pnew not too hard.
• Pnew ? NP

Test in poly-time if a given solution is valid
Pnew
GCD
71
NP-Complete Problems

• Problem Pnew is NP-Complete
• Pnew not too hard.
• Pnew ? NP
• Pnew not too easy
• ? Prob ? NP, Prob ?poly Pnew

Pnew
?poly Prob
GCD
72
NP-Complete Problems
SAT
SAT is NP-Complete!
GCD
73
NP-Complete Problems

• Problem Pnew is NP-Complete
• Pnew not too hard.
• Pnew ? NP
• Pnew not too easy
• Sufficient to show Pnew ?poly SAT
• Because then
• Pnew ?poly SAT ?poly Prob
• By Cook, Pnew Prob ?poly SAT

SAT
Pnew
GCD
74
NP-Complete Problems

• A very large class of problems
• Industry would love to solve them quickly
• We dont know how.
• If we could solve one quickly, Þ we could solve
  all quickly.
• If one impossible to solve quickly Þ all
  impossible to solve quickly.

75
End