Fundamental Techniques

1
Fundamental Techniques

• CS 5050
• Chapter 5
• Goal review complexity analysis
• Talk of categories used to descirbe algorithms.

2
Algorithmic Frameworks

• Greedy
• Divide-and-conquer
• Discard-and-conquer
• Top-down Refinement
• Dynamic Programming
• Backtracking/Brute Force
• Branch-and-bound
• Local Search (Heuristic)

3
Greedy Algorithms

• Short-view, tactical approach to minimize the
  value of an objective function.
• Eating whatever you want no regard for health
  consequences or leaving some for others.
• Solutions are often not global, but can be when
  problems satisfy the greedy-choice property
• Example Making change with US money to minimize
  number of coins not true with other money
  systems.
• Example Knapsack problem. Given items weight
  and value. Want best value in knapsack within
  total weight limit. Cannot be done using greedy
  methods. Doesnt have greedy choice property

4
Fractional Knapsack Problem

• Many items, each with their own positive weight
  and positive benefit, and a maximum total weight
• Can choose a fraction of the items quantity
• maximize benefit, staying within weight limits.
• Algorithm
• Items by maximal benefit (value/weight) on
  priority queue (or merely sort) O(n log n)
• Withdraw highest benefit item and take it all, or
  as much as will fit under the total quantity max
• greedy choice property proof by contradiction

5
Task Scheduling Problem

• Set of tasks with specific start and end times
  (no flexibility. All must be scheduled at
  specific times)
• try to minimize number of machines
• Sort by start time O(n log n) as simpler to check
  for conflict.
• For each, put it on the first available machine
  or use an additional machine
• Optimal - Proof of this by contradiction. Let k
  be last new machine. Let i be first task
  scheduled on last machine. Show i conflicts with
  all other tasks at same time.

6
Divide and Conquer Algorithms

• Divide the problem into smaller subproblems -
  these should be equal sized
• Eventually get to a base case
• Examples mergesort or quicksort
• Generally recursive
• Usually efficient, but sometimes not
• For example, Fibonacci numbers or Pascal triangle
• Desirability depends on work in spliting/combining

7
Consider Merge sort

• Sketch out the solution to the merge sort.
• What is the complexity? What skills do you have
  to help you?

8
Recurrence Relations

• Often of the form (e.g., merge sort)
• T(n) b if n lt 2
• 2T(n/2) bn if n gt 2
• Recursion tree bn effort at each of log n
  levels
• Plug and chug telescoping. Start with specific
  value of n to see pattern better. Math intensive.
• Iterative solution T(n) 2iT(n/2i) ibn
  becomes T(n) bn bnlog n
• Guess and test eyeball, then attempt an
  inductive proof

9

• Recursion tree bn effort at each of log n
  levels. Show pictures (see posted handout)
• Guess and test eyeball, then attempt an
  inductive proof. If cant prove, try something
  bigger/smaller. This is not a good way for
  beginning students.

10
Look at several algorithms

• Use picture method to find complexity.
• Use master method to find complexity

11
Master method (different from text)method of
text can give tighter bounds

• Of the form
• T(n) c if n lt d
• aT(n/b) O(nk) if n gt d
• a gt bk then T(n) is O(n logb a )
• a bk then T(n) is O(nk log n)
• a lt bk then T(n) is O(nk)
• Can work this out on your own using telescoping
  and math skills.

12
Large Integer multiplication

• Normal way is n2
• Breaking up into two parts
• I Ih2n/2 Il J Jh2n/2 Jl
• Shift is cheap O(n)
• I J (Ih2n/2 Il )(Jh2n/2 Jl )
• IhJh2n IlJh2n/2 IlJh2n/2 IlJl
• doesnt help as work is the same (4(n/2)2)
• However, complicated patterns of
  sum/difference/multiple reduce the number of
  subproblems to ¾ or 7/8
• Big integer multiplication is O(n1.585)
• Matrix multiplication is O(n2.808)

13
Large Integer multiplication

• Idea How think of? You want a way of reducing
  the total number of pieces you need to compute.
• Observe (Ih-Il)(Jl-Jh) IhJl-IlJl IhJh IlJh
• Key is for one multiply, get two terms we need if
  add in two terms we already have
• So, instead of computing the four pieces shown
  earlier, we do this one multiplication to get two
  of the pieces we need!

14
Large Integer multiplication

• IJ IhJh2n (Ih-Il)(Jh-Jl)IhJhIlJl 2n/2
  IlJl
• Tada three multiplications instead of four
• by master formula O(n1.585)
• a 3
• b 2
• k 1

15
Try at seats!For Example Mult 7553

• 7 5 2
• 5 3 -2
• 15 -4 Products
• 35100 10(-4 35 15) 15
• 75533975

16
Matrix multiplication

• Same idea Breaking up into four parts doesnt
  help
• Strassens algorithm complicated patterns of
  sum/difference multiply reduce the number of
  subproblems to 7 (from 8)
• Wont go through details, as not much is learned
  from the struggle.
• T(n) 7T(n/2) bn2
• Matrix multiplication is O(n2.808)

17
Discard and ConquerTop-Down Refinement

• Both are similar to divide and conquer
• Discard and conquer requires only that we solve
  one of several subproblems
• Corresponds to proof by cases
• Binary search is an example
• Finding kth smallest is an example (Quicksearch)
• Top-down assembles a solution from the solutions
  to several subproblems
• Subproblems are not self-similar or balanced
• This is the standard problem-solving method

18
Dynamic Programming Algorithms

• Reverse of divide and conquer, we build solutions
  from the base cases up
• Avoids possible duplicate calls in the recursive
  solution
• Implementation is usually iterative
• Examples Fibonacci series, Pascal triangle,
  making change with coins of different relatively
  prime denominations

19
Good Dynamic Programming Algorithm Attributes

• Simple Subproblems
• Subproblem Optimality optimal solution consists
  of optimal subproblem.
• Can you think of a real world example without
  subproblem optimality? Round trip discounts.
• Subproblem Overlap (sharing)

20
The 0-1 Knapsack Problem

• 0-1, means take item or leave it (no fractions)
• Now given units which we can take or leave
• Obvious solution of enumerating all subsets is
  T(2n)
• Difficulty is in characterizing subproblems
• Find best solution for first k units no good,
  as optimal solution doesnt build on earlier
  solutions
• Find best solution, first k units within quantity
  limit
• Either use previous best at this limit, or this
  plus previous best at reduced limit O(nW)
• Pseudo-polynomial as it depends on a parameter
  W, which is not part of other solutions.

21
You could try them exhaustively, deciding about
the last thing first

• int valueMAX // value of each item
• int weightMAX // weight of each item
• //You can use item "item" or items with lower
  number
• //The maximum weight you can have is maxWeight
• int bestValue(int item, int maxWeight)
• if (item lt 0) return 0
• if (maxWeight lt weightitem)
• // current item can't be used, skip it
• return bestValue(item-1, maxWeight)
• useIt bestValue(item-1, maxWeight -
  weightitem) valueitem
• dontUseIt bestValue(item-1, maxWeight)
• return max (useIt, dontUseIt)

22
Price per Pound

• The constant price-per-pound' knapsack problem
  is often called the subset sum problem, because
  given a set of numbers, we seek a subset   that
  adds up to a specific target number, i.e. the
  capacity of our knapsack.
• If we have a capacity of 10, consider a tree in
  which each level corresponds to considering each
  item (in order). Notice, in this simple example,
  about a third of the calls are duplicates.

23
(No Transcript)
24

• We would need to store whether a specific weight
  could be achieved using only items 1-k.
• possibleitemmax given the current item (or
  earlier in the list) and max value, can you
  achieve max?

25

• Consider the weights 2, 2, 6,5,4 with limit of
  10
• We could compute such a table in an iterative
  fashion

26
From the table

• Can you tell HOW to fill the knapsack?

27

• Let's compare the two strategies
• Normal/forgetful wait until you are asked for a
  value before computing it. You may have to do
  some things twice, but you will never do anything
  you don't need.
• Compulsive/elephant you do everything before you
  are asked to do it. You may compute things you
  never need, but you will never compute anything
  twice (as you never forget).
• Which is better? At first it seems better to wait
  until you need something, but in large
  recursions, almost everything is needed somewhere
  and many things are computed LOTS of times.

28
Consider the complexity for a max capacity of M
and N different items.

• Normal For each item, try it two ways (useIt or
  dontUseIt). O(N2)
• Compulsive Fill in array O(MN)
• Which is better depends on values of M and N.
  Notice, the two complexities depend on different
  variables.

29
Clever Observation

• Since only the previous row is ever accessed,
  dont really need to store all rows.
• However, couldnt easily read back optimal choices

30

• The Matrix Chain Problem
• B2x10C10x50D50x20 Is it associative?
• Does order matter? BC 21050
• (BC)D 21050 25020 25030 (best)
• B(CD) 105020 21020 521020
• Reduce number of multiplies by proper association
• Naïve algorithm to find the proper association is
  exponential in number of matrices
• Let Ni,j denote the minimum number of
  multiplications to compute AiAi1...Aj

31
At seats -What is Algorithm?

• For each cell N(i,j) represents the cost of
  computing the multiplication of matrices i thru
  j.
• Look at each possible division
• Pick the best of the possibilities
• k is division point
• N(i,k) N(k1,j) gives each piece
• multiply two pieces is di xdk1 and dk1 x dj1
• subscripting remember di is rowsize of ith
  matrix

32
Backtracking AlgorithmsBrute Force Algorithms

• Exhaustive (brute force) search - depth-first
  through the solution space if it is structured
• Bad if happen to pick a first path that is very
  deep.
• Backtrack when we cant go forward
• Brute force if try everything.
• Example - tree traversal looking for a key or
  finding a solution to Eight Queens problem
• Heuristics help a lot, as for instance, knowledge
  about the structure of a binary search tree
• Usually implemented with a stack

33
Observe divide and conquer

• Observe that divide and conquer may be brute
  force.
• We can do depth first better for storage
• We can do breath first may be better for
  optimality, but must use a queue to store
  subproblems yet to explore
• What about a best first solution?

34

• PRUNING
• However, sometimes we can determine that a given
  node in the solution space does not lead to the
  optimal solution--either because the given
  solution and all its successors are infeasible or
  because we have already found a solution that is
  guaranteed to be better than any successor of the
  given solution.
• In such cases, the given node and its successors
  need not be considered. In effect, we can prune 
  the solution tree, thereby reducing the number of
  solutions to be considered.

35
Branch and Bound Algorithms

• Definition An algorithmic technique to find the
  optimal solution by keeping the best solution
  found so far. If a partial solution cannot
  improve on the best, it is abandoned.
• Has a scoring mechanism to always choose the more
  promising best-first search.

36
Branch and Bound Algorithms

• For each subtree, best possible solution is
  computed. If it is worse than best so far,
  prune.
• Rather than wanting the absolute best solution,
  we may say at least as good as bound. If we
  find no solution, can relax the bound and start
  over.
• Backtrack sooner when we realize the branch is
  going bad - this is called pruning

37
Branch and Bound Algorithms

• Need an evaluation function of some kind
• Examples - games, integer linear programming
• Besides pruning, the evaluation function may give
  us hints on which branch to follow so never
  really throw out a case, just give it less
  priority.
• Greedy algorithms are extreme examples of branch
  and bound algorithms as ignore all other branches.

38
Example

• consider the scales balancing problem or dividing
  a set into approximately equal weight pieces
• What problem is this most like?
• Consider a partial solution in which we have
  placed k weights onto the pans (0 lt k lt n ) and,
  therefore, n-k weights remain to be placed. The
  difference between the weights of the left and
  right pans is computed
• the sum of the weights still to be placed is
  computed
• For any given subproblem, if the sum of the
  weights to be placed is less than the current
  difference in the pans, you have a measure of how
  close you can come to the desired goal.
• Prune if your best possible solution is worse
  than the best so far.

39
Local Search Algorithms

• Start at a random spot, then follow a gradient of
  improving solutions, termed hill climbing
• Locally optimal, but globally suboptimal
• Differs from greedy in that there is a sequence
  of solutions
• Example - travelling salesman. Come up with an
  initial route which visits all cities, but may
  not be optimal. Then iteratively try to improve
  it. Break and reconnect.

40
Relatives to Big Oh

• f(n) is ?(g(n)) (pronounced big Omega) if
• g(n) is O(f(n)) In other words
• there exists c gt 0 and integer constant n0gt1
• such that f(n) ? cg(n) for n ? n0.
• f(n) is ?(g(n)) (pronounced big Theta) if
• f(n) is O(g(n)) and f(n) is ?(g(n)). In
  other words,
• there exists cgt0 and cgt0 and n0gt1 such that
• cg(n) ? f(n)? cg(n) for n ? n0