Dynamic Programming

1
Dynamic Programming

• Presented by Saman Halgamuge
• University of Melbourne
• Material and tutorial support
• Suhinthan Maheshwararajah

2
Dynamic Programming

• Introduction
• Some examples
• Problem formulation
• Deterministic Problems
• Stochastic Problems
• Approximate Dynamic Programming

3
A puzzle

• You have N number of pens on the table. You and
  your opponent are allowed to pick up 1,2 or 3
  pens at each turn. The player who picks up the
  last pen looses. Assume that you are the first
  player.
• what will be your first pick and the subsequent
  strategy to make sure that you will win (assume N
  15)?
• At what N values you are going to loose as the
  first player to pick (assume that your opponent
  knows how to play optimally)?

4
Introduction

• Dynamic programming (DP) is a recursive technique
  for finding an optimal solution.
• DP solves problems by dividing them into
  sub-problems (e.g. Divide and Conquer )
• Each sub-problem is solved once and the results
  are stored. Therefore, the results of the
  sub-problems are accessible during the DP
  process.
• DP efficiently computes or eliminate recurrences
  by storing partial results.

5
Introduction

• DP has been used in many real world applications
• Mechanical and Manufacturing Engineering
• Machine Scheduling
• Inventory control
• Mechatronic Engineering
• Control (Cruise control, Robotics, Thermostats)
• Sensor scheduling
• Biomedical Engineering
• Biological Sequence data analysis

6
Examples

• 1. Calculating Fibonacci Numbers

Calculate ?
7
Examples
Calculating recurrences makes the algorithm
Computationaly expensive.
8
Examples

• 2. Matrix Multiplication

Determining the value of
Number of operations required is 12
Number of operations required for
9
Examples

• Minimize the number of operations required for

• How many possible ways of calculating?

• Which one has the minimum number of operations?

10
Examples
3. Network Problem
How many possible ways from Uni to Knox
city? Which one is the shortest way?
11
Examples
1 2 3 T
1 2
N-1 N
Greedy method goes through all the available
paths in the network TN
Intractable for large N and T
12
Terminologies used in Dynamic programming

• States
• Stages (horizon, period)
• cost-to-go function
• Controls or Policies
• Optimal policy
• Principle of optimality

13
Problem formulation
System equation
Where
14
Problem formulation
15
Problem formulation

g (x )
There can be a terminal cost
N N
at the end of the process
16
Recall the examples

• 1. Calculating Fibonacci Numbers

Calculate ?
a. Divide the main problem into sub-problem . b.
Solve the sub-problem and store the results.
17

• 2. Matrix Multiplication

• Minimize the number of operations required for

18

• 2. Matrix Multiplication

19
Shortest path problem
Start from Knox city to Melbourne University
20
Shortest path problem
2320
1640
1030
1570
2220
2870
1390
2150
1660
The Shortest path is Melbourne Uni, Fitzroy,
Doncaster, Nunawading, Knox city
21
Shortest path problem
Melbourne Uni, Fitzroy, Doncaster, Nunawading,
Knox city
What are the shortest paths from Fitzroy to Knox
city and Doncaster to Knox city?
Is it Fitzroy, Doncaster, Nunawading, Knox
city ?
Is it Doncaster, Nunawading, Knox city ?
Yes , this is called Principle of Optimality
22
Principle of Optimality
Consider the sub-problem
23
Principle of Optimality

• Optimal Policy can be built up in piecemeal
  fashion by solving the tail sub-problem until we
  cover the entire problem.
• Whatever the initial state is, remaining
  decisions must be optimal with regard to the
  state following from the first decision

24
Longest simple path (without repeated nodes or
loops) problem(e.g. exploring new places)
Principle of Optimality
The longest simple path from A to D is A, B, C
,D However, the sub-path A B is not the
longest simple path from A to B (A, C, B is
longer) The principle of optimality is not
satisfied for this problem
25
Limitations of using Dynamic Programming

• The problem must observe the principle of
  optimality.
• Cost function that additive over time
• To avoid infeasible memory space, the number of
  possible sub-problems should not be large.
• Dynamic programming works best on objects which
  are linearly ordered and cannot be rearranged -
  characters in a string, matrices in a chain,
  points around the boundary of a polygon, the
  left-to-right order of leaves in a search tree.