Stochastic Optimal Control

1
Stochastic Optimal Control

• Lecture XXVIII

2
Stochastic Process

• Introduction to Stochastic Process.
• Most of the material in the section originates
  from Dixit, Avinash The Art of Smooth Pasting
  (Buffalo, NY Gordon and Breach Publishing Co.,
  1993). The publisher can be reached on the
  network at http//www.gbhap.com/.

3

• Brownian Motion
• Brownian motion refers to a continuous time
  stochastic process where xt evolves over time
  according to some stochastic differential
  equation
• where dt represents an increment in time and dw
  denotes a random component. The expected value
  of xt given an initial condition x0 becomes x0mt
  with a variance of s2t.

4

• Random Walk Representation To make the
  formulation more concrete, we will begin by
  depicting Dx as a random walk with a probability,
  p, that the value of x will increase Dh at any
  given time period and a probability (1-p) that
  the value of x will decrease by Dh in the same
  time period. A graphical representation for the
  value of x can be depicted as

5
(No Transcript)
6

• The expected value of Dx is
• Similarly, the variance of Dx is

7
(No Transcript)
8
Stochastic Optimal Control (VII)

• Extending this result to time period Dt, assuming
  that each event is independent yields a mean of
• and a variance of
• This distribution can be linked to a Binomial
  distribution. Specifically, if we take Dh as a
  success, then the Binomial distribution for the
  probability of k successes is

9

• Taking the limit of the Binomial distribution
  derive a normal distribution with mean mt and
  variance s2t. Thus, the mean of any (xt-x0) is
  mt and the standard deviation is s(t)1/2.

10
Itos Lemma

• Itos Lemma Suppose that x follows Brownian
  motion with parameters (m,s), what is the
  distribution of stochastic process y which is
  defined as yf(x) (given that f(x) is a nonrandom
  function). The expression for dy is then
  computed by taking a Taylor series expansion of y

11

• The expected value of dy, Edy can then be
  computed as
• with the variance of the change in y becoming
  Vyt-y0f'(x0)2s2t.

12

• This enables us to rewrite the change in y as
• In a slightly more general form

13
Geometric Brownian Motion

• Geometric Brownian Motion Applying Itos Lemma
  letting Xex and assuming that x follows a
  Brownian motion process yields
• with a variance of

14

• Therefore, the stochastic process can be
  rewritten as
• This form of stochastic process is fairly
  useful, because (d X/X) is a manifestation of the
  rate of return on assets.

15

• Thus, if the rate of return on assets follows a
  Brownian motion process
• Then the asset value itself follows an absolute
  or ordinary Brownian motion process

16
Stochastic Optimal Control

• Stochastic Optimal Control
• The basic concepts behind Stochastic Optimal
  Control are stochastic calculus via Itos Lemma
  and dynamic programming.
• Previously our equation of motion for optimal
  control was written as

17

• Now following our discussion from Dixits Art of
  Smooth Pasting we shift to a stochastic
  differential equation
• where g(t,x,u) is the deterministic component of
  the differential equation and s(t,x,u) dz is the
  stochastic portion with s(t,x,u) being some
  deterministic function and dz being a Brownian
  motion increment.

18

• Following our discussion above, the stochastic
  differential equation can then be rewritten as

19

• Solution

20

• Rewriting in terms of the value function
• Applying the Bellman-Jacobi framework

21

• Expanding around J(t,x)
• Applying Itos Lemma to the stochastic
  differential equation

22

• Substituting both of these expressions into the
  Bellman-Jacobi formulation
• Subtracting J from both sides and dividing
  through by Dt and taking Dt to zero yields