Stochastic Processes

1
Stochastic Processes

• Shane Whelan
• L527

2
Chapter 3 Markov Chains
3
Markov Chain - definition

• Recall a Markov chain is a discrete time Markov
  process with an at most countable state space,
  i.e.,
• A Markov chain is a sequence of random variables,
  X0, X1, such that
• PXnjX0a,X2b,,XmiPXnjXmi
• for all mltn.

4
Overview Example

• Markov chains are often displayed by a transition
  graph states linked by arrows when positive
  probability of transition in that direction,
  generally with transition probabilities shown
  alongside, e.g.

6
3
1
1
1
2/3
4
1
2
1
5
1
1/3
3/5
1/5
0
5
Overview Example

• Starting from 0, show that the prob. of hitting 6
  is ¼.
• Starting from 1, show that the prob. of hitting 3
  is 1.
• Starting from 1, show that it takes on average 3
  steps to hit 3.
• Starting from 1, show that the long-run
  proportion of time spent in 2 is 3/8.
• As the number of steps increases without number ,
  show that the probability that starting from
  state 0 one ends up in state 1 (after all the
  steps) limits to 9/32.

6
Transition Probabilities

• Transition probabilities are denoted
• Read as probability of being in state j at time
  n, given that at time m the process is in state i
  
• And a one-step transition probability is

7
Complete Specification of Markov Chain

• The distribution of a Markov chain is fully
  specified once the following are given
• The initial probability distribution,
  qkPX0k
• The one-step transition probabilities,
• As then the probability of any path,
  PX0a,X1b,,Xni, is readily deduced.
• Whence, with time, we can answer most
  questionsbut can be tricky as the overview
  example shows.
• Is there a more systematic approach?

8
The Chapman-Kolmogorov Equations

• Proposition The transition probabilities of a
  Markov chain obey the Chapman-Kolmogorov
  equations, i.e., for all times mltn
• Proof Direct consequence of Theorem of Total
  Probabilities.
• Above result justifies the term chain can
  you see the link?

where SState Space
9
Time-homogeneous Markov chains

• Definition A Markov chain is said to be
  time-homogeneous if
• i.e., the transition probabilities are
  independent of timeso knowing what state the
  process is in uniquely identifies the transition
  probabilities.

10
Time-homogeneous Markov chains
is called the (n-m)-step transition probability
and simplifies the Chapman-Kolmogorov equations
to...
11
Time-homogeneous Markov chains

• Define the transition matrix P by
• (P)ijpij (an NxN matrix where N is the
  cardinality of the state space) then the k-step
  transition probability is given by
• Clearly, we have
• Matrices with this latter property known as
  stochastic matrices.

12
Example of Time Homogeneous Markov Chain Simple
Random Walk

• As before, we have Xn?nZi, where
• PZi1 p, PZi-11-p.
• The process has independent increments hence is
  Markovian.
• The transition graph and transition matrix are
  infinite

13
Models based on Markov Chains (Revisited)

• Model 1 The No Claims Discount (NCD) system is
  where the motor insurance premium depends on the
  drivers claims record.
• Consider a three-state NCD system with a 0
  discount 25 discount and 50 discount state. A
  claim-free year results in a transition to a
  higher discount (or remain at the highest). One
  or more claims in a year moves the policyholder
  to the next lower discount level (or remain at
  0).
• Assume probability of one or more claims in a
  year is 0.25. Draw the transition graph and write
  out the one-step transition matrix. What is the
  probability in being in the highest discount
  state in year 3 if in year 1 one is on the 25
  discount?

14
Answer

• Transition Graph
• Transition Matrix

15
Model 2 (Revisited)

• Consider the 4-state NCD model given by
• State 0 0 Discount
• State 1 25 Discount
• State 2 40 Discount
• State 3 60 Discount
• Here the transition rules are move up one
  discount level (or stay at max) if no claim in
  the previous year. Move down one-level if claim
  in previous year but not the year before move
  down 2 levels if claim in two immediately
  preceding years.
• Now assume that the probability of a claim in any
  year is 0.25 (irrespective of the state the
  policyholder is in).

16
Model 2

• This is not a Markov chain
• PXn0Xn2, Xn-11? PXn0Xn2, Xn-13
• But
• We can simply construct a Markov chain from
  Model 2. Consider the 5-state model with states
  0,1,3 as before but define
• State 2 40 discount and no claim in previous
  year.
• State 2- 40 discount and claim in the previous
  year.
• This is now a 5 state Markov chain.
• Check!

17
Model NCD 2 Transition Matrix
18
More Complicated NCD Models

• Two possible enhancements to models
• Make accident rate dependent on state (this is of
  course the notion behind this up-dating risk
  assessment system)
• Make the transition probabilities time-dependent
  (a time-inhomogeneous Markov chain) to reflect,
  say, faster motorbikes, younger drivers or
  increasing traffic congestion.

19
Boardwork

• Look at general solution to two-state Markov
  chain.
• Generalise to n-state Markov chain.
• Indicate how to calculate any power of a matrix
  by writing it as PADA-1, for some invertible
  matrix A and where D is a diagonal matrix.
• But, for many applications, with the power a
  reasonable size, simply use a computer to do the
  calculations!
• But what happens in the very long term (i.e.,
  when powers exceed any reasonable integer)?

20
The Long-Term Distribution of a Markov Chain

• Definition We say that ?j, j?State space S, is a
  stationary probability for a Markov chain with
  transition matrix P if
• ? ?P, where ? (?1, ?2,.., ?n), nS
• or, equivalently,

21
The Long-Term Distribution of a Markov Chain

• So if the Markov chain comes across a stationary
  prob. distribution in its evolution then, from
  then on, the distribution of the Xns are
  invariant the chain becomes a stationary
  process from then on.
• It may come across a stationary distribution on
  the first step see example 2 next.
• But, even if a Markov chain has a stationary
  distribution, given some initial conditions
  (i.e., distribution of X0) it might never end up
  in the stationary distribution see example 3.
• In general Markov chains do not have a stationary
  distribution and, if they do, they can have more
  than one.
• The simple random walk does not have a stationary
  distribution.

22
Example 1 Solving for Stationary Distribution

• Consider a chain with only two-states and a
  transition matrix given by . Find its
  stationary distribution.
• Answer (2/5, 3/5).

23
Example 2

• Consider the time-homogeneous Markov chain on
  state space 0,1 with transition matrix P given
  by
• Now PnP for all n.
• Hence lim pij(n) exists.
• Here lim pij(n) ½ as n??.
• In fact, on first step, irrespective of the
  initial distribution of X0, the process reaches
  the stationary distribution.
• Remark A similar example (based on a immediate
  generalisation) can be given for the general
  n-state Markov process.

24
Example 3

• Consider the time-homogeneous Markov chain on
  state space 0,1, given by the following
  transition matrix
• We can solve explicitly for the stationary
  distribution, a find a unique one, ? (0.5, 0.5)
• However, the process will never reach it unless
  (trivially) it starts in the stationary
  distribution
• Obvious from transition graph.

25
Pointers in Solving for Stationary Distributions

• The n equations for the general n-state Markov
  chain, found from the relationship ? ?P, are
  not independent as rows in matrix sum to unity.
• Equivalently, this can be seen by the
  normalisation (or scaling) requirement, which
  says that we are only solving for n-1 unknowns as
  we have
• Hence one can delete one equation without losing
  information.
• Often solve first in terms of one of the ?i and
  then apply normalisation.
• The general solving technique is Gaussian
  Elimination.
• The discarded equation gives check on solution.

26
Exercise NCD 2

• Compute the stationary distribution of NCD model
  2. Recall the transition matrix is given by
• Answer (13/169,12/169,9/169,27/169, 108/169)
• 1/169(13,12,9,27,108)

27
Three Key Questions

• Can we identify those Markov chains that do have
  (at least one) stationary distribution?
• Can we say when the stationary distribution is
  unique?
• Can we determine, given any initial conditions
  (i.e., distribution of X0), whether or not the
  Markov chain will eventually end up in the
  stationary distribution, assuming it to be unique?

28
The Long-Term Distribution of a Markov Chain

• Theorem A Markov chain with a finite state
  space has at least one stationary probability
  distribution.
• Proof NOT ON COURSE

29
When is Solution Unique?

• Definition A Markov chain is irreducible if for
  any i,j, pij(n)gt0 for some n. That is any state j
  can be reached in a finite number of steps from
  any other state i.
• The best way to determine irreducibility is to
  draw the transition graph.
• Examples the NCD models (12) and the simple
  random walk model are all irreducible.

30
When is Solution Unique?

• Theorem An irreducible Markov chain with a
  finite state space has a unique stationary
  probability distribution.
• Proof NOT ON COURSE

31
The Long-Term Behaviour of Markov Chains

• Definition A state i is said to be periodic with
  period dgt1 if pii(n)0 unless n (mod d) 0. If a
  state is not periodic then it is called
  aperiodic.
• If a state is periodic then lim pii(n) does not
  exist as n??.
• Interesting fact an irreducible Markov chain is
  either aperiodic or all its states have the same
  period proved in class.

32
Theorem

• Theorem Let pij(n) be the n-step transition
  probability of an irreducible aperiodic Markov
  chain on a finite state space. Then for every
  i,j,
• Proof NOT ON COURSE
• Importance no matter what the initial state, it
  will converge to (unique) stationary probability
  distribution, i.e., most of the time the Markov
  chain will be arbitrarily close to the stationary
  probability distribution (in the long run).
• The above result is a consequence of Blackwells
  Theorem.

33
Example

• Is the process with the following transition
  matrix irreducible?
• What is the stationary distribution(s) of the
  process?

34
Example

• Is the following Markov chain irreducible? What
  is/are its stationary distribution(s)?

35
Testing Markov Models/Estimation of Parameters

• So we have a real process that we think can
  adequately be modelled using a Markov process.
  Two key problems
• How do I estimate parameters from gathered data?
• How do I check that the model is adequate for my
  purpose?

36
Markov Chain Estimation

• Suppose we have x1,x2,,xN observations of our
  process.
• For time homogeneous Markov chain, the transition
  probabilities can be estimated as
• Where ni is the number of times t, 1?t?(N-1),
  such that xtI
• Where nij is the number of times t, 1?t?(N-1),
  that xti and xt1j.
• Clearly, nijBinomial (Ni, pij), so we can
  calculate confidence intervals for our estimates.

37
Markov Chain Evaluation

• The key property assumed in the model, and to be
  tested against the data, is the Markov property.
• An effective test statistic is the chi-square
  goodness of fit, checking to see that transition
  probability of successive triplets only depend on
  the final transition probability
• Which has sr-q-1 degrees of freedom, where
• s is number of states visited before time N
  (i.e., nigt0)
• q is number of pairs (i,j) where nijgt0
• r is the number of triplets where nijnjkgt0

38
Estimation Evaluation of Time Inhomogeneous
Markov processes

• An order of magnitude more complicated.
• Special techniques usedsee other actuarial
  courses to estimate and test decrements in,
  say, the simple survival model (mortality
  statistics), the sickness model, etc.

39
Ends Chapter 3 Markov Chains
40
Simple Random Walk

• A simple random walk is not just
  time-homogeneous, it is also space-homogeneous,
  i.e.,
• for all k.
• The only parameters affecting the transition
  probabilities for n-steps in a random walk are
  overall distance (j-i) covered and no. of steps
  (n).

41
Simple Random Walk

• Transition matrix given by
• The n-step probabilites are calculated as

42
Simple Random Walk with Boundary Conditions

• Basic model as before but this times with the
  added boundary conditions
• Reflecting boundary at 0 PXn11Xn01
• Absorbing boundary at 0 PXn10Xn01
• Mixed boundary at 0 PXn10Xn0? and
  PXn11Xn01- ?.
• One can, of course, have upper boundaries as well
  as lower ones and both in one model.
• Practical applications prob. of ruin for
  gambler or, with different Zi, a general
  insurance company.

43
Simple Random Walk with Boundary Conditions

• The transition matrix with mixed boundary
  conditions, upper and lower, is given by
• Take ?1 for a lower absorbing barrier (at 0),
  and ?0 for a lower reflecting barrier (at 0).

44
A Model of Accident Proneness

• Let us say that only one accident can occur in
  the unit time period so that Yi is a Bernoulli
  trial (Yi1 or 0 only).
• Now it seems reasonable to put
• i.e., the prob. of an accident at time n1 is a
  function of the past number of claims.
• Also, f(.) and g(.) are increasing functions with
  0?f(m) ?g(m) for all m.
• Clearly the Yis are not Markovian but the
  cumulative number of accidents Xn?Yi is a Markov
  chain with state space 0,1,2,3,

45
A Model of Accident Proneness

• Does it make sense to have a time-independent
  accident proneness model (i.e., g(n) a constant) ?

46
Exercise Accident Proneness Model

• Let f(xn)0.5xn and g(n)n1
• Hence, PYn1Xn(0.5xn)/(n1)
• What is prob.of driver with no accident in first
  year not having an accident in second year too?
• What is prob of an accident in 11th year given an
  accident in each of the previous ten years?
• What is the ijth entry in the one-step transition
  matrix of the Markov chain Xn?

47
Stochastic Processes
48
Exercise

• Let Xn be a time-homogeneous Markov chain. Show
  that
• That is, the kth step transition probability from
  i to j is just ij-entry of the 1-step transition
  matrix (P(1)ij) taken to the power of k (for
  time-homogeneous Markov chain).