Discrete Time Markov Chains

1
Discrete Time Markov Chains
Many real-world systems contain uncertainty and
evolve over time
Stochastic processes (and Markov chains) are
probability models for such systems.
A discrete-time stochastic process is a
sequence of random variables X0, X1, X2, . . .
typically denoted by Xn .
where index T or n takes countable discrete values
2
An Example Problem -- Gamblers Ruin
At time zero I have X0 2, and each day I make
a 1 bet. I win with probability p and lose with
probability 1 p. Ill quit if I ever obtain 4
or if I lose all my money.
Xt amount of money I have after the bet on day
t.
if Xt 4 then Xt1 Xt2 4
if Xt 0 then Xt1 Xt2 0.
The possible values of Xt is S 0, 1, 2, 3, 4
3
Components of Stochastic Processes
The state space of a stochastic process is the
set of all values that the Xts can take. (we
will be concerned with stochastic processes
with a finite of states )
Time t 0, 1, 2, . . . State m-dimensional
vector, s (s1, s2, . . . , sm ) or s (s1,
s2, . . . , sm ) or (s0, s1, . . . , sm-1)
Sequence Xt, Xt takes one of m values, so Xt ? s.
4
Markov Chain Definition
A stochastic process Xt is called a Markov
Chain if Pr Xt1 j X0 k0, . . . , Xt-1
kt-1, Xt i Pr Xt1 j Xt i ?
transition probabilities for every i, j, k0,
. . . , kt-1 and for every t.
The future behavior of the system depends only on
the current state i and not on any of the
previous states.
5
Stationary Transition Probabilities
Stationary Markov Chains
Pr Xt1 j Xt i Pr X1 j X0 i
for all t (They dont change over time) We will
only consider stationary Markov chains.
The one-step transition matrix for a Markov chain
with states S 0, 1, 2 is
where pij Pr X1 j X0 i
6
Property of Transition Matrix
If the state space S 0, 1, . . . , m1 then
we have ?j pij 1 ? i and pij ? 0 ? i,
j
(we must (each transition
go somewhere) has prob ?
0)
Stationary Property assumes that these values
does not change with time
7
Transition Matrix of the Gamblers problem
At time zero I have X0 2, and each day I make
a 1 bet. I win with probability p and lose with
probability 1 p. Ill quit if I ever obtain 4
or if I lose all my money.
Xt amount of money I have after the bet on day
t.
Transition Matrix of Gamblers Ruin Problem
8
State Transition Diagram
State Transition Diagram Node for each
state, Arc from node i to node j if pij gt 0
The state-transition diagram of Gamblers problem
Notice nodes 0 and 4 are trapping node
9
Printer Repair Problem

• Two printers are used for Russ Center.
• When both working in morning, 30 chance that one
  will fail by evening and a 10 chance that both
  will fail.
• If only one printer is serving at the beginning
  of the day, there is a 20 chance that it will
  fail by the close of business.
• If neither is working in the morning, the office
  sends all work to a printing service.
• The repair takes one day and the printer will be
  returned earlier the next day

10
States for Computer Repair Example
11
Events and Probabilities for Computer Repair
Example
12
State-Transition Matrix and Network
State-Transition Matrix
The major properties of a Markov chain can be
described by the m ? m matrix
P (pij). For printer repair example ?
State-Transition Network Node for each
state, Arc from node i to node j if pij gt 0.
For printer repair example ?
13
Market Share/Brand Switching Problem
Market Share Problem
You are given the original market of three
companies. The following table gives the number
of consumers that switches from brand i to brand
j in two consecutive weeks
How to model the problem as a stochastic process ?
14
Empirical Transition Probabilities for Brand
Switching, pij
Transition Matrix
15
Assumption Revisited

• Markov Property

Pr Xt1 j X0 k0, . . . , Xt-1 kt-1, Xt
i Pr Xt1 j Xt i ?
transition probabilities for every i, j, k0,
. . . , kt-1 and for every t.

• Stationary Property

Pr Xt1 j Xt i Pr X1 j X0 i
for all t (They dont change over time) We will
only consider stationary Markov chains.
16
Transform a Process to a Markov Chain
Sometimes a non-Markovian stochastic process can
be transformed into a Markov chain by expanding
the state space.
Example Suppose that the chance of rain
tomorrow depends on the weather conditions for
the previous two days (yesterday and today).
Specifically, P rain tomorrow?rain last 2 days
(RR) .7 P rain tomorrow?rain today but not
yesterday (NR) .5 P rain tomorrow?rain
yesterday but not today (RN) .4 P rain
tomorrow?no rain in last 2 days (NN) .2
Does the Markovian Property Hold ??
17
The Weather Prediction Problem
How to model this problem as a Markovian Process
??
The state space
0 (RR) 1 (NR) 2(RN) 3(NN)
The transition matrix
0 (RR) 1 (NR) 2(RN) 3(NN)
0 ( RR) .7 0 .3 0 P 1 (NR) .5 0 .5 0 2
(RN) 0 .4 0 .6 3 (NN) 0 .2 0 .8
This is a Discrete Time Markov Process
18
Repair Operation Takes Two Days
One repairman, two days to fix computer. ? new
state definition required s (s1, s2) s1
number of days the first machine has been in the
shop s2 number of days the second machine has
been in the shop
For s1, assign 0 if 1st machine has not failed
1 if it is in the first day of repair 2 if it
is in the second day of repair For s2, assign 0
or 1
19
State Definitions for 2-Day Repair Times
20
State-Transition Matrix for 2-Day Repair Times
0 1 2 3 4
21
Choosing Balls from an Urn
An urn contains two unpainted balls at present.
We choose a ball at random and flip a coin.
If the chosen ball is unpainted and the coin
comes up heads, we paint the chosen unpainted
ball red
If the chosen ball is unpainted and the coin
comes up tails, we paint the chosen unpainted
ball blue.
If the ball has already been painted, then
(whether heads or tails has been tossed), we
change the color of the ball (from red to blue or
from blue to red)
Model this problem as a Discrete Time Markov
Chain (represent it using state diagram
transition matrix)
22
Multi-step (t-step) Transitions
Example IRS auditing problem
Assume that whether a tax payer is audited by IRS
or not in the n 1 is dependent only on whether
he was audit in the previous year or not.

• If he is not audited in year n, he will not be
  audited with prob 0.6, and will be audited with
  prob 0.4

• If he is audited in year n, he will be audited
  with prob 0.5, and will not be audited with prob
  0.5

How to model this problem as a stochastic process
?
23
An IRS Auditing Problem
State Space Two states s0 0 (no audit), s1
1 (audit)
Transition matrix
Transition Matrix P is the prob. of transition in
one step
How do we calculate the probabilities for
transitions involving more than one step?
Notice p01 0.4, is conditional probability of
audit next year given no audit this year.
p01 p (x1 1 x0 0)
OR
24
2-Step Transition Probabilities
p (x2 0x0 0)
p (x1 1x0 0) ? p (x2 0x1 1)
p (x1 0x0 0) ? p (x2 0x1 0)

25
n-Step Transition Probabilities
This idea generalizes to an arbitrary number of
steps P(3) P(2) P P2 P P3 or more
generally P(n) P(m) P(n-m)
The ij'th entry of this reduces to Pij(n) ?
Pik(m) Pkj(n-m) 1 ? m ? n?1
m
k0
Chapman - Kolmogorov Equations
The probability of going from i to k in m steps
then going from k to j in the remaining n?m
steps, summed over all possible intermediate
states k
26
Transition Probabilities for n Steps
Property 1Let Xn n 0, 1, . . . be a Markov
chain with state space S and state-transition
matrix P. Then for i and j ? S, and n 1, 2, .
. . PrXn j X0 i pij, where the
right-hand side represents the ijth element of
the matrix P(n) and
P(n) P ? P ? ? P
(n)
n
27
Conditional vs. Unconditional Probabilities
Perform the following calculations q(t)
q(0)P(t) or q(t) q(t1)P where q(0) is
initial unconditional probability. The
components of q(t) are called the transient
probabilities.
28
Brand Switching Example
The initial unconditional qi(0) can be obtained
by dividing total customers using brand i by
total sample size q(0) (125/500, 230/500,
145/500) (0.25, 0.46, 0.29)
To predict market shares for, say, 2 weeks into
the future, we simply apply equation with t 2
q(2) q(0)P(2)
(0.327, 0.406, 0.267) expected market share
from brands 1, 2, 3
29
Steady-State Solutions n Steps
What happens with t get large?
the IRS example.
30
Steady-State Solutions n Steps
the IRS example -- Continued.
31
Steady State Transition Probability
Observations as n gets large, the values in row
of the matrix becomes identical OR they
asymptotically approach a steady state value
What does it mean?
The probability of being in any future state
becomes independent of the initial state as time
process
? j limn?? Pr (Xnj X0i limn?? pij (n)
for all i and j
These asymptotical values are called
Steady-State Probabilities
32
Compute Steady-State Probabilities
Let p (p1, p2, . . . , pm) is the
m-dimensional row vector of steady-state
(unconditional) probabilities for the state space
S 1,,m.
Brand switching example
p1 p2 p2 1, p1 ? 0, p2 ? 0, p3 ? 0
Solve linear system p pP,
?pj 1, pj ? 0, j 1,,m
33
Steady-State Equations for Brand Switching Example
p1 0.90p1 0.02p2 0.20p3 p2 0.07p1
0.82p2 0.12p3 p3 0.03p1 0.16p2 0.68p3 p1
p2 p3 1 p1 ? 0, p2 ? 0, p3 ? 0
Total of 4 equations in 3 unknowns.

• Discard 3rd equation and solve the remaining
  system
• We get
• p1 0.474, p2 0.321, p3 0.205
• ? Recall q1(0) 0.25, q2(0) 0.46, q3(0)
  0.29

34
Comments on Steady-State Results
1. Steady-state predictions are never achieved in
actuality due to a combination of (i) errors in
estimating P, (ii) changes in P over time, and
(iii) changes in the nature of dependence
relationships among the states. Nevertheless,
the use of steady-state values is an important
diagnostic tool for the decision maker.
2. Steady-state probabilities might not exist
unless the Markov chain is ergodic.
35
A stead State Does not Always Exist-- Gamblers
Ruin Example
For the Gamblers Problem, assume p 0.75, t
30
0 1 2 3 4 0 1 0 0 0 0 1 .325
0 0 0 .675 2 .1 0 0 0 .9 3
.025 0 0 0 .975 4 0 0 0 0 1
P(30)
What does matrix mean?
A Steady State Probability Does Not Exist in This
Case
36
Existence of Steady-State Probabilities
A Markov chain is ergodic if it is aperiodic and
allows the achievement of any future state from
any initial state after one or more transitions.
If these conditions hold, then
37
Classification of States in Markov Chain
Example
38
A state j is accessible from state i if pij(t) gt
0 for some t gt 0. In example, state 2 is
accessible from state 1 state 3 is
accessible from state 5 but state 3 is not
accessible from state 2.
States i and j communicate if i is accessible
from j and j is accessible from i. States 1 2
communicate also states
3, 4 5 communicate. States 2 4 do not
communicate
States 1 2 form one communicating class.
States 3, 4 5 form a 2nd communicating class.
39
If all states in a Markov chain communicate
(i.e., all states are members of the same
communicating class) then the chain is
irreducible.
The current example is not an irreducible Markov
chain.
How about the Gamblers Ruin Problem
Not an irreducible Markov Chain. It has 3
classes 0, 1, 2, 3 and 4.
Let fii probability that the process will
return to state i (eventually)
given that it starts in state i. If fii 1
then state i is called recurrent. If fii lt 1
then state i is called transient.
40
If pii 1 then state i is called an absorbing
state. Above example has no
absorbing states States 0 4 are absorbing in
Gamblers Ruin problem.
The period of a state i is the smallest n gt 1
such that all paths leading back to i have a
length that is a multiple of n i.e., pii(t)
0 unless t n, 2n, 3n, . . . If a process can
be in state i at time t or time t 1 having
started at state i then state i is aperiodic.
Each of the states in the current example are
aperiodic
41
Example of Periodicity - Gamblers Ruin
States 1, 2 and 3 each have period 2.
0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0 2 0
1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1
If all states in a Markov chain are recurrent,
aperiodic, the chain is irreducible then it
is ergodic.
42
Classification of States (continued)
An absorbing state is one that locks in the
system once it enters.
This diagram might represent the wealth of a
gambler who begins with 2 and makes a series of
wagers for 1 each. Let ai be the event of
winning in state i and di the event of losing in
state i. There are two absorbing states 0 and 4.
43
Illustration of Concepts
Example 1
Every pair of states communicates, forming a
single recurrent class however, the states are
not periodic. Thus the stochastic process is
aperiodic and irreducible.
44
Illustration of Concepts
Example 2
States 0 and 1 communicate and for a recurrent
class. States 3 and 4 form separate transient
classes. State 2 is an absorbing state and forms
a recurrent class.
45
Illustration of Concepts
Example 3
Every state communicates with every other state,
so we have irreducible stochastic process.
Periodic?
Yes, so Markov chain is irreducible and periodic.
46
Existence of Steady-State Probabilities
A Markov chain is ergodic if it is aperiodic and
allows the achievement of any future state from
any initial state after one or more transitions.
If these conditions hold, then
Conclusion chain is ergodic.
47
Game of Crap
The Game of Crap in Las Vegas plays as follows
The player rolls a pair of dice and sums the
numbers showing.
A total of 7 or 11 on the first rolls wins for
the player
Where a total of 2, 3, 12 loses
Any other number is called the point.
The player rolls the dice again.
If he/she rolls the point number, she wins
If he/she rolls number 7, she loses
Any other number requires another roll
The game continues until he/she wins or loses
48
Game of Crap as a Markov Chain
All the possible states
Start
Lose
Win
P4
P5
P6
P8
P9
P10
Continue
49
Game of Craps Network
50
Game of Craps
Probability of win Pr 7 or 11 0.167
0.056 0.223 Probability of loss Pr 2, 3, 12
0.028 0.56 0.028 0.112
51
Transient Probabilities for Craps
0.026


Recall, This is not an ergodic Markov Chain
Where you start counts
52
Absorbing State Probabilities for Craps
53
Interpretation of Steady-State Conditions

• Just because an ergodic system has steady-state
  probabilities does not mean that the system
  settles down into any one state.

2. ?j is simply the likelihood of finding the
system in state j after a large number of steps.
3. The limiting probability pj that the process
is in state j after a large number of steps is
also equals the long-run proportion of time that
the process will be in state j.
4. When the Markov chain is finite, irreducible
and periodic, we still have the result that the
pj, j Î S, uniquely solves the steady-state
equations, but now pj must be interpreted as the
long-run proportion of time that the chain is in
state j.
54
Insurance Company Example
An insurance company charges customers annual
premiums based on their accident history in
the following fashion

• No accident in last 2 years 250
  annual premium
• ? Accidents in each of last 2 years 800 annual
  premium
• ? Accident in only 1 of last 2 years 400 annual
  premium

• Historical statistics
• If a customer had an accident last year then they
  have a 10 chance of having one this year
• If they had no accident last year then they have
  a 3 chance of having one this year.

55
Find the steady-state probability and the
long-run average annual premium paid by the
customer.
Solution approach Construct a Markov chain with
four states (N, N), (N, Y), (Y, N), (Y,Y) where
these indicate (accident last year, accident
this year).
56
State-Transition Network for Insurance Company
.03
.90
.90
.03
.97
.10
.97
.10
This is an ergodic Markov chain All states
communicate (irreducible) Each state is
recurrent (you will return, eventually) Each
state is aperiodic.

57
Solving the steady state equations
?(N,N) 0.97 ?(N,N) 0.97 ?(Y,N) ?(N,Y)
0.03 ?(N,N) 0.03 ?(Y,N) ?(Y,N) 0.9
?(N,Y) 0.9 ?(Y,Y) ?(N,N) ?(N,Y)?(Y,N)
?(Y,Y) 1

Solution ?(N,N) 0.939, ?(N,Y) 0.029, ?(Y,N)
0.029, ?(Y,Y) 0.003
the long-run average annual premium
is 0.939250 0.029400 0.029400 0.003800
260.5
58
First Passage Times
Let ?ij expected number of steps to
transition from state i to state j If
the probability that we will eventually visit
state j given that we start in i is less
than one then we will have ?ij ?.
For example, in the Gamblers Ruin problem,
?20 ? because there is a positive probability
that we will be absorbed in state 4 given
that we start in state 2 (and hence visit
state 0).
59
Computations for All States Recurrent
If the probability of eventually visiting state
j given that we start in i is 1 then the
expected number of steps until we first visit
j is given by
We go from i to r in the first step with
probability pir and it takes mrj steps from r to
j.
It will always take at least one step.
For j fixed, we have linear system in m equations
and m unknowns mij , i 0,1, . . . , m1.
60
First-Passage Analysis for Insurance Company
Suppose that we start in state (N,N) and want to
find the expected number of years until we have
accidents in two consecutive years (Y,Y). This
transition will occur with probability 1,
eventually.
For convenience number the states 0 1
2 3 (N,N) (N,Y) (Y,N) (Y,Y) Then,
?03 1 p00 ?03 p01 ?13 p02?23 ?13
1 p10 ?03 p11 ?13 p12?23 ?23 1
p20 ?03 p21 ?13 p22?23
61
(N, N) (N, Y) (Y, N) (Y, Y)
(N, N) .97 .03 0 0 (N, Y) 0 0 .90 .10 (Y,
N) .97 .03 0 0 (Y, Y) 0 0 .90 .10
0 1 2 3
Using P
?03 1 0.97?03 0.03?13 ?13 1
0.9?23 ?23 1 0.97?03 0.03?13
Solution ?03 343.3, ?13 310, ?23
343.3
So, on average it takes 343.3 years to transition
from (N,N) to (Y,Y).
62
Expected First Passage Times
The average time it takes to reach other states
Recurrence time first passage time from a state
back to itself
It is the inverse of the
steady state probability
uii 1/?i
63
Absorbing States
An absorbing state is a state j with pjj 1.
Given that we start in state i, we can
calculate the probability of being absorbed in
state j. We essentially performed this
calculation for the Gamblers Ruin problem by
finding
P(t) (pij(t)) for large t.
But we can use a more efficient analysis like
that used for calculating first passage times.
64
Let 0, 1, . . . , k be transient states and k
1, . . . , m 1 be absorbing states.
Let qij probability of being absorbed in state
j given that we start in
transient state i. Then for each j we have the
following relationship
qij pij ? pirqrj , i 0, 1, . . . , k

k r0
Go directly to j Go to r and then to j
For fixed j (absorbing state) we have k1 linear
equations in k1 unknowns, qrj , i 0, 1, . . .
, k.
65
Absorbing States Gamblers Ruin
Suppose that we start with 2 and want to
calculate the probability of going broke, i.e.,
of being absorbed in state 0. We know p00 1
and p40 0, thus q20 p20 p21 q10 p22
q20 p23 q30 ( p24 q40) q10 p10
p11 q10 p12 q20 p13 q30 0 q30
p30 p31 q10 p32 q20 p33 q30
0 where P
0 1 2 3 4 0 1 0 0 0 0 1 1-p 0 p 0 0
2 0 1-p 0 p 0 3 0 0 1-p 0 p 4 0 0 0 0 1
66
Solution to Gamblers Ruin Example
Now we have three equations with three
unknowns. Using p 0.75 (probability of winning
a single bet) we have q20 0 0.25 q10
0.75 q30 q10 0.25 0.75 q20 q30 0
0.25 q20 Solving yields q10 0.325, q20
0.1, q30 0.025 (This is consistent with the
values found earlier.)
67
Applications of Markov Chain(Reading)
Linear Programming Min cx
Subject to Ax b, x ?0 where A is
a m ?n matrix, n variables, m constraints. n?m
Simplex algorithm Search along the boundary for
improving extreme points (vertex)
,exponential , of vertexes,
There might be as many as
It seems that simplex algorithm, on average, need
exponential number of iterations?
NO, see a Markov Chain Model for simplex
Algorithm (Handout)
68
Economic Analysis

• Two kinds of economic effects
• those incurred when the system is in a specified
  state,
• those incurred when the system makes a transition
  from one state to another.

The cost (profit) of being in a particular state
is represented by the m-dimensional column vector

where each component is the cost associated with
state i.
The cost of a transition is embodied in the m ? m
matrix
where each component specifies the cost of going
from state i to state j in a single step.
69
n-Step Transition Probabilities

Property 1 Let Xn n 0, 1, . . . be a
Markov chain with state space S and
state-transition matrix P.
Let
denote the probability of being in state j
after n steps given that we started
in state i

Then, we have
(n)
(n)
(n)


p

p

p
00
01
02
(n)
(n)
(n)
P(n)


p
P ? P ? P ? P
p
p

10
12
11
(n)

(n)
(n)
p
p
p
20
21
22
(n)
Notice pij be conditional t-step transition
probability that is, PrXn j
X0 i
70
Expected Cost for Markov Chain
Expected cost of being in state i
Let C (c1, . . . cm)T ei (0, 0, 1, 0, 0) be
the ith row of the m ? m identity matrix, and ft
a random variable representing the economic
return associated with the stochastic process at
time t.
Property 3 Let Xt t 0, 1, . . . be a
Markov chain with finite state space S,
state-transition matrix P, and expected state
cost (profit) vector C. Assuming that the
process starts in state i, the expected cost
(profit) at the tth step is given by  Eft X0
i eiP(t)C.
71
Additional Cost Results
What if the initial state is not known? Property
5Let Xt t 0, 1, . . . be a Markov chain
with finite state space S, state-transition
matrix P, initial probability vector q(0), and
expected state cost (profit) vector C. The
expected economic return at the tth step is given
by   Eft q(0) q(0)P(t)C.
Property 6Let Xt t 0, 1, . . . be a Markov
chain with finite state space S, state-transition
matrix P, steady-state vector p, and expected
state cost (profit) vector C. Then the long-run
average return per unit time is given by   Si?S
pici pC.