Continuous Time Markov Chains

1
Continuous Time Markov Chains
A stochastic process is a sequence of random
variables indexed by an ordered set T.
Generally, T records time.
A discrete-time stochastic process is a
sequence of random variables X0, X1, X2, . . .
typically denoted by Xn .
where index T or n takes countable discrete values
Discrete Time Markov chains 1) discrete we do
not care how long each step takes 2) Markov
Property What happens only depends on the
previous state, not on how you get there.
2
Continuous Time Markov Chain
A Continuous Time Markov Chain is a sequence of
random variables Xt , t ? 0 which satisfies the
following
Markovian Property
Pr Xts j Xu, 0 ? u ? s Pr X ts j
Xs
At time s, the future behavior of the system in t
times Xts depends only on the current state
Xs but not on the past Xu, where 0ltults
Stationary Property Pr Xts j Xs Pr X
t j X0
3
Property of Continuous Time Markov Chain
If the state space S 0, 1, . . . , m1.
Suppose we start at state i, and have entered
state i for about s minutes, what is the
probability that a transaction will not occur in
the next t times?
According to Markovian Property
Look at Inter-Arrival Time T
?Pr T i gt t s T i gt s, 0 ? u ? s Pr
T i gt t
There is only one random variable, exponential,
that satisfies this memory-less property
4
Continuous Time Markov Chain Definition
Examples
Poisson Process The Birth/Death Processes in
General The M/M/s queue Brown Motion
Illustration
The M/M/s queue
5
An ATM example (M/M/1/5 Queue)
Consider an ATM located at the foyer of a bank.
Only one person can use the machine, so a queue
forms when two or more customers are present.
The foyer is limited in size and can hold only
five people. When there are more than five
people, arriving customers will balk when the
foyer is full
Statistics indicates that the average time
between arrivals is 30 seconds, or 2 customer per
minutes whereas the time for service averages 24
seconds. or 2.5 customers per minutes. Both times
follows exponential distribution
Try to help the manager of the bank and answer
the following questions.
6
An ATM example (M/M/1/5 Queue)
Managers Questions
a) The proportion of time that the ATM is idle ?
b) The efficiency of the ATM?
c) The throughput rate of the system?
Customers Questions
d) The proportion of customer that obtain
immediate service?
e) The proportion of a customer who arrive and
find the system is full?
f) The average number of time in the system?
7
DTMT Model for the ATM example
What are the State Space of the system
Number of customers in the system 0, 1, 2,
3, 4, 5
For DTMC, we use Transition Matrix, P (BTW,
can you model this problem as a DTMC?)
For CTMC, we use Rate Matrix, R
8
Steady State Probability
What is the long term steady state probability
We will investigate steady-state (not transient)
results for CTMC based on the same Flow Balance
Principle (Rate in Rate out)
Let pn steady-state probability of being in
state n.
p0 ?1 ?2 ?3 ?4 ?5 1
9
Balance Equations
Flow into 0 ? ?p1 ?p0 ? flow out of 0 Flow
into 1 ? ?p0 ?p2 (? ?)p1 ? flow out of 1
Flow into 2 ? ?p1 ?p3 (? ?)p2 ? flow
out of 2 Flow into 5 ?p4 ?p5 ?
flow out of n
10
Rate Matrix Solution with M/M/1/5
Rate Matrix
Solution
11
Solution Analysis
Managers Questions
?0 27
a) The proportion of time that the ATM is idle ?

1??0 73
b) The efficiency of the ATM?
?(1??5) 1.822
c) The throughput rate of the system?
d)What is the average number of customers in the
system?
1?1 2?2 3?3 4?4 5?5 1.868
Customers Questions
d) The proportion of time a customer obtain
immediate service?
?0 27
e) The proportion of a customer find the system
is full?
?5 9
Littles Law, see queuing
f) The average time in the system?
12
Additional Questions
What if we want to add a new ATM machine, what
will the system perform? M/M/2/5
What if we want to add two new ATM machines, what
will the system perform? M/M/3/5
What if we want to add more spaces so that 8
customer can wait. What will the system perform?
M/M/1/8
What if we want to add more spaces so that 12
customer can wait. What will the system perform?
M/M/1/8
What if we want to add a teller them with a
service time exponential distributed at 1 minute
a customer, what will the system perform? This
is not a Standard Queue
13
M/M/2/5
What if we want to add a new ATM machine there,
what will the system perform? M/M/2/5
What will the state-transition network looks like?
14
Rate Matrix Solution with M/M/2/5
Rate Matrix
Solution
15
M/M/3/5
What if we want to add two new ATM machines, what
will the system perform? M/M/3/5
What will the state-transition network looks like?
16
Rate Matrix Solution with M/M/3/5
Rate Matrix
Solution
17
Comparison of different alternativesM/M/1/5 and
M/M/2/5 and M/M/3/5
18
Comparison of different alternativesM/M/1/5 and
M/M/2/5 and M/M/3/5
19
The addition of spaces to the foyer
What if we want to add more spaces so that 8
customer can wait. What will the system perform?
M/M/1/8
What if we want to add more spaces so that 12
customer can wait. What will the system perform?
M/M/1/12
20
Comparison of different alternativesM/M/1/5 and
M/M/2/5 and M/M/3/5
21
Comparison of different alternativesM/M/1/5 and
M/M/2/5 and M/M/3/5
22
Adding a Human Server
The manager decide to add a human teller with a
service time exponential distributed at 1 minute
a customer
However, when a customer enters into the system,
he/she would prefer to go to the human server
first if the server is available?
In this case, how would the system perform?
Approach
Notice you have to differentiate the two server
now
Let us use (HS, MS, waiting) to represent the
system, suppose the foyer can hold at most 5
people
(0,0,0), (0,1,0), (0,1,0), (1,1,0), (1,1,1),
(1,1,2), (1,1,3)
23
Addition of a Human Server
State (HS, MS, waiting)
?
?
?
?
?
(1,0,0)
?m
1,1,1)
1,1,0)
1,1,3)
1,1,2)
(0,0,0)
?h
?h
?m ?h
?m ?h
?m ?h
(0,1,0)
?
?m
?m 2.5, ATM service rate
?h 1, human service rate
24
Addition of Human Server
Rate Matrix
Solution
25
A Queue With Finite Input Sources
A taxi company with a fleet of 6 cabs and a
repair shop to handle breakdowns
Assume that taxis are identical and are
exponential distributed with breakdown rate of
1/3 per month
The company is thinking of setting up several
service bays and the estimate repair time is
exponential distributed with a rate of 4 per month
Do a analysis to help the company to figure out
how many service bay to set up
26
A Queue With Finite Input Sources
One Bay
Two Bay
27
A Queue With Finite Input Sources (1 Bay)
Rate Matrix
Solution
28
A Queue With Finite Input Sources (2 Bay)
Rate Matrix
Solution
29
Economics With These Results
Suppose each taxi on average can bring a revenue
of 1200 a day, what would be the expected
revenue for each configuration?
One Bay 7200??0 6000??14800??23600??3
2400??41200??50??6 6630
Two Bay 7200??0 6000??14800??23600??3
2400??41200??50??6 6392
If it costs 300 dollars a day to operate a bay,
would it be beneficial to the company
30
Probability Transitions Service with Rework
Consider a machine operation in which there is a
0.4 probability that on completion, a processed
part will not be within tolerance.
If the part is unacceptable, the operation is
repeated immediately. This is called rework.
Assume that the second try is always successful
What will the system looks like if
a) Arrivals can occur only when the machine is
idle
b) Arrivals can occur any time
31
Probability Transitions Service with Rework
a) Arrivals can occur only when the machine is
idle
b) Arrivals can occur any time
0.6d1
0.6d1
32
An ATM with a Human Server
Consider an ATM located together with a Human
Server at the foyer of a bank. The foyer is
limited in size and when there are more than five
people, arriving customers will balk
Statistics indicates that the average time
between arrivals is exponential distributed with
an average of 30 seconds, or 2 customer per
minute
The service time of the ATM is exponential
distributed with an average of 24 seconds. or
2.5 customers per minutes.
The service time of human server is exponential
distributed with an average of 1 minute per
customer.
It is further assumed that when a customer enters
into the system, he/she would prefer to go to the
human server first if the server is available.
33
CTMC Model for ATM and Human Server
State (HS, MS, waiting)
?
?
?
?
?
(1,0,0)
?m
1,1,1)
1,1,0)
1,1,3)
1,1,2)
(0,0,0)
?h
?h
?m ?h
?m ?h
?m ?h
(0,1,0)
?
?m
?m 2.5, ATM service rate
?h 1, human service rate
34
CTMC Model for ATM and Human Server
Rate Matrix
Solution
35
The Embedded DTMC in a CTMC (?t interval)
1) Divide Time into Very Small Intervals ?t
Exponential Distribution
The probability of 1 event happening in the next
?t is
?
PrT ?t ) 1- e ? ?t 1 1 (? ?t )?
(? ?t )n/n! )

n2
When ?t is small, (? ?t )n ?0
? ?t
36
The Embedded DTMC in a CTMC (?t interval)
1) Divide Time into Very Small Intervals ?t
Transition Matrix
Let ?t 0.1
37
The Embedded DTMC in a CTMC (?t interval)
Rate Matrix
Solution
Same solution was obtained
38
The Embedded DTMC in a CTMC(Until Next Event
Happens)
2) Set Time Intervals until the next event happens
Rate Matrix
39
Properties of Exponential Distribution
Minimum of Two Exponentials
If X1, X2 , , Xn are independent exponential
r.v.s where Xn has parameter (rate) li, then
min(X1, X2 , , Xn) is exponential with
parameter (rate) l1 l2 ln
Competing Exponentials

• If X1 and X2 are independent exponential r.v.s
  with parameters (rate) l1 and l2 respectively,
  then
• P(x1ltx2) l1/(l1l2)
• That is, the probability X1 occurs before X2 is
  l1/(l1l2)

40
The Embedded DTMC in a CTMC (Until Next Event
Happens)
Transition Matrix
Solution
41
The relations between these two steady state
distributions
(?t interval)
This dist. represents the proportion of time in
each states
Until Next Event Happens
This dist. represents the proportion of Steps in
each states
42
The relations between these two steady state
distributions
The relations of these two distribution is
described in the textbook Page 534 on ORMM
textbook by Jensen and Bard
Interested reader could consult
43
Birth Death Process
Pure Birth Process e.g., Hurricanes
Pure Death Process e.g., Delivery of a
truckload of parcels
Birth-Death Process M/M/s/k/
will be picked up in Queuing Theory
44
Pure Birth Process Poisson Process
Pure Birth Process Poisson Process
Poisson Process Rate Matrix
45
Poisson Process
NO Steady State Probability
The embedded Markov Chain is not ergodic
The number in the system is increasing with time
Transient Probability number of events within
time t
This is a random variable with Poisson
distribution.
A general scheme is what we call a counting
process,
Exponential Random and Poisson Process
46
Pure Death Process
Pure Death Process
NO Steady State Probability
The embedded Markov Chain is not ergodic
The number in the system is decreasing with time
Transient Probability number of events within
time t
This again is a random variable with Poisson
distribution.
47
An Example
Suppose that you arrive at a single teller bank
to find five other customers in the bank. One
being served and the other four waiting in line.
You join the end of the line. If the service
time are all exponential with rate 5 minutes.
Suppose that you arrive at a single teller bank
to find five other customers in the bank. One
being served and the other four waiting in line.
You join the end of the line. If the service
time are all exponential with rate 5 minutes.
What is the prob. that you will be served in 10
minutes ?
What is the prob. that you will be served in 10
minutes ?
What is the prob. that you will be served in 20
minutes ?
What is the prob. that you will be served in 20
minutes ?
What is the expected waiting time before you are
served?
What is the expected waiting time before you are
served?
48
Assumption Revisited
Markov Property Inter-arrival has to
exponential distributed
Arrival and service time
Steady-State Probability Flow Balance ? Rate
in Rate Out Solve a set of linear equations
Arrival time Exponential ?
A large population n, each one has a small
percentage p of entering a store, When n is
large, p is small, exponential is a good
approximation.
49
Assumption Revisited
Service time Exponential ?
Grocery Store, might still be valid
Hair Cut Might not be an exponential now
What if they are not exponential distributed?
Markovian Property does not hold any.
Might not be able to use the rate in rate out
principle
It will be much difficult to get analytical
results, A lot of times, simulation will have to
be used.
50
Problem Sets

• Chapter 14
• Problems 1, 2, 3, 4, 5, 6,11, 13
• The orange ones are homework
• Chapter 15
• Problems 3, 4, 6, 10
• The orange ones are homework

51
A Machine Repair Example
A factory contains two major machines which fails
independently according to a exponential
distribution with a mean time of 10 per hour
The repair of a machine takes an avearage of 8
hours and the repair time is distributed
according to a exponential distribution.
Model the problem as a CTMC or a queuing model
and give analytic results.
52
State-Transition Diagram
? rate at which a single machine breaks down
1/10 hr ? rate at which machines are
repaired 1/8 hr State of the system of
broken machines.
53
Balance Equations for Repair Example
?p1 2?p0 2?p0 ?p2 (?
?)p1 ?p1 ?p2
54
Here, ?0 2? ?1 ? ?1 ? ?2
? ?2 ?
l
l
l
l
2
l
2
2

C1 C2 and C0 1 (by
definition). Thus p0 0.258 ,
p1 p0 0.412
1
0
0



m
m
m
m

m
2
2
1
1
l
2

m
l
2
2
p2
p0 0.330

m
2
L 0p0 1p1 2p2 1.072 (avg machines in
system) Lq 0p1 1p2 0.33 (avg waiting for
repair)
55

?npn ?0p0 ?1p1 ?2p2
(2?)p0 ?p1 0.0928
S
? average arrival rate
ns
1
1
1
1
(0.33)
Wq
Lq
(1.072)
W L
0.0928
0.0928
hours
3.56
11.55 hours
Average amount of time that a machine has to
wait until the repairman initiates the work.
Average amount of time that a machine has to
wait to be repaired, including the time until
the repairman initiates the work.