Markov Processes

1
Markov Processes
TexPoint fonts used in EMF. Read the TexPoint
manual before you delete this box. AAAAAAA
2
Markov processes

• Stochastic process is Markov process ? prob. of
  future state depends only on present state.
• Formally X(t), tgt 0 is a MP if for any set of
• n1 times t1 lt lt tn1, any set of states,
  x1,xn1,
• Note BD process, Bernoulli process, Poisson
  process are Markov processes

3
Markov Chains

• Discrete-state Markov process is a Markov chain
  (MC)
• Consider first discrete-time MC Xn, n0,1,
• Pij P(Xn1 j Xn, i), i,j 0,1, n gt 0
• Define transition probability matrix
  P Pij. No. of states finite or countably
  infinite.

4

• Q what is state after n transitions?
• A define
• Note that

5

• For n,m gt 1,

6
Markov Chains

• Define n-th step transition probability matrix
• We have
• These are the Chapman Kolmogorov equations
• We conclude
• Q what happens as n ??? does converge to
  something?

7
Equilibrium Behavior of MCs

• Need to study behavior of MC a more carefully
• j reachable from i if for some n
• if i also reachable from j, then i and j
  communicate,
• (i ? j). Note that ? is an equivalence
  relation
• if i,j communicate, they are in the same class.
• MC divides into one or more (equivalence)
  classes.

8
Equilibrium Behavior of MCs

• a MC is irreducible if there exists exactly one
  class.
• i has period d if
• d is largest such integer

9
Transience and Recurrence

• - prob. return to i for first time at time
  n
• -
• -
• prob ever return to
  i
• -- if fi 1, then i is recurrent ,
• -- if fi lt 1, then i is transient
• Note i is recurrent in finite state MC iff for
  every state j such that i reaches j, then i ? j.

10
Recurrence

• mi - mean time to return,
• -
• if mi lt ?, then i is positive recurrent
• if mi ?, then i is null recurrent
• Note all recurrent states in a finite state MC
  are positive recurrent

11
Ergodicity

• Thm Let Xn be an irreducible MC. Then exactly
  one of following holds
• all states are positive recurrent
• all states are null recurrent
• all states are transient
• Note any irreducible finite state MC is positive
  recurrent
• Defn. If a MC is irreducible, aperiodic, and
  positive recurrent, then it is ergodic.
• Thm An ergodic MC has a unique limiting state
  probability distribution, namely ?j , j 0,1,
• ?j also known as stationary prob. distr.,
  steady-state prob. distr., equilibrium prob.
  distr.

12
Ergodicity

• Let ? (?0, ?1, ), then it satisfies
• ? ? P,
• ? 1 1
• where 1 (1, )T

13
Example Statistical Multiplexer

• discrete time system
• An, n 0,1, - iid sequence where An no.
  of packets arriving in time interval (n,n 1)
• let A be rv such that A d An
• one packet transmitted in (n,n1) if there are
  waiting packets at time n

14

• Xn - no. of packets in system at time n
• where (x) 0 if x lt 0, and x otherwise

15
Statistical Multiplexer

• Q is Xn Markov?
• assumption that An is iid implies that An is
  independent of Xn , n 0,1,
• consider Xn 0

16

• consider Xn i gt 0
• It follows that, ? i
• P( Xn1 j X0 i0,... , Xn i)
• P(Xn1 j Xn i)

17
Statistical Multiplexer

• Q what is P
• From proof that system is Markovian
• P looks like

18
Statistical Multiplexer

• Q When is MC ergodic?
• EA gt 1 all states are transient
• EA lt 1 two cases
• - a0 a1 1, states 0,1 are positive
  recurrent, all other are transient
• - a0 a1 lt 1, all states positive recurrent

19

• EA 1, two cases
• - a1 1 state 1 is positive recurrent other
  states transient
• - otherwise, all states null recurrent
• Interesting case is EA lt 1, a0 a1 lt 1 in
  which case stationary distribution ? exists and
  can be obtained as described earlier.

20
Finite Capacity Statistical Multiplexer

• Q what if no more than K packets can be in
  multiplexer at one time?
• system is Markovian
• throughput, Tput 1 - ?0

21
Finite Capacity Statistical Multiplexer

• probability packet arrives to a full system and
  is lost, Pf,
• tricky to compute
• here is indirect approach
• Pf 1 Tput /EA