Markov chain models

1
Markov chain models
2
A CTMC with 3 states
transition rate from A to B
3
Examples of observations of the process
state
time
state
time
4
Meaning of Transition rates
state
time
?AB and ?AC determine distribution of durations
of A bouts and probabilities of subsequent states
5
Durations
termination rate of state A
?A ? hazard rate probability per time unit
that event stop A happens given that it has not
happened before
?AB ? cause specific hazard rate probability
per time unit that stop A and start B happens
6
Distribution of durations
CTMC all termination rates are constant -gt all
bout length distributions are exponential
E.g. log-survivors of bout lengths
-tangens ?A
?B
?C
7
Sequences
pAB
transition probability from A to B
8
Simulating a Markov chain
Choose the initial state, say I Pick an
exponentially (?I) distributed bout length Choose
next state with probability pIJ it is J If the
next state is M Pick an exponentially (?M)
distributed bout length etc...
9
Markov1.R
simulation of a 3-state CTMC creation of a
matrix with the transition rates transitionslt-matr
ix(0,nrow3,ncol3) transitions1,2lt-2
transitions1,3lt-2.5 transitions2,1lt-1transit
ions2,3lt-0.1 transitions3,1lt-0.9transitions
3,2lt-2.1 number of observed bouts nlt-100 matrix
with observations col1 begin time col2
state col3 duration col4 next
state observationlt-matrix(0,nrown1,ncol4) ilt-1
observation1,2lt-1 while(iltn1) statelt-observat
ioni,2 calculation of termination
rate termlt-sum(transitionsstate,13) simulat
ion of duration boutlengthlt-rexp(1,rateterm) o
bservationi,3lt-boutlength calculation of
transition probabilities probslt-transitionsstate
,13/term simulation of next
state nextstatelt-rmultinom(1,1,probs) seqlt-whic
h.max(nextstate) observationi,4lt-seq jlt-i1
observationj,1lt-observationi,1boutlength o
bservationj,2lt-seqilt-j observationlt-observatio
nc(1n),c(14) example selection of state 3
bouts subset(observation,observation,23)
10
Generalizations
11
Function of a CTMC
Result when states are lumped some
transitions unobserved.
A or B ?
12
Observation
state
time
A and B are not distinguished transitions between
A and B are not observed
13
Function of a CTMC
A or B ?
?
Distribution of bout lengths in (A or B) is a
mixture of exponentials Apparent higher order
sequential dependency
14
Apparent higher order dependency
15
Semi-Markov chains
First order sequential dependency Non-exponential
bout length distributions

• e.g.
• - Increasing/decreasing transition rates (Weibull
  model log-normal etc.)
• Minimum bout length
• - Fixed bout duration

16
Semi-Markov chains 2 types
Transition rates proportional
Non-proportional
e.g. ?ABconstant, ?AC increasing
e.g. ?A(t) 0 for t lt ?, constant after ?
17
Inhomogeneous Markov Chains
Transition rates change during observation
Period 1
Period 2
Used to model motivational changes e.g.
drowsiness of infant rhesus monkeys
18
Assignments 1
1.1 Simulate a 3-state Markov chain with the
provided R-program Markov1. Make sure you
understand how the program works. 1.2 Use the
result to simulate a function of a Markov
chain. 1.3 Write a program to simulate a
semi-Markov model with three states, where one
state has a constant termination rate, one state
a monotonically decreasing termination rate (use
the Weibull model), and one state a fixed
duration. 1.4 Write a program to simulate an
inhomogeneous Markov chain where transition rates
change abruptly at a given moment.
19
Fitting Models
20
Time structure of behaviour
day 2
day 1
daily variation
hungry
satiated
motivational changes
time-inhomogeneity
alternation between acts
groom
rest
walk
rest
CTMC/semi Markov
alternation between subacts
scan
pause
scan
pause
21
Outline of analysis
Time inhomogeneity?
split up record
Exponential Markov
Examine bout length distributions
Mixture of exp. redefine acts
Other semi-Markov?
1st order (semi-)Markov
Examine sequential dependencies
Other redefine acts
22
Tests for time inhomogeneity
Graphical cumulative bout length plot
sum
bout No.
Formal Likelihood ratio change point tests
Non-parametric change point tests
23
Tests of exponentiality
Graphical shape of log-survivor plots
Exponential
Minimum duration
Mixture
Increasing termination rate
Formal Kolomogorov-Smirnov test, test against
Weibull distribution, Morans test...
24
Sequential dependency properties

• In (semi-) Markov chains
• At most 1st order dependence in sequence of acts
• Given sequence, bout lengths are independent
• No dependence between bout lengths and preceding
  acts
• In Markov chains
• No dependence between bout lengths and following
  acts
• Deviations can be caused by
• time inhomogeneity inadvertent lumping of acts

25
Tests of sequential dep. prop.
At most 1st order dependence in sequence of
acts Chi-squared test Expected number of
sequences acts X-A-Y if 1st order dep NXA
NAY/NA compare with observed NXAY Given
sequence, bout lengths are independent Auto-correl
ation between subsequent bouts of same
act Cross-correlations between subsequent bouts
of different acts No dependence between bout
lengths and preceding acts Compare bout lengths
of act X preceded by act A with those preceded by
act B, C etc. With a k-sample test, such as
Kruskal-Wallis (nonparametric) or exponential
(parametric)
26
Assignments 2

• 2.1 Make log-survivorplots of the boutlengths
• of all the acts of your simulated models
• function of a Markov chain.
• the semi-Markov chain
• the inhomogeneous Markov chain.
• (see helpfile survivorplot.R)
• 2.3 Make a cumulative bout length plot for the
  inhomogeneous Markov chain and see if you can
  estimate the place of the change point.
• 2.4 Test (at least) one of the sequential
  dependency properties for all of the three
  different models.

27
Parameter estimation and analysis
28
CTMC parameters
Two possible complete characterizations (1)
Transition rates (2) Termination rates plus
transition probabilities
29
M.L. estimators
Termination rate
1/(mean bout length)
proportion of A bouts followed by B
Transition probability
Transition rate
30
Properties of estimators transition rates
For large observation time and/or total number of
bouts
-gt 95 confidence interval
31
Properties of estimators termination rates
For large observation time and/or total number of
bouts
-gt 95 confidence interval
Estimators of termination rates and transition
probabilities are independent
32
Properties of estimators transition probabilities
For large observation time and/or total number of
bouts
-gt 95 confidence interval
33
Two-sample tests for CTMCs
Ho two samples of A bouts have equal termination
rates
Ho two samples of A bouts have equal transition
rates to B
34
Assignments 3
3.1 Simulate two CTMCs with three states each and
make some (not all!) of their transition rates
very different. Take a simulation length of 2000
bouts. Make a note of the parametervalues that
you used. 3.2 Exchange your simulated CTMCs with
a fellow student. Estimate the transition rates
of the two chains and test whether they are
different at a significance level of 5.
35
Analysis of CTMCs
-Ancova on log-transformed transition(term.)
rates NB unequal variances! -Likelihood ratio
tests for full model -With survival analysis
methods and exponential regression.
36
Example of a regression model
Effect of 1 covariate with two levels (1,2) on
the transition rates from state 1 to states 2 and
3
Model
Survival analysis family Exponential
37
Analysis in R
Input
Covariate 1 samples 0 0 0 ... 0 0 0 ... 1 1 1 ...
Covariate 2 next state 0 0 0 ... 1 1 1 ... 0 0 0
...
Bout length t11 t12 t13 ... t11 t12 t13 ... t211 t
221 t231 ... t211 t221
Censor 1 0 1 ... 0 1 0 ... 1 1 0 ...
1 if followed by state 2
state 1 bouts of sample 1
1 if followed by state 3
state 1 bouts of sample 2
38
R code for analysis
library(survival) boutlengthslt-samtot,1 censorslt
-samtot,2 cov1lt-samtot,3 cov2lt-samtot,4 mod
el without interactions modelboutslt-survreg(Surv(b
outlengths,censors)cov1cov2, dist"exponential")
summary(modelbouts) with interaction modelbouts
2lt-survreg(Surv(boutlengths,censors)cov1cov2, di
st"exponential")
39
Assignments 4
4.1 Analyse the transition rates from state 1 for
your two CTMCs in this way. Discuss the results
with the student you exchanged files
with. (Helpfile Markov2) 4.2 Simulate an
inhomogeneous Markov chain of length 1000 bouts,
with a change in the transition rates from state
1 after bout 500. Estimate and test the effects
of the change on transition rates from this state.
40
Semi-Markov parameters
A- bouts followed by B are observed failure
times, other A-bouts are censored Estimation
and analysis with survival analysis methods or
full parametric model and likelihood methods for
semi-Markov chains
41
Assignments 5
The file Markov3.R contains R code for simulating
a model where state 3 is semi-Markov, with
different shape/scale parameters for the
transition rates to states 1 and 2. Make sure
you understand the code, and then simulate two
semi-Markov chains with length 1000 bouts each,
and different transition rates from state 3 to
states 1 and 2 Choose the shape parameters for
samples 1 and 2 equal, i.e. ?31(t) in sample 1
has the same ro value as in sample 2 ?32(t)
idem but make the shape parameters of the two
rates different within each sample. Estimate and
test the differences between the transition rates
of the two samples with a proportional hazards
model. Test the proportionality assumption with a
survivor-plot.
42
Assignments 6

• Simulate two samples of a (generalization of a)
  Markov model,
• which may contain
• EITHER inhomogeneity, lumped states, or
  semi-Markov states (No combinations),
• and diffferences in transition rates between the
  samples
• Exchange them with a fellow student and analyse
  each others results.
• Discuss the results with each other.