INTRODUCTION TO MARKOV MODELS OUTLINES

1
INTRODUCTION TO MARKOV MODELSOUTLINES
DECISION TREES AND MARKOV MODELS Markov model is
a recursive (repetitive) decision tree that is
for modeling conditions that have events that may
occur repeatedly over time or for modeling
predictable events that occur over time
(e.g., screening for disease at fixed
intervals) e.g., Cycling among heart failure
classes or screening for colerectal cancer Use of
Markov model simplifies the presentation of the
tree structure Markov model explicitly accounts
for timing of events, whereas time usually is
less explicitly accounted for in decision trees
2
BUSHINESS OF RECURSIVE TREES
If there are 4 outcomes that can occur each
period (or cycle), the decision tree will have 4C
end nodes (where C equals the number of cycles)
3
STATE TRANSITION OR MARKOV MODELS (I) Develop a
description of the disease (e.g., a system) by
simplifying it into a series of states e.g.,
models of heart failure (HF) might be constructed
with five or six health states Five state model
(if everyone in the model begins with HF) HF
subdivided into New York Heart Association (NYHA)
classes I through 4, and death (either from heart
failure or other causes) Six state model (if the
model predicts onset of disease) No disease, HF
subdivided into New York Heart Association (NYHA)
classes I through 4, and death (either from heart
failure or other causes). Disease progression
(e.g., dynamics of the system) described
probabilistically as a set of transitions among
the states in periods, often of fixed duration
(e.g., months, years, etc.)
4
Transition likelihoods defined as a set of
transition probabilities (Pi,j) where P
probability, i the state one was in during
period t and j represents the state one moves to
in t1 STATE TRANSITION OR MARKOV MODELS
(II) Assess outcomes such as resource use, cost,
and QALYs based on the resource use, cost and
QALY weight experienced From being in a state
for a period (e.g., average cost of being in NYHA
class 1 for a year) or From making a transition
from one state to another (e.g., average cost
among patients who begin a period in NYHA class 1
and begin the next period in NYHA class 2)
5
Develop a mathematical description of the effects
of an intervention as a change in The transition
probabilities among the states (e.g., by reducing
the probability of death) or The outcomes within
the states (e.g., after the intervention, those
in NYHA class 1 cost 500 less than do those
without the intervention) State Transition Model
NYHA Class and Death
6
Necessary steps in Developing Markov Model -
Medical Applications STEPS IN DEVELOPING MARKOV
MODEL 1. Enumerate the states 2. Define allowable
state transitions 3. Associate probabilities with
the transitions 4. Identify a cycle length 5.
Identify an initial distribution of patients
within the states 6. Identify the outcome
values 7. Analyze the Markov model 8. Perform
sensitivity analysis
7
SYSTEMATIC LUPUS ERYTHEMATOSUS (SLE) (I) The
example used here is a Markov model predicting
prognosis in SLE (1) The study sample 98
patients followed from 1950-1966 (the steroid
period), 58 of whom were treated with
steroids All patients were seen more than once
and were followed at least yearly until death or
study termination No patient was lost to
follow-up Time 0 was time of diagnosis Silverstein
MD, Albert DA, Hadler NM, Ropes MW. Prognosis in
SLE comparison of Markov model to life table
analysis. J Clin Epi. 198841623-33. SYSTEMATIC
LUPUS ERYTHEMATOSUS (SLE) (II)
8
Diagnosis of SLE was based on the presence of 3
of the 4 following criteria Skin rash Nephritis
(based on the recording of urinary sediment
abnormality, with greater than 2 proteinuria on
two or more successive visits) Serositis Joint
involvement All patients would have fulfilled the
ARA diagnostic criteria for SLE (which have a
sensitivity and specificity of 90 and 91
respectively) A set of 11 clinical findings and 9
laboratory values were used to classify patients
disease into four severity grades, 1 through 4
9
STEPS IN DEVELOPING MARKOV MODEL Step 1.
Enumerate the states Markov models usually
conceptualized as being made up of crisp
sets/states Under the assumption of crisp states
States must be all inclusive and mutually
exclusive (all patients must be in one and only
one state at all times in the model) Clearly
defined, usually according to standard
literature-based notions of disease Distinguished
by their prognosis or transition
probabilities Transition probabilities per unit
time estimable from data or the literature Able
to assign costs / outcome weights (e.g., QALYs,
etc.) IDENTIFY A SET OF STATES FOR MODELING
SYSTEMIC LUPUS
10
Four disease states State 1 Remission No
disease activity State 2 Active Severity grades
1 through 3 State 3 Flare Severity grade 4 State
4 Death (from any cause) Each patient year was
classified by the greatest severity of disease
activity during the year, even if severity was
only present during a portion of the year e.g., a
patient whose disease activity was severity grade
4 during any visit in a calendar year was
considered to have a flare year No patient was
observed to have more than 1 flare per year and
all patients were seen at least once a year
11
STEPS IN DEVELOPING MARKOV MODEL Step 2. Define
allowable state transitions (e.g., in Lupus may
not want to allow improvements from flare to
remission) Nonabsorbing states once in the
state, one can move out of it Absorbing states
once in the state, one cannot move out of it
(e.g., death) DEFINE THE ALLOWABLE TRANSITIONS
FOR A MODEL OF SYSTEMIC LUPUS

12
Should one, instead, conceptually describe all
transitions, and allow the data to define the
ones that actually are observed (i.e., those that
are not observed will have a probability of
0.0)? TYPES OF SYSTEMS Closed System - Movement
occurs among the states in the system, but the
number of individuals in it is the same all the
time. Leakage from the system - Movement occurs
among the states, but some individuals are lost
(out migration) Open system - Recruitment (in
migration and births) and leakage STEPS IN
DEVELOPING MARKOV MODEL Step 3. Associate
probabilities with the transitions Published
reports of clinical and epidemiological studies
13

• Intervention Study
• Underlying disease course
• Effect of interventionbservational and
  Intervention Study (common)
• Underlying disease course observational study
• Intermediate effect of intervention intervention
  study
• Final effect of intervention observational or
  intervention study
• "Expert" Opinion (source of last resort)
• TYPES OF MARKOV MODELS
• Markov Chain - Transition probabilities constant
  over time (e.g., constant mortality rates) has
  special properties when analyzing the model
• Most appropriate for steady-state open systems or
  when approximately true

14
Markov process - Transition probabilities vary
over time (time dependent transition
probabilities) Absorbing Markov process - Markov
model with time-dependent transition
probabilities, where everyone eventually ends in
one of the absorbing states (e.g., death) LUPUS
PROBABILITIES Suppose you had data from a lupus
registry that was following 98 patients.
Observations were made at the beginning and end
of each year. During the period of observation,
100 patients began a year in remission, 935 began
a year with active disease, and 80 began a year
with a flare Suppose that among the 100 who began
a year in remission, 59 were still in remission
at the end of the year, 41 were experiencing
active disease, and none were experiencing a
flare or had died
15
Suppose that among the 935 with active disease at
the beginning of the year, 66 were in remission
at the end of the year, 806 were still
experiencing active disease, 56 were experiencing
a flare, and 7 had Died Suppose that among the 80
experiencing a flare at the beginning of the
year, 0 were in remission at the end of the year,
22 were experiencing active disease, 18 were
still experiencing a flare, and 40 had died !What
are the annual transition probabilities? LUPUS
PROBABILITIES (II)
16
Counts are approximations of actual data (not
provided in article)
17
LUPUS PROBABILITIES (III) Probabilities
routinely reported as a matrix Matrix routinely
partitioned between non absorbing and absorbing
states (if there is one absorbing state, all
information is captured by the transition
probabilities among non absorbing states) Matrix
routinely partitioned between nonabsorbing and
absorbing states (if there is one absorbing
state, all information is captured by the
transition probabilities among nonabsorbing
states)
18
Time t1 Nonabsorbing Absorbing Time t
Remis. Active Flare Dead Remission 0.59
0.41 0.00 0.00 Active 0.07 0.86 0.06
0.01 Flare 0.00 0.27 0.23 0.50 Dead
0.00 0.00 0.00 1.00
HYPOTHETICAL LUPUS INTERVENTION
PROBABILITIES magine a hypothetical intervention
that has the following effect on the probability
of being in remission and does not change the
costs or QALYS associated with being in a state
or making transitions among states
19
Time t1 Nonabsorbing Absorbing Time t
Remis. Active Flare Dead Remission 0.65
0.35 0.00 0.00 Active 0.07 0.86 0.06
0.01 Flare 0.00 0.27 0.23 0.50 Dead 0.00
0.00 0.00 1.00
MARKOV PROBABILITY MATRICES By convention, the
rows of the transition matrix represent the
individuals state at the beginning of the period
(often referred to as time t) The columns
represent the individuals state at the beginning
of the next period (often referred to as time
t1). Ex. The probability in row 1, column 2 is
the probability that the person starts out in
state 1 and ends up in state 2 after the cycle By
convention, the death state is often omitted from
the matrix (which is represented by the lines in
the matrix)
20
What if the row probabilities sum to one? If they
do not? PROBABILITY ESTIMATION Large number of
methods exist for estimating transition
probabilities Simple methods as suggested in
Lupus example If available data are hazard rates
(i.e., instantaneous failure rates) per unit of
time (Rijt), then can be translated into
probabilities as follows
where Pij (t) equals the probability of moving
from state i at the beginning of period t to
state j at the beginning of period t1 Rij
equals the instantaneous hazard rate per period
(e.g., per year) and t equals the length of the
period Logistic regression (multivariate or
multivariable) / survival analysis Case/control
designs problems similar to those encountered
with interpretation of intercept for logistic
regression requires a set of sample weights
21
STEPS IN DEVELOPING MARKOV MODEL Step 4. Identify
a cycle length Currently accepted
practice Strategy 1 Have the cycle length
approximate clinical follow-up Strategy 2 Allow
the cycle length to be determined by the study
question or available data ignore differences
that dont make a difference In the prediction of
risks for primary CHD, how much more fidelity is
gained by annual versus 5-yearly cycles CONSTANT
CYCLE LENGTH A common recommendation -- and a
design feature of many software packages -- is to
maintain a constant cycle length for the duration
of the model Appropriate when age or time with
disease is not the same for all individuals being
modeled For models of inception cohorts (e.g.,
birth cohort), it is possible to relax the
requirement of a constant cycle length If you are
modeling hepatitis b in a birth cohort, there
might be periods in the subjects lives where
incidence and progression rates vary more and
less E.g., might use shorter cycles in
new-borns, adolescents, and patients in their
twenties and thirties, and longer cycles among
those older than these ages
22
LUPUS CYCLE LENGTH Current probabilities are for
annual cycles CYCLE LENGTH ISSUES Suppose there
are multiple transitions within a single
cycle? Option 1 Do not change the cycle length,
but capture their effects as part of the payoff
for the cycle I.e., if one is expected to expend
small lengths of time in worse disease states,
increase the average costs and reduce the average
QOL associated with spending a period in the
state Option 2 Use Tunnel States Option 3
Shorten the cycle length STEPS IN DEVELOPING
MARKOV MODEL Step 5. Identify an initial
distribution of patients within the states Use a
population approach e.g., one might want to use
the distribution in which patients present to the
registry Remis Active Flare 0.10 0.85
0.05
23
Alternatively, start everyone in one state,
(e.g., to determine what will happen to patients
who begin in remission, make the probability of
being in remission 1.0) Remis Active
Flare Remission 1.0 0.0 0.0 Active 0.0
1.0 0.0 Flare 0.0 0.0 1.0 HYPOTHETICAL
LUPUS INITIAL DISTRIBUTION Remission
0.10 Active 0.85 Flare 0.05 STEPS IN
DEVELOPING MARKOV MODEL Step 6. Identify the
outcome values Basic result of model calculation
is years of survival in the different states Also
should identify Costs of being in a state or of
making a transition from one state to another
state
24
Health outcomes other than survival (e.g.,
quality-adjusted life expectancy) that will be
assessed OUTCOMES FOR TRANSITIONS VS
STATES Outcomes can be modeled as a function of
making a transition from one state to
another i.e., number of hospitalizations or QALYs
experienced by patients who at the beginning of
time t are state I and at the beginning of time
t1 are in state j (e.g., the transition from
remission to active disease) They also can be
modeled as a function of being in a state for a
period i.e., number of hospitalizations
experienced during time t by patients who at the
beginning of time t are in state I (independent
of the transition they make by the beginning of
time t1) LUPUS OUTCOME VARIABLES (I)
25
Hypothetical Cost Data Costs modeled as of
hospitalizations Hospitalizations modeled
based on probability of making a transition from
one state to another state Hypothetical numbers
of hospitalizations Remis. Active Flare
Death Remission 0.05 0.25 0.00
0.00 Active 0.10 0.20 1.00 0.50 Flare
0.00 0.25 1.25 0.75 (i.e., patients who
begin in remission and remain in remission will
have 0.05 hospitalizations during the period
those who begin with active disease and develop a
flare will have 1 hospitalization during the
period) LUPUS OUTCOME VARIABLES (II)
26
Hospitalizations (cont.) Alternatively,
hospitalizations can be modeled based on expected
time in a state (i.e., not on the probability of
making a transition from one state to another
state) Hypothetical number of hospitalizations Re
mission 0.132 Active 0.244 Flare
0.730 e.g., 0.132 (0.59 0.05) (0.41
0.25) LUPUS OUTCOME VARIABLES (III) Hypothetical
QALYs QALYs also can be modeled based on the
probability of making a transition from one state
to another state. The following matrix assumes
that the transition occurs at the mid-interval
27
Remis. Active Flare Death Remission 0.90
0.80 0.00 0.00 Active 0.80 0.70
0.60 0.35 Flare 0.00 0.60 0.50
0.25 e.g., 0.80 (0.5 0.9) (0.5
0.7) Alternatively, QALYs can be modeled based
on expected time in a state (i.e., not on the
probability of making a transition from one state
to another state) Hypothetical QALY weights
Remission 0.90 Active 0.70 Flare
0.50 SUMMARY RESULTS OF STEPS 1-6 Disease
Problem Systemic lupus Disease
States Remission Active Disease Flare
Death
28
Disease Transition Probabilities Time t
Remis. Active Flare (Time t1) Remission
0.59 0.41 0.00 Active 0.07 0.86
0.06 Flare 0.00 0.27 0.23 Probability of
Death (1 - (Sum of Row Probabilities)) Remission
0.0 Active 0.01 Flare 0.5 Disease
Transition Probabilities After Hypothetical
Intervention Time t Remis. Active Flare
(Time t1) Remission 0.65 0.35
0.00 Active 0.07 0.86 0.06 Flare 0.00
0.27 0.23
29
SUMMARY RESULTS OF STEPS 1-6 (cont.) Cycle
Length 1 Year Initial Distribution Remission
0.10 Active 0.85 Flare 0.05 Outcomes
Life expectancy by stage of disease (Hospitalizat
ion QALYs) STEPS IN DEVELOPING MARKOV
MODEL Step 7. Analyze the Markov model The
principal analysis can be performed in 1 of 3
ways Iterate the model Matrix algebra
solution (Markov chain) Not discussed Monte
Carlo simulation
30
ANALYSIS STRATEGY 1. ITERATION OF THE MODEL
LUPUS EXAMPLE Assuming that the probability that
patient is in the three states at the beginning
of the model is 0.1, 0.85, and 0.05, what is the
probability a patient will be in remission next
year?
(i.e., multiply the initial distribution times
the first column of the transition matrix)


31
Will have active disease?
(i.e., multiply the initial distribution times
the second column of the transition
matrix) ITERATION OF THE MODEL (II) Will be
experiencing a flare
32
(i.e., multiply the initial distribution times
the third column of the transition matrix) Will
die
(i.e., multiply the initial distribution times
the difference between 1 and the sum of the rows
of the transition matrix)
33
EXPECTED HOSPITALIZATIONS Expected
hospitalizations pi pi j h I j where
equals matrix multiplication and equals
simple multiplication What is the expected number
of hospitalizations among patients who next
period are in remission?
What is the expected number of hospitalizations
among patients who next period have active
disease?
34
EXPECTED HOSPITALIZATIONS (II) What is the
expected number of hospitalizations among
patients who next period are experiencing a flare?
What is the expected number of hospitalizations
among patients who die by the end of the period?
35
Total number of hospitalizations in the
period 0.2571 0.0089 0.159825 0.065375
0.023 ALTERNATIVE CALCULATION OF
HOSPITALIZATIONS Multiply time in state times
probability of hospitalization
Calculations assume transitions occur at the end
of the period EXPECTED QALYS, PERIOD 1 Expected
QALYs pi pi j qi j where q equals
matrix multiplication and equals
simple multiplication
36
What is the expected number of QALYs among
patients who next period are in remission?
Calculations assume transitions occur at the
mid-interval And so on... Total QALYS 0.6980
0.1007 0.5517 0.03635 0.009225 ALTERNATIVE
CALCULATION OF EXPECTED QALYS, PERIOD 1 Multiply
time in state times QALY weight
37
Calculations assume transitions occur at the
end of the period And so on... I TERATION OF THE
MODEL (III) Alternatively, we can write the
transition calculations as a series of equations
multiplying the vector representing the
distribution of patients among the states times
the column representing the probability of making
a transition to a state
38
After 1 period, the distribution will be as
follows Remission 0.1185 (0.100.59)
(0.850.07) (0.050.00) Active 0.7855
(0.100.41) (0.850.86) (0.050.27) Flare
0.0625 (0.100.00) (0.850.06)
.050.23) Survive 0.9665 (0.1185 0.7855
0.0625) Death 0.0335 (1 - 0.9665) After
period 2, the distribution will be as
follows Remission 0.1249 (.11850.59)
(.78550.07) (.06250.00) Active 0.74099
(.11850.41) (.78550.86)
(.06250.27) Flare 0.061505 (.11850.00)
(.78550.06) (.06250.23) Survive 0.927395
(.1249 .74099 .061505) Death 0.039105
(.9665 - .927395) this period Total deaths
0.072605 (.0335 .039105) And so on.... RESULTS
(I)
39

• For a patient who initially has a 0.1, 0.85, and
  0.05 probability of being in the three states,
  respectively
• Natural History
• - Life expectancy (Undisc) 24.48
• - Life expectancy (Disc) 14.44
• - QALYs (Undisc) 17.54
• - QALYs (Disc) 10.34
• - Discounted Hospitalization 3.82
• 24.98 years if transition to death occurs at
  end of interval
• Intervention
• - Life expectancy (Undisc) 25.10
• - Life expectancy (Disc) 14.63
• - QALYs (Undisc) 18.09
• - QALYs (Disc) 10.53
• - Discounted Hospitalization 3.80
• 25.60 years if transition to death occurs at end
  of interval
• RESULTS (II)
• For a patient who initially has a 100 chance of
  being in remission,

40
Natural History - Life expectancy (Undisc)
27.44 - Life expectancy (Disc) 16.08 - QALYs
(Undisc) 19.98 - QALYs (Disc) 11.83 -
Discounted Hospitalization 3.94 Intervention -
Life expectancy (Undisc) 28.45 - Life
expectancy (Disc) 16.45 - QALYs (Undisc)
20.88 - QALYs (Disc) 12.21 - Discounted
Hospitalization 3.94 RESULTS (III) For a
patient who initially has a 100 chance of having
active disease, Natural History - Life expectancy
(Undisc) 25.10 - Life expectancy (Disc)
14.75 - QALYs (Undisc) 17.88 - QALYs (Disc)
10.53 - Discounted Hospitalization 3.90
41

• Intervention
• - Life expectancy (Undisc) 25.59
• - Life expectancy (Disc) 14.92
• - QALYs (Undisc) 18.41
• - QALYs (Disc) 10.71
• Discounted Hospitalization 3.88
• RESULTS (IV)
• For a patient who initially has a 100 chance of
  having a Flare,
• Natural History
• - Life expectancy (Undisc) 9.74
• - Life expectancy (Disc) 5.94
• - QALYs (Undisc) 6.79
• - QALYs (Disc) 4.07
• - Discounted Hospitalization 2.25
• Intervention
• - Life expectancy (Undisc) 9.95
• - Life expectancy (Disc) 6.00
• - QALYs (Undisc) 6.98

42
COST EFFECTIVENESS ANALYSIS (I) Suppose we
estimated the cost-effectiveness of the
intervention as follows
where DLEi Discounted life expectancy
associated with the intervention CI equals the
cost of the intervention CH equals the average
incremental cost of hospitalization (this
formulation assumes that the cost per
hospitalization is the same between the groups)
Hospi and Hospn equal the average number of
hospitalizations in the intervention and natural
history groups, respectively and DQALYi
and DQALYn equal the discounted QALYS in the
intervention and natural history groups,
respectively COST EFFECTIVENESS ANALYSIS
(II) Suppose that the average incremental cost
of hospitalization is 10,000 the
cost-effectiveness ratios for the hypothetical
intervention for varying costs per day of
intervention are as follows
43
ANALYSIS STRATEGY 3. FIRST ORDER MONTE CARLO
SIMULATION Models progression of multiple
individuals through model using results of random
number generation and probabilities Ranges for
transitions based on lupus transition matrix -
Initial Period, Natural History and
Intervention Remis Active Flair
Dead 001-100 101-950 951-1000 NA

44
- Other Periods, Natural History Remis 001-590
591-1000 NA NA Active 001- 70 71- 930
931-990 991-1000 Flair NA 001- 270
271-500 501-1000 - Other Periods,
Hypothetical Intervention Remis 001-650
651-1000 NA NA Active 001- 70 71- 930
931-990 991-1000 Flair NA 001- 270
271-500 501-1000 FIRST ORDER MONTE CARLO
SIMULATION (II)
45
Sample Projection, Individual 1, Natural
History Random Markov Cycle Number
State 0 59 Remission 1 999
Active 2 504 Active 3 948 Flare 4
447 Flare 5 615 Dead Sample
Projection, Individual 2, Natural History Random
Markov Cycle Number State 0 952
Flare 1 725 Dead Allows incorporation
of patient history into transition probabilities,
given one knows the states the patient has passed
through
46
1ST ORDER MONTE CARLO SIMULATION RESULTS All
patients, natural history - Life expectancy
25.08 ( 23.87) - Discounted life expect
14.35 ( 9.27) - Discounted QALYs 10.57 (
6.98) - Discounted Hosp 3.85 (
1.68) Initially in remission, natural history -
Life expectancy 27.67 ( 24.53) - Discounted
life expect 15.91 ( 8.30) - Discounted QALYs
11.70 ( 6.22) - Discounted Hosp 3.95 (
1.55) Initially with active disease, natural
history - Life expectancy 25.62 ( 24.99) -
Discounted life expect 14.49 ( 8.92) -
Discounted QALYs 10.62 ( 6.72) - Discounted
Hosp 3.89 ( 1.67)
47

• Initially experiencing flare, natural history
• - Life expectancy 9.65 ( 19.03)
• - Discounted life expect 6.10 ( 8.85)
• - Discounted QALYs 4.22 ( 6.39)
• - Discounted Hosp 2.19 ( 1.91)
• 2000 iterations for each result
• SECOND ORDER MONTE CARLO SIMULATION ADDRESSING
  UNCERTAINTY
• The variances estimated in the matrix algebra and
  Monte Carlo solutions to the Markov model capture
  the binomial variation (i.e., the variance due to
  the coin flip) it does not capture the
  stochastic error
• related to the fact that had your data been
  derived from a different sample, the point
  estimates would have differed
• If in the Monte Carlo simulation one substitutes
  distributions for the point estimates, and
  samples from these distributions, one can
  incorporate this uncertainty

48
First-order (individual uncertainty) Mean and
standard deviation sampling can be done per
patient or per cycle Method is equivalent to
changing the number of patients included in a
trial Second-order (parameter uncertainty)
(option in DATA Pro) Mean and standard error
sample once per trial solve the model using
methods 1, 2, or 3 above each iteration
represents a trial (i.e., results equivalent to a
bootstrap procedure) If all data not derived from
a single trial, may lose some of the correlation
structure in the sampling STOCHASTIC VARIATION
AND MODELS DERIVED FROM A SINGLE DATA SET If the
decision analytic model is based on a trial or on
primary data from an observational study, one can
assess the effects of stochastic variation (i.e.,
variation due to the fact that your model is
based on a sample drawn from a larger population)
using one of the resampling techniques (jackknife
or bootstrap)
49
THE JACKKNIFE (I) Develop a set of repeated
samples by deleting every subject one time Delete
one at a time perform N analyses for N
subjects Delete 1/10th to 1/20th of subjects
perform 10 to 20 analyses 10 to 20 samples
efficient Delete subjects in an informed way
(e.g., by individual center or by important
clinical characteristics) perform 10 to 20
analyses THE JACKKNIFE (II) Analysis is
performed in each sample Results combined to
estimate a standard error (using a pseudo
value) CI derived by multiplying the standard
error times the appropriate Students T
statistic For efficiency of data storage and
calculation, can estimate 1 model for the full
sample plus 10 subsamples each with 1/10th of
observations deleted
50
Construct 11 models Calculate standard errors
with 9 degrees of freedom (t 2.262) JACKKNIFE
AND SENSITIVITY ANALYSIS Nonstatistical
variation (e.g., unit costs or discount
rate) Combine jackknife and sensitivity
analysis Within each jackknife sample perform
sensitivity analysis Derive confidence intervals
for sensitivity analyses INTERPRETATION OF TIME
IN A STATE Distribution of population by period
51
In non-health care simulations, transitions
traditionally have been assumed to occur at the
end of the interval (This is the assumption made
by the matrix algebra solution to the
model) Routinely in health care, transitions are
assumed to occur at the midinterval Computation
of transitions is unaffected by this assumption,
but calculation of the payoffs for the
transitions is STEPS IN DEVELOPING MARKOV MODEL
52

• Step 8. Perform sensitivity analysis
• Assess the sensitivity of the model to
• Changes in the transition probabilities
• Changes in the hospitalization rate
• Changes in the QALY weights
• STABILITY OF MARKOV ESTIMATES (I)
• How stable are the life expectancy and other
  estimates from the lupus model?
• STABILITY OF MARKOV ESTIMATES (II)
• Recall the "Active" row of the SLE transition
  matrix
• Remis. Active Flare
• Active 0.07 0.86 0.06
• Also recall the estimated life expectancy in the
  states

53
Suppose the probabilities in the transition
matrix related to active disease changed as
follows Remis. Active Flare Active 0.07
0.85 0.06 Would you anticipate that the life
expectancy estimates would change by 0.5, 1.5, or
more than 2.5 years? STABILITY OF MARKOV
ESTIMATES (III) Revised "Active" Row, SLE
Matrix Remis. Active Flare Active 0.07
0.85 0.06 Life expectancy in the
states Initial Revised Difference Remission
27.44 23.12 4.32 Active 25.00 20.68
4.32 Flare 9.74 8.22 1.52
54
STABILITY OF MARKOV ESTIMATES (IV) Hypothetical
Disease Transition Probabilities (I) Remis.
Act. Flare (Time t1) Remission 0.59 0.41
0.00 Active 0.25 0.45 0.24 Flare 0.00
0.27 0.23 (Time t) Hypothetical Disease
Transition Probabilities (II) Remis. Act.
Flare (Time t1) Remission 0.59 0.41
0.00 Active 0.25 0.44 0.24 Flare 0.00
0.27 0.23 (Time t) STABILITY OF MARKOV
ESTIMATES (V) Mean Duration in States,
Hypothetical Disease Hyp I Hyp II
Difference Remission 11.34 10.95
0.39 Active 8.90 8.51 0.39 Flare 4.42
4.28 0.14
55

• STABILITY OF MARKOV ESTIMATES (VI)
• Can situations where small changes in
  probabilities lead to large changes in predicted
  survival be diagnosed?
• The problem is similar to the one posed to
  regression estimates by correlations among the
  independent variables
• In both cases, a matrix inversion is being
  performed
• In both cases, the results of the inversion can
  vary greatly due to the structure of the matrix
  being inverted
• The condition number, the ratio of the largest
  and smallest eigenvalues of the matrix, is an
  indicator of how sensitive the results of the
  inversion will be to small changes in the
  probabilities. Conditions numbers of 30 and above
  are indicative of matrices that will have
  problems. Condition numbers can be computed with
  programs such as MatLab
• ASSESSING VALIDITY
• What is being assessed?
• Predictions of
• Natural history
• Impact of intervention

56

• How is it assessed?
• Discrimination
• Calibration
• PREDICTION OF DEATH AT 5 AND 10 YEARS
• Discrimination 5 years, ROC area, 0.82
• 10 years, ROC area, 0.74
• Calibration

PREDICTION OF INSTITUTIONALIZATION AT 5
YEARS Discrimination ROC area, 0.91 Calibration
57

• HISTORY (I)
• All patients in a state in a given time period
  have the same transition probabilities to other
  states, no matter what course they followed to
  get there
• i.e., the conditional probability of any future
  event is independent of past events and depends
  only on the present state
• Suppose two CHF patients were in NYHA class II in
  a model that predicted progression and regression
  of disease as a series of transitions between
  NYHA classes
• a. Individual 1 has had CHF for five years and
  has spent two of
• them in NYHA class 4
• b. Individual 2 has newly diagnosed CHF

58
Should these two patients be expected to have the
same disease progression? HISTORY (II) Potential
Solutions Ignore the problem It is an empirical
question whether the overall mean durations and
estimates of resource utilization will be
affected by these relatively fine
distinctions Define more states For example,
define two states representing NYHA class 2, one
for individuals who never been in a higher class
and one for those who have Adopt a modified
version of a Markov Model, such as a Monte Carlo
simulation, which allows for the incorporation of
transition probability matrices that account for
an individuals history in prior states MODELING
POPULATIONS Use population estimates ('s) in the
vector representing the distribution across the
states Allow for new recruits (e.g., births)
between each period by defining a recruitment
vector that is added to the vector representing
the distribution across states Allow for people
to leave the model without dying (e.g., emigrate
from the country)