Saturday Agenda

1
Markov Chains

2
Markov ChainsGeneral Description

• We want to describe the behavior of a system as
  it moves (makes transitions) probabilistically
  from state to state.
• States may be qualitative or quantitative
• Basic Assumption
• The future depends only on the present (current
  state) and not on the past. That is, the future
  depends on the state we are in, not on how we
  arrived at this state.

3
Example 1 - Brand loyalty or Market Share

• For ease, assume that all cola buyers purchase
  either Coke or Pepsi in any given week. That is,
  there is a duopoly.
• Assume that if a customer purchases Coke in one
  week there is a 90 chance that the customer will
  purchase Coke the next week (and a 10 chance
  that the customer will purchase Pepsi).
  Similarly, 80 of Pepsi drinkers will repeat the
  purchase from week to week.

4
Example 1 - Developing the Markov Matrix

• States
• State 1 - Coke was purchased
• State 2 - Pepsi was purchased
• (note states are qualitative)
• Markov (transition or probability) Matrix
• From\To Coke Pepsi
• Coke 0.9 0.1
• Pepsi 0.2 0.8

5
Example 1 Understanding Movement

• From\To Coke Pepsi
• Coke 0.9 0.1
• Pepsi 0.2 0.8
• Quiz If we start with 100 Coke purchasers and
  100 Pepsi purchasers, how many Coke purchasers
  will there be after 1 week?

6
Graphical Description 1The States
7
Graphical Description 2Transitions from Coke
.9
.1
8
Graphical Description 3All transitions
.9
.8
.1
.2
9
Example 1 - Starting Conditions

• Percentages
• Identify probability of (percentage of shoppers)
  starting in either state
• (We will assume a 50/50 starting market share in
  our example that follows.)
• Assume we start in one specific state (by setting
  one probability to 1 and the remaining
  probabilities to 0)
• Counts (numbers)
• Identify number of shoppers starting in either
  state

10
Example 1

• From\To Coke Pepsi
• Coke 0.9 0.1
• Pepsi 0.2 0.8
• Starting Probabilities 50 (or 50 people) each
• Questions
• What will happen in the short run (next 3
  periods)?
• What will happen in the long run?
• Do starting probabilities influence long run?

11
Graphical Solution After 1 Transition
.9(50)45
.8(50)40
.1(50)5
(50)Coke(55)
(50)Pepsi(45)
.2(50)10
12
Graphical Solution After 2 Transitions
.9(55)49.5
.8(45)36
.1(55)5.5
(55)Coke(58.5)
(45)Pepsi(41.5)
.2(45)9
13
Graphical Solution After 3 Transitions
.9(58.5)52.65
.8(41.5)33.2
.1(58.5)5.85
(58.5)Coke(60.95)
(41.5)Pepsi(39.05)
.2(41.5)8.3
14
Analyzing Markov Chains

• Open QM for Windows
• Module Markov Chains
• Number of states 2
• Number of transitions - 3

15
Example 1 After 3 transitionsn-step Transition
probabilities

• End of Period 1 Coke Pepsi
• Coke 0.8999 0.1000
• Pepsi 0.2000 0.8000
• End prob (given initial) 0.5500 0.4500
• End of Period 2 Coke Pepsi
• Coke 0.8299 0.1700
• Pepsi 0.3400 0.6600
• End prob (given initial) 0.5849 0.4150
• End of Period 3 Coke Pepsi
• Coke 0.7809 0.2190
• Pepsi 0.4380 0.5620
• End prob (given initial) 0.6094 0.3905

1 step transition matrix
2 step transition matrix
3 step transition matrix
16
Example 1 - Results (3 transitions, start .5,
.5)

• From\To Coke Pepsi
• Coke 0.78100 0.21900
• Pepsi 0.43800 0.56200
• Ending probability 0.6095 0.3905
• Steady State probability 0.6666 0.3333
• Note We end up alternating between Coke and Pepsi

3 step transition matrix
Depends on initial conditions
Independent of initial conditions
17
Example 2 - Student Progression Through a
University

• States
• Freshman
• Sophomore
• Junior
• Senior
• Dropout
• Graduate
• (note again, states are qualitative)

18
Example 2 - Student Progression Through a
University - States
Freshman
Sophomore
Junior
Senior
Drop out
Graduate
Note that eventually you must end up in Grad or
Drop-out.
19
Example 2 ResultsLazarus paper data

• First yr Soph Junior Senior Grad Drop out
• First year 0.0000 0.0000 0.0000 0.0000 0.8565 0.14
  34
• Sophomore 0.0000 0.0000 0.0000 0.0000 0.8860 0.113
  9
• Junior 0.0000 0.0000 0.0000 0.0000 0.9273 0.0726
• Senior 0.0000 0.0000 0.0000 0.0000 0.9690 0.0310
• Graduate 0.0000 0.0000 0.0000 0.0000 1.0000 0.0000
  
• Drop out 0.0000 0.0000 0.0000 0.0000 0.0000 1.0000
  
• End prob 0 0 0 0 0.8565 0.1434
• Steady State 0 0 0 0 1 1

20
From the paper

• If there are an equal number of freshmen,
  sophomores, juniors and seniors at the beginning
  of an academic year then
• The percentage of this mixed group of students
  who will graduate is
• (.857.886.927.969)/4 91

21
Classification of states

• Absorbing
• Those states such that once you are in you never
  leave.
• Graduate, Drop Out
• Recurrent
• Those states to which you will always both leave
  and return at some time.
• Coke, Pepsi
• Transient
• States that you will eventually never return to
• Freshman, Sophomore, Junior, Senior

22
State Classification Quiz
State 1
State 2
State 3
State 4
State 5
23
State Classification Article

• A non-recursive algorithm for classifying the
  states of a finite Markov chain
• European Journal of Operational Research
• Vol 28, 1987

24
(No Transcript)
25
Example 3 - Diseases

• States
• no disease
• pre-clinical (no symptoms)
• clinical
• death
• (note again states are qualitative)
• Purpose
• Transition probabilities can be different for
  different testing or treatment protocols

26
Example 4 - Customer Bill paying

• States
• State 0 Bill is paid in full
• State i Bill is in arrears for i months,
• i 1,2,,11
• State 12 Deadbeat

27
Example 5 - Oil Market

• State
• State 0 - oil market is normal
• State 1 - oil market is mildly disrupted
• State 2 - oil market is severely disrupted
• State 3 - oil production is essentially shut down
• Note States are qualitative
• Phila Inq, 3/24/04, Strategic oil reserve
  fill-up will continue

28
Example 6 HIV infections

• Based on Can Difficult-to-Reuse Syringes Reduce
  the Spread of HIV among Injection Drug Users
• Caulkins, et. al.
• Interfaces, Vol 28, No. 3, May-June 1998, pp
  23-33
• State
• State 0 Syringe is uninfected
• State 1 Syringe is infected
• Notes
• P(0, 1) .14
• 14 of drug users are infected with HIV
• P(1, 0) .33.05
• 5 of the time the virus dies 33 of the time it
  is killed by bleaching

29
Example 7 Mental HealthLazarus

• depressed
• manic
• euthymic/remitted
• mortality

30
Example 8 - Baseball

• States
• State 0 - no outs, bases empty
• State 1 - no outs, runner on first
• State 2 - no outs, runner on second
• State 3 - no outs, runner on third
• State 4 - no outs, runners on first, second
• State 5 - no outs, runners on first, third
• State 6 - no outs, runners on second, third
• State 7 - no outs, runners on first, second,
  third
• . Repeat for 1 out and 2 outs for a total of 24
  states
• Moneyball by Michael Lewis, p 134

31
Example 9 Football OvertimePlayoffs (no time
limit)

• States
• Team A has ball
• Team B has ball
• Team A scores (absorbing)
• Team B scores (absorbing)
• Win, Lose, or Draw A Markov Chain Analysis of
  Overtime in the National Football League,
  Michael A. Jones, The College Mathematics
  Journal, Vol. 35, No. 5, November 2004, pp
  330-336

32
Additional References from Interfaces

• Managing Credit Lines and Prices for Bank One
  Credit Cards. By Trench, Margaret S. Pederson,
  Shane P. Lau, Edward T. Lizhi Ma Hui Wang
  Nair, Suresh K.. Interfaces, Sep/Oct2003, Vol. 33
  Issue 5, p4, 18p
• Real Applications of Markov Decision Processes.
  By White, Douglas J.. Interfaces, Nov/Dec85,
  Vol. 15 Issue 6, p73, 11p
• Further Real Applications of Markov Decision. By
  White, D.J.. Interfaces, Sep/Oct88, Vol. 18 Issue
  5, p55, 7p
• A Markovian Model for the Valuation of Human
  Assets Acquired by an Organizational Purchase.
  By Flamholtz, Eric G. Geis, George T. Perle,
  Richard J.. Interfaces, Nov/Dec84, Vol. 14 Issue
  6, p11, 5p
• STUDENT FLOW IN A UNIVERSITY DEPARTMENT RESULTS
  OF A MARKOV ANALYSIS. By Bessent, E. Wailand
  Bessent, Authella M.. Interfaces, 1980, Vol. 10
  Issue 2, p52, 8p

33
Markov Chains

• The end