Chapter 10 Markov Chains

1
Chapter 10Markov Chains

• Section 10.1
• Basic Properties
• of Markov Chains

2
Some Background Information

• Mathematical models that evolve over time in a
  probabilistic manner are called stochastic
  processes.
• A special kind of stochastic process is a Markov
  Chain, where the outcome of an experiment depends
  only on the outcome of the previous experiment.

3
Why Study Markov Chains?

• Markov chains are used to analyze trends and
  predict the future. (Weather, stock market,
  genetics, product success, etc.

4
A Sociology Example
Sociologists classify people by income as
lower-class, middle-class, and upper-class.
They have found that the strongest determinant
of the income class of an individual is the
income class of that persons parents.
5
Transition Diagrams
6
Transition Matrices
7
Characteristics of a Transition Matrix

• It is a square matrix.
• All entries are between 0 and 1 (because all the
  entries are probabilities).
• The sum of the entries in any row must be 1.
• Denoted by P .

8
Key Features of Markov Chains

• A sequence of trials of an experiment is a Markov
  chain if
• 1.) the outcome of each experiment
• is one of a set of discrete states
• 2.) the outcome of an experiment
• depends only on the present state,
• and not on any past states
• 3.) the transition probabilities remain
• constant from one transition to the
• next.

9
Transition Probabilities from One State to
Another

• The transition matrix shows the probabilities of
  moving from state-to-state from the current
  generation to the next.
• P gives the probabilities of a transition from
  one state to another in k repetitions of an
  experiment, provided the transition probabilities
  remain constant from one repetition to the next.

k
10
Example 1

• Use the transition matrix from the Sociology
  example to find the probabilities of change for
  the grandchildren of the current generation.

11
Example 2

• Write a transition matrix for the following
  situation. Then find the probabilities
  associated with a third and fourth purchase from
  the bookstores.
• In a study of the market share of the three
  bookstores in a university town, it was found
  that 75 of those who had bought from University
  Bookstore would buy from it again, 15 from
  Campus Bookstore and 10 from Bookmart.
• Of those who bought from Campus Bookstore, 90
  would buy from it again and 5 each would buy
  from University Bookstore and Bookmart.
• 85 of those who bought from Bookmart would buy
  from it again, 5 from University Bookstore and
  10 from Campus Bookstore.

12
Probability Vector (Matrix)

• When a study is first begun, the probabilities of
  the states are called the initial distribution.
• This distribution is written as a matrix of only
  one row.
• Denoted by X0 .

13
Characteristics of aProbability Vector

• It is a row matrix.
• Each entry must be between 0 and 1 inclusive.
• The sum of the entries of the row must be 1.

14
Making Predictions about the Population
Proportion with Markov Chains

• 1.) Create a probability vector, X0 .
• The entries are the initial probabilities of
  the states.
• 2.) Create the transition matrix, P .
• The entries are the probabilities of passing
  from current states (rows) to the first
  following states (columns).
• 3.) Calculate X0 P
• This is the probability vector after n
  repetitions of
• the experiment. In other words, it is a
  prediction
• for the proportion of the population after n
• repetitions.
• Note The columns and rows of the probability
  vector and the transition matrix must be labeled
  the same.

n
15
Example 3

• A marketing analysis shows that KickKola
  currently commands 14 of the cola market.
• Further analysis indicates that 12 of the
  consumers who do not currently drink KickKola
  will purchase KickKola the next time they buy a
  cola (in response to a new advertising campaign)
  and that 63 of the consumers who currently drink
  KickKola will purchase it the next time they buy
  a cola.
• Predict KickKolas market share at
• a.) the next following purchase
• b.) the second following purchase

16
Example 4

• It has been noted that George the golfer tends to
  repeat himself. In his first shot, he will hit
  his ball in the fairway (F), in the rough (R), or
  out of bounds (B).
• On his first shot he hits in the fairway 60 of
  the time, in the rough 30 of the time, and out
  of bounds 10 of the time.
• However, on subsequent shots, if he hit in the
  fairway the first time, he will hit in the
  fairway next time 90 of the time and in the
  rough 10 of the time.
• If he hit in the rough the first time, he will
  hit the rough next time 50 of the time and out
  of bounds 5 of the time.
• If he hits out of bounds the first time, he will
  hit out of bounds next time 70 of the time and
  in the fairway 15 of the time.
• Find the probability that his next following
  shot is in
• a.) the fairway
• b.) the rough
• c.) out of bounds