In class exercise: Transition matrices

1 Gambler’s ruin

Recall the Gambler’s ruin problem from the last session. Suppose someone plays the following gambling game. They bet \(1\) pound on a fair flip of a coin and win \(1\) pound if it comes up heads, and lose if come up tails. They start with \(1\) pound, and stop playing once they have reached \(5\) pounds or have no money left. Express this problem up as a Markov chain. In particular:

• Specify the state space and transition matrix.
• Take high powers of the transition matrix in R; comment on what you see.

2 Anything is possible

Consider the transition matrix \(P\) on the states \(\{1,2,3,4\}\) given by:

P <- matrix(c(0, 1/2, 1/2, 0, 0, 1/2, 0,0,0,1/2, 1/2,0,0, 1/2, 0, 0,0,0,1/2, 1/2, 1/3, 1/3, 0, 0, 1/3), nrow =5)
P <-t(P)
P

##           [,1]      [,2] [,3] [,4]      [,5]
## [1,] 0.0000000 0.5000000  0.5  0.0 0.0000000
## [2,] 0.5000000 0.0000000  0.0  0.0 0.5000000
## [3,] 0.5000000 0.0000000  0.0  0.5 0.0000000
## [4,] 0.0000000 0.0000000  0.0  0.5 0.5000000
## [5,] 0.3333333 0.3333333  0.0  0.0 0.3333333

• Compute powers of \(P\) until you see that all the entries are positive.

• What does it mean if all the entries are positive?

3 Convex combinations

• Let \(\lambda\) and \(\mu\) be probability measures on \(\Omega\) with events \(\mathcal{F}\).

Recall that \(\lambda\) is a probability measure if

• \(\lambda(\Omega) =1\)
• \(\lambda(A ) \in [0,1]\) for all events \(A \in \mathcal{F}\)
• \(\lambda( \bigcup_{i=1} ^{\infty} A_i) = \sum_{i=1}^{\infty} \lambda(A_i)\) for any countable number of pairwise disjoint events \(A_1, A_2, \ldots\).

Show that \[\rho = (1-t)\lambda + t \mu\] is a probability measure on \(\Omega\) for all \(t \in [0,1]\).

• Let \(P\) be a transition matrix on a state space \(S\). Show that if \(\lambda\) and \(\mu\) are stationary measures of \(P\), then any convex combination of \(\lambda\) and \(\mu\) is a stationary measure for \(P\).

• Consider the transition matrix on four states given by \[ \left( \begin{array}{cccc} 1/2 & 1/2 & 0 & 0 \\ 1/2 & 1/2 & 0 & 0 \\ 0 & 0 & 1/3 & 2/3 \\ 0 & 0 & 1/2 & 1/2 \\ \end{array} \right) \]

• Find three stationary measures for \(P\).
  

• Does \(P\) have finitely many stationary measures?

4 Periodicity

Consider the transition matrix \(P\) on the states \(\{1,2,3\}\) given by:

P <- matrix( c( 0,0,1,0, 0,0,0,1, 1/2,1/2,0,0, 1,0,0,0), nrow =4)
P <- t(P)
P

##      [,1] [,2] [,3] [,4]
## [1,]  0.0  0.0    1    0
## [2,]  0.0  0.0    0    1
## [3,]  0.5  0.5    0    0
## [4,]  1.0  0.0    0    0

• Take powers of \(P\), what do you observe?
• Does \(P\) have any stationary distributions? Explain.
• Consider the transition matrix \(Q = P^2\). Does \(Q\) have any stationary distributions. Explain.

5 Version: 14 October 2020

• Rmd Source