```{exercise} Consider the Markov chain $X$ on five states $\{1,2,3,4,5\}$ started at $1$ with transition matrix given by ``` ```{r} P <- matrix(c(1/4, 1/4, 1/2, 0,0, 1/4, 1/8,1/8,0,1/2, 1/4,0,0, 1/2, 1/4, 0,0,0,1/2, 1/2, 1/3, 1/3, 0, 0, 1/3), nrow =5) P <-t(P) P ``` For each $s \in \{1,2,3,4,5\}$, let $T^s = \inf \{ n \geq 1: X(n)=s | X(0)=s\}$. * By running simulations, estimate $\mathbb{E} T^s$, for each $s$. * Using our theory of Markov chains, compute *exactly* $\mathbb{E} T^s$, for each $s$. ```{exercise} By running simulations, verify the central limit theorem for renewal processes, in the case where the inter-arrival times are given by a gamma distribution with shape $n=2$ and rate $\lambda=3$. ```

```{exercise} Check (by pen and paper), the law of large numbers and the central limit theorem for renewal processes, for the special case where the renewal process is a Poisson process. ```

```{exercise} Prove that if $s$ is a recurrent state of a Markov chain that is started at $s$, then with probability one, it must return to that state for infinitely many $n \in \mathbb{Z}^{+}$ ```

```{exercise} Prove that if an irreducible Markov chain has a recurrent state, then all the states must also be recurrent. ```

* Version: `r format(Sys.time(), '%d %B %Y')` * [Rmd Source](https://tsoo-math.github.io/ucl/QHW6.Rmd)