Markov Chains

We begin this section with a motivational example. Consider the connected graph on six vertices shown in . The first move is to choose a vertex at random and move there. Afterwards, we follow the following recursive procedures. If after \(i\) moves, you are at a vertex \(x\) and \(x\) has \(d\) neighbors, choose one of the neighbors at random, with each having probability \(1/d\) and move there. We then attempt to answer questions of the following flavor:

1. For each vertex \(x\), let \(p_{x,m}\) denote the probability that you are at vertex \(x\) after \(m\) moves. Does \(\lim_{m\rightarrow\infty}p_{x,m}\) exist and if so, how fast does the sequence converge to this limit?

2. How many moves must I make in order that the probability that I have walked on every edge in the graph is at least \(0.999\)?

This example illustrates the characteristics of an important class of computational and combinatorial problems, which are collectively referred to as Markov Chains:

1. There is a finite set of states \(S_1\), \(S_2,\dots,S_n\), and at time \(i\), you are in one of these states.

2. If you are in state \(S_j\) at time \(i\), then for each \(k=1,2,\dots,n\), there is a fixed probability \(p(j,k)\) (which does not depend on \(i\)) that you will be in state \(S_k\) at time \(i+1\).

The \(n\times n\) matrix \(P\) whose \(j,k\) entry is the probability \(p(j,k)\) of moving from state \(S_j\) to state \(S_k\) is called the transition matrix of the Markov chain. Note that \(P\) is a stochastic matrix, , all entries are non-negative and all row sums are\(1\). Conversely, each square stochastic matrix can be considered as the transition matrix of a Markov chain.

For example, here is the transition matrix for the graph in . \[\begin{aligned}\end{aligned}\]

A transition matrix \(P\) is regular if there is some integer \(m\) for which the matrix \(P^m\) has only positive entries. Here is a fundamental result from this subject, one that is easy to understand but a bit too complex to prove given our space constraints.

Given the statement of , the computation of the row vector \(W\) can be carried out by eigenvalue techniques that are part of a standard undergraduate linear algebra course. For example, the transition matrix \(P\) displayed in is regular since all entries of \(P^3\) are positive. Furthermore, for this matrix, the row vector \(W=(5/13, 3/13, 2/13, 2/13, 1/13, 1/13)\). However, the question involving how fast the convergence of \(P^m\) is to this limiting vector is more subtle, as is the question as to how long it takes for us to be relatively certain we have made every possible transition.

A state \(S_i\) in a Markov chain with transition matrix \(P\) is absorbing if \(p_{i,i}=1\) and \(p_{i,j}=0\) for all \(j\neq i\), , like the infamous Hotel California, once you are in state \(S_i\), you can never leave.

Now we might consider the following game. Start at one of the four vertices in \(\{1,2,3,4\}\) and proceed as before, making moves by choosing a neighbor at random. Vertex \(4\) might be considered as an escape point, a safe harbor that once reached is never left. On the other hand, vertex \(5\) might be somewhere one meets a hungry tiger and be absorbed in a way not to be detailed here.

We say the Markov chain is absorbing if there is at least one absorbing state and for each state \(S_j\) that is not absorbing, it is possible to reach an absorbing statealthough it may take many steps to do so. Now the kinds of questions we would like to answer are:

1. If we start in non-absorbing state \(S_i\), what is the probability of reaching absorbing state \(S_j\) (and then being absorbed in that state, a question which takes on genuine unpleasantness relative to tigers)?

2. If we are absorbed in state \(S_j\), what is the probability that we started in non-absorbing state \(S_i\)?

3. If we start in non-absorbing state \(S_i\), what is the expected length of time before we will be absorbed?