Restricted versions of the Markov Property lead to different types of Markov Processes. These may be classified based on whether the state space is a continuous variable or discrete and whether the process is observed over continuous time or only at discrete time instants.
These may be summarized as -
(a) Markov Chains over a Discrete State Space
(b) Discrete Time and Continuous Time Markov Processes and Markov Chains
| Markov Chain | State Space is discrete (e.g. set of non- negative integers) |
| Discrete Time | State changes are pre-ordained to occur only at the integer points 0, 1, 2, ......, n (that is at the time points t0 , t1 , t2 ,......, tn ) |
| Continuous Time | State changes may occur anywhere in time |
In the analysis of simple queues, the state of the queue may be represented by a single random variable X(t) which takes on integer values {i, i = 0, 1, ...... } at any instant of time. The corresponding process may therefore be treated as a Continuous Time Markov Chain since time is continuous but the state space is discrete.
Homogenous Markov Chain
A Homogenous Markov Chain is one where the transition probability P{Xn+1=j | Xn = i } is the same regardless of n .
Therefore, the transition probability pij of going from state i to state j may be written as
Note that this property implies that if the system is in state i at the nth time instant then the probability of finding the system in state j at the (n+1)th time instant (i.e. the next time instant) does not depend on when we observe the system, i.e. the particular choice of n. This simplifies the description of the system considerably as the transition probability pij will be the same, regardless of when we choose to observe the system.
| It should be noted that for a Homogenous Markov chain, the transition probabilities depend only on the terminal states (i.e. the initial state i and the final state j ) but do not depend on when the transition ( i → j ) actually occurs. |