Contents

Stochastic Process

Definition

When we study the behavior of a random system, we are interested in how the system evolves in time. The evolution of the system is a random process or stochastic process.

With the following assumption:

  • The system occupy one and only one state at any moment in time;
  • The evolution of the system can be represented by the transitions from state to state.

In this way, the process can be represented by all possible states that the system may occupy and which state the system occupy in different time stamp throughout the whole lifetime of the system. Mathematically,

  • Stochastic Process can be defined as a family of random variables \(\{X(t), t\in T\}\), each \(r.v. X(t)\) is defined on some probability space.
  • \(T\) is called parameter space on which the parameter \(t\) is defined. When the parameter \(t\) is discrete, \(T\) is also called index set. Parameter \(t\) usually represent time.
  • The values of \(r.v. X(t)\) are called states. The set of all possible states forms the state space of the stochastic process.

Classification

Parameter space and State space

graph LR; A(Stochastic Process)-->B(state space); A-->C(parameter space); B-->D(discrete states, i.e. chain); B-->E(continuous states); C-->F(discrete parameters); C-->G(continuous parameters);
  • For instance, we could deal with the stochastic process of discrete state space where states are usually identified with a subset of the set of natural numbers \(\{0,1,2,\ldots\}\). This stochastic process is also referred to as a Chain.
  • If the stochastic process is of discrete parameter space, we could call it discrete parameter stochastic process or discrete-time stochastic process.
  • Markov Process is a special type of stochastic process. If the state space of a Markov Process is discrete, then this process is a Markov Chain. Also, there are discrete-time Markov Chain and continuous-time Markov Chain with regards to different type of parameter space.

Time Dependency of Evolution

graph LR; A(Stochastic Process)-->B(initial time); A-->C(elapsed time); B--depending-->D(non-stationary); B--not depending-->E(stationary); C--depending-->F(non-homogeneous); C--not-depending-->G(homogeneous);
  • A stationary stochastic process is invariant under an arbitrary shift of time origin, i.e. mathematically \(\forall \alpha, \forall n, t_i, x_i\)(with \(i = 1,2,\ldots ,n\)),
\[\mathbf{Pr}(X(t_1)<x_1,X(t_2)<x_2,\ldots ,X(t_n)<x_n) \\ = \mathbf{Pr}(X(t_1+\alpha )<x_1,X(t_2+\alpha )<x_2,\ldots ,X(t_n+\alpha )<x_n)\]
  • The transitions of a homogeneous stochastic process is not depending on the elapsed time from initiation.
  • In either homogeneous or non-homogeneous case, the stochastic process may or may not be stationary.

Dependency of Passed Evolution

graph LR; A(Stochastic Process)-->B(
Relation between
Future Evolutions and
Passed Transitions
); B--not depending-->C(
Depending on
current state ONLY
); B--depending-->D(
Depending on
current state AND passed transitions
); C-->E(Markov Process);

Markov Process

Markov Property

  • Markov Process is a type of stochastic process with Markov Property.
  • Markov Property, i.e. Memoryless property, states that the future evolution of the process depends ONLY on its current state, not depending on the passed transitions of the process.

Classification

Applying the classification rules of stochastic process, we also have the following types of Markov Process.

graph LR; A(Markov Process)-->B(
state space
parameter space
); B-->C(
discrete states
i.e. Markov Chain
); B-->D(continuous states); C-->E(
Discrete Time Markov Chain
i.e. DTMC
); C-->F(
Continuous Time Markov Chain
i.e. CTMC
); D-->G(Discrete Time Markov Process); D-->H(Continuous Time Markov Process); A-->I(initial time); A-->J(elapsed time); I-->K(stationary or non-stationary); J-->L(homogeneous or non-homogeneous);

Markov Chain

Markov Chain is a stochastic process with discrete state space and Markov property.

Further more in DTMC: Discrete Time Markov Chains.

Reference

There are some books and articles I read while I was writing this post.1

  1. William J. Stewart, Probability, Markov Chains, Queues, and Simulation(The Mathematical Basis of Performance Modeling), 2009, Princeton University Press, Link. ↩