Trang Le
#math graduate. Postdoc fellow with Jason Moore.
Random walk, Markov chain and Brownian motion
Trang Le, PhD
2020-04-15
Guest lecture, Fundamentals of AI
Concepts
- Random walk
- Stochastic processes
- Markov chain
- Martingale
A stochastic process
is a collection of random variables indexed by time.
\(X_0, X_1, X_2, X_3, \cdots\) discrete time
\(\{X_t\}_{t\geq 0}\) continuous time
Alternatively, ...
a stochastic process is a probability distribution over a space of paths; this path often describes the evolution of some random value/system over time.
- Dependencies in a sequence of values?
- Long term behavior?
- Boundary events (if any)?
Simple random walk
\(Y_i\) i.i.d random variables \( \begin{cases} 1 & (\textrm{prob} \frac{1}{2}) \\ -1 & (\textrm{prob} \frac{1}{2}) \end{cases} \)
X_t = \sum_{i = 1}^tY_i
\forall t > 0
X_0 = 0
\(X_0, X_1, X_2, ...\) is called simple random walk.
hard to visualize the time component
2-D simple random walk
Long term behavior?
Some properties
If \(0 = t_0\leq t_1 \leq \cdots \leq t_k\)
then \(X_{t_{i+1}}- X_{t_i}\) are mutally independent.
- \(E(X_k) = 0\)
- Independent increment
- Stationary
For all \(h \geq 1, t \geq 0\), the distribution of \(X_{t+h} - X_t\) is the same as the distribution of \(X_h\).
Example. A coin toss game...
At each turn, my balance goes up by $1 or down by $1.
My balance = Simple random walk
If I play until I win $100 or lose $100,
what is the probability of me winning?
If I play until I win $100 or lose $50,
what is the probability of me winning?
Code...
Brownian motion
is the "limit" of simple random walks.
Let \(Y_0, Y_1, ..., Y_n\) be a simple random walk.
\[Z\left(\frac{t}{n}\right) = Y_t\]
Interpolate linearly and take \(n \to \infty\).
Then, the resulting distribution is the Brownian motion.
A Markov chain
is a stochastic process where effect of the past on the future is summarized only by the current state.
Formally, ...
A discrete-time stochastic process \(X_0, X_1, X_2, ...\) with state space \(S\) has Markov property if
P(X_{t+1}=s|X_t, \cdots, X_1, X_0)= P(X_{t+1}=s|X_t)
\forall t \geq 0, \forall s \in S
If the state space \(S\) (\(X_t \in S\)) is finite,
all the elements of a Markov chain model
can be encoded in a transition probability matrix.
p_{ij} = P(X_{t+1}=j|X_t = i) \quad \forall i, j \in S
\sum_{j\in S}p_{ij} = 1
Why?
\begin{pmatrix}
p_{11} & p_{12} & \cdots & p_{1m}\\
p_{21} & p_{22} & \cdots & p_{2m}\\
\vdots & \vdots & \ddots & \\
p_{m1} & p_{m2} & & p_{mm}
\end{pmatrix}
transition probability
(jumping from state i to state j)
Is simple random walk a Markov chain?
Does simple random walk have a transition probability matrix?
P = \begin{pmatrix}
p_{11} & p_{12} & \cdots & p_{1m}\\
p_{21} & p_{22} & \cdots & p_{2m}\\
\vdots & \vdots & \ddots & \\
p_{m1} & p_{m2} & & p_{mm}
\end{pmatrix}
This matrix contains all the information about the stochastic process.
Why?
What is the probability of going from state 7 to state 9 in one steps?
What is the probability of going from state 7 to state 9 in two steps?
\[P_{79}\]
\[P^2_{79}\]
What is the probability of going from state 7 to state 9 in \(n\) steps?
An unrealistic example...
What is our transition probability matrix?
state
state
P = \begin{pmatrix}
0.9& 0.1 \\
0.3 & 0.7\\
\end{pmatrix}
Stationary distribution
\pi P = \pi \begin{pmatrix}
0.9& 0.1 \\
0.3 & 0.7\\
\end{pmatrix} = \pi
\pi = (? \quad ?)
Hidden Markov Model (HMM)
Hidden Markov Model (HMM)
Hidden Markov Model (HMM)
Speech recognition
- Observation: sounds
- hidden states: words
Face detection
- Observation: overlapping rectangles of pixel intensities
- hidden states: facial features
DNA sequence region classification
- Observation: nucleotides
- Hidden states: introns, exons
A few words on
Markov Chain Monte Carlo method (MCMC)
A martingale
is a stochastic process which is "fair game".
Formally, ...
A discrete-time stochastic process \(X_0, X_1, X_2, ...\) is a martingale if
X_t = E[X_{t+1}|\{X_0, X_1,\cdots, X_t\}]
\forall t \geq 0.
i.e., expected gain in the process is zero at all times.
Is a random walk a martingale?
A random variable \(X\) is distributed according to either \(f\) or \(g\). Consider a random sample \(X_1, \cdots, X_n\). Let \(Y_n\) be the likelihood ratio
\[Y_n = \prod_{i = 1}^n \frac {g(X_i)}{f(X_i)}\]
If X is actually distributed according to the density \(f\) rather than according to \(g\), then \[\{Y_n: n = 1, 2, 3, \cdots\}\] is a martingale with respect to\(\{X_n: n = 1, 2, 3, \cdots\}\).
Likelihood-ratio testing
Review
- Random walk
- Stochastic processes
- Markov chain
- Hidden Markov model
- MCMC
- Martingale
Resources
Random walk, Markov chain and Brownian motion
By Trang Le
Random walk, Markov chain and Brownian motion
Guest lecture for Penn BMIN 520-401 course, Spring 2020
