Sarah Dean PRO
asst prof in CS at Cornell
Prof. Sarah Dean
MW 2:55-4:10pm
255 Olin Hall
1. Recap: Markov Decision Process
2. Trajectories and Distributions
3. State Distributions
action \(a_t\)
state \(s_t\)
\(\sim \pi_t(s_t)\)
reward
\(r_t\sim r(s_t, a_t)\)
\(s_{t+1}\sim P(s_t, a_t)\)
state
\(s_t\)
Markovian Assumption: Conditioned on \(s_t,a_t\), the reward \(r_t\) and next state \(s_{t+1}\) are independent of the past.
When state transition are stochastic, we will write either:
\(0\)
\(1\)
Example:
\(0\)
\(1\)
$$\mathbb E\left[\sum_{t=0}^{H-1} r_t\right]$$
\(\mathcal M = \{\mathcal{S}, \mathcal{A}, r, P, H\}\)
1. Recap: Markov Decision Process
2. Trajectories and Distributions
3. State Distributions
\(s_0\)
\(a_0\)
\(s_1\)
\(a_1\)
\(s_2\)
\(a_2\)
...
\(0\)
\(1\)
stay: \(1\)
move: \(1\)
stay: \(p_1\)
move: \(1-p_2\)
stay: \(1-p_1\)
move: \(p_2\)
\(s_0\)
\(a_0\)
\(s_1\)
\(a_1\)
\(s_2\)
\(a_2\)
...
First recall Baye's rule: For events \(A\) and \(B\),
$$ \mathbb P\{A \cap B\} = \mathbb P\{A\}\mathbb P\{B\mid A\} = \mathbb P\{B\} \mathbb P\{B\mid A\}$$
Why?
Then we have that
$$\mathbb P_{\mu_0}^\pi(s_0, a_0) = \mathbb P\{s_0\} \mathbb P\{a_0\mid s_0\} = \mu_0(s_0)\pi(a_0\mid s_0)$$
then
$$\mathbb P_{\mu_0}^\pi(s_0, a_0, s_1) = \mathbb P_{\mu_0}^\pi(s_0, a_0)\underbrace{\mathbb P\{s_1\mid a_0, s_0\}}_{P(s_1\mid s_0, a_0)}$$
and so on
\(s_0\)
\(a_0\)
\(s_1\)
\(a_1\)
\(s_2\)
\(a_2\)
...
Note: the above assumes that the reward function is deterministic
1. Recap: Markov Decision Process
2. Trajectories and Distributions
3. State Distributions
Example: \(\pi_t(s)=\)stay and \(\mu_0\) is each state with probability \(1/2\).
Probability of state \(s\) at time \(t\) $$ \mathbb{P}\{s_t=s\} = \displaystyle\sum_{\substack{s_{0:t-1}\in\mathcal S^t}} \mathbb{P}^\pi_{\mu_0} (s_{0:t-1}, s) $$
\(1\)
\(1-p_1\)
\(p_1\)
\(0\)
\(1\)
Why? First recall that
$$ \mathbb P\{A \cup B\} = \mathbb P\{A\}+\mathbb P\{B\} - \mathbb P\{A\cap B\}$$
If \(A\) and \(B\) are disjoint, the final term is \(0\) by definition.
If all \(A_i\) are disjoint events, then the probability any one of them happens is
$$ \mathbb P\{\cup_{i} A_i\} = \sum_{i} \mathbb P\{A_i\}$$
$$d_0 = \begin{bmatrix} 1/2\\ 1/2\end{bmatrix} \quad d_1 = \begin{bmatrix} 1-p_1/2\\ p_1/2\end{bmatrix}\quad d_2 = \begin{bmatrix} 1-p_1^2/2\\ p_1^2/2\end{bmatrix}$$
Example: \(\pi_t(s)=\)stay and \(\mu_0\) is each state with probability \(1/2\).
\(1\)
\(1-p_1\)
\(p_1\)
\(0\)
\(1\)
$$d_0 = \begin{bmatrix} 1/2\\ 1/2\end{bmatrix} \quad d_1 = \begin{bmatrix} 1-p_1/2\\ p_1/2\end{bmatrix}\quad d_2 = \begin{bmatrix} 1-p_1^2/2\\ p_1^2/2\end{bmatrix}$$
Example: \(\pi_t(s)=\)stay and \(\mu_0\) is each state with probability \(1/2\).
\(1\)
\(1-p_1\)
\(p_1\)
\(0\)
\(1\)
State distribution lets us compute \(\mathbb E[r_t] = \sum_{s\in\mathcal S}d_t[s]r(s,\pi_t(s))\)
Given a policy \(\pi_t(\cdot)\) and a transition function \(P(\cdot\mid \cdot,\cdot)\)
\(P_{\pi_t}=\)
\(s\)
\(s'\)
\(P(s'\mid s,\pi_t(s))\)
Proposition: The state distribution evolves according to $$ d_t = P_{\pi_{t-1}}^\top\dots P_{\pi_1}^\top P_{\pi_0}^\top d_0\quad t\geq 1$$
Proof: (by induction)
\(s\)
\(s'\)
\(P(s'\mid s,\pi_t(s))\)
\(P_{\pi_t}^\top=\)
Proof of claim that \(d_{k+1} = P_{\pi_k}^\top d_k\)
\(d_{k+1}[s'] =\sum_{s\in\mathcal S} P(s'\mid s, \pi_k(s))d_k[s]\)
\(\textcolor{orange}{d_{k+1}[s']} =\langle\)\(\begin{bmatrix} P(s'\mid 1, \pi_k(1)) & \dots & P(s'\mid S, \pi_k(S))\end{bmatrix}\)\(,\textcolor{red}{d_k}\rangle \)
Each entry of \(d_{k+1}\) is the inner product of a column of \(P_{\pi_k}\) with \(d_k\)
in other word, inner product of a row of \(P_{\pi_k}^\top\) with \(d_k\)
By the definition of matrix multiplication, \(d_{k+1} = P_{\pi_k}^\top d_k\)
\(s\)
\(s'\)
\(P(s'\mid s,\pi_k(s))\)
\(P_{\pi_k}=\)
=
\(1\)
\(1-p_1\)
\(p_1\)
\(0\)
\(1\)
Example: \(\pi(s)=\)stay and \(\mu_0\) is each state with probability \(1/2\).
$$P_\pi = \begin{bmatrix}1& 0\\ 1-p_1 & p_1\end{bmatrix}$$
1. Markov Decision Process
2. Trajectories and Distributions
3. State Distributions
By Sarah Dean