Toddler demo, Russ Tedrake thesis, 2004
Markov Decision Processes Definition
Policy Evaluation
Policy Optimization
Markov Decision Processes Definition
Policy Evaluation
Policy Optimization
like computing the loss: the weights are given, we score them
like training: we search for good weights
like computing the loss: the weights are given, we score them
like training: we search for good weights
Normally, we get to the “intended” state;
E.g., in state (7), action “↑” gets to state (4)
If an action would take Mario off the grid, he stays put;
E.g., in state (9), “→” gets back to state (9)
In state (6), action “↑” leads to two possibilities:
20% chance to (2)
80% chance to (3)
Running example: Mario in a grid-world
reward of (3, \(\downarrow\))
reward of \((3,\uparrow\))
reward of \((6, \downarrow\))
reward of \((6,\rightarrow\))
Mario in a grid-world, cont'd
The reward is for taking the action, not for arriving.
e.g. moving up from state 6 earns a reward of \(-10\), regardless of whether we end in state 2 or 3
e.g. the reward 1 from \((3, \leftarrow)\) is worth \(1,\ 0.9,\ 0.9^2, \dots\) at time steps 0, 1, 2, …
Markov Decision Process (MDP)
\(\mathrm{T}\left(7, \uparrow, 4\right) = 1\)
\(\mathrm{T}\left(9, \rightarrow, 9\right) = 1\)
\(\mathrm{T}\left(6, \uparrow, 3\right) = 0.8\)
\(\mathrm{T}\left(6, \uparrow, 2\right) = 0.2\)
In 6.390,
Markov Decision Process (MDP)
reward of \((3,\uparrow\))
reward of \((6,\rightarrow\))
\(\mathrm{R}\left(3, \uparrow \right) = 1\)
\(\mathrm{R}\left(6, \rightarrow \right) = -10\)
In 6.390,
The goal of an MDP is to find a good policy.
Markov Decision Process (MDP)
In 6.390,
Policy \(\pi(s)\)
Transition \(\mathrm{T}\left(s, a, s^{\prime}\right)\)
Reward \(\mathrm{R}(s, a)\)
time
a trajectory (also called an experience or rollout) of horizon \(h\)
\(\quad \tau=\left(s_0, a_0, r_0, s_1, a_1, r_1, \ldots, s_{h-1}, a_{h-1}, r_{h-1}\right)\)
initial state
all depend on \(\pi\)
\(0\)
\(h\)
\(1\)
\(h-1\)
\(2\)
\(h-2\)
\(3\)
\(h-3\)
\(4\)
\(h-4\)
\(5\)
\(h-5\)
\(h-2\)
\(2\)
\(h-1\)
\(1\)
clock \(t\)
remaining horizon
Markov Decision Processes Definition
Policy Evaluation
State value functions: \(\mathrm{V}^{\pi}\)
Bellman recursions and Bellman equations
Policy Optimization
Starting in state \(s\), how good is it to follow a given policy \(\pi\) for \(h\) time steps?
One idea:
But if we start at \(s = 6\) and follow the “always-up” policy:
👈
states and one special transition:
rewards:
trajectory:
Value functions
(eq. 1️⃣)
Outside 6.390, the expectation can also be over random rewards and random policies.
\( h\) terms
evaluate \(\mathrm{V}_h^\pi(s)\) under the “always-up” policy
states and
one special transition:
rewards
horizon \(h\) = 0: no steps left
horizon \(h\) = 1: receive the rewards
horizon \(h = 2\)
states and
one special transition:
rewards
\( 2\) terms
action \(\uparrow\)
action \(\uparrow\)
action \(\uparrow\)
horizon \(h = 2\)
states and
one special transition:
rewards
\( 2\) terms
action \(\uparrow\)
action \(\uparrow\)
action \(\uparrow\)
action \(\uparrow\)
action \(\uparrow\)
states and
one special transition:
rewards
horizon \(h = 3\)
\( 3\) terms
the immediate reward for taking the policy-prescribed action \(\pi(s)\) in state \(s\).
horizon-\(h\) value in state \(s\): the expected sum of discounted rewards, starting in state \(s\) and following policy \(\pi\) for \(h\) steps.
\((h-1)\) horizon future value at a next state \(s^{\prime}\)
sum of future values weighted by the probability of reaching that next state \(s^{\prime}\)
discounted by \(\gamma\)
(eq. 2️⃣)
Bellman Recursion (eq. 2️⃣)
states and
one special transition:
rewards
states and
one special transition:
rewards
Bellman Recursion (eq. 2️⃣)
Check-in:
to fill one entry of the \(\mathrm{V}_3\) table, which tables do we need?
Just \(\mathrm{V}_2\) and \(\mathrm{R}\).
states and
one special transition:
rewards
Bellman Recursion (eq. 2️⃣)
states and
one special transition:
rewards
Value functions converge as \(h \to \infty\)
Typically, \(\gamma < 1\) to ensure \(\mathrm{V}_{\infty}\) is finite.
Recursion (finite \(h\)) 2️⃣
As horizon \(h \to \infty,\) the Bellman recursion becomes the Bellman equation
Equation \((h\to \infty)\) 3️⃣
A system of \(|\mathcal{S}|\) self-consistent linear equations, one for each state
two different tables: \(\mathrm{V}_h\) is filled in from \(\mathrm{V}_{h-1}\)
the same table on both sides
Bellman Equation (eq. 3️⃣)
states and
one special transition:
rewards
finite horizon: Bellman recursion
infinite horizon: Bellman equation
Policy Evaluation
Quick summary
By definition:
Or, by recursion:
1️⃣
2️⃣
3️⃣
\(\mathrm{V}^{\pi}_h(s)\): the value of state \(s\), under policy \(\pi\), with \(h\) steps to go
Markov Decision Processes Definition
Policy Evaluation
Policy Optimization
Optimal policies \(\pi^*\)
Optimal state-action value functions: \(\mathrm{Q}^*\)
Value iteration
For a given horizon \(h\), an optimal policy \(\pi^*_h\) is one such that:
\(\mathrm{V}^*_h(s) = \mathrm{V}^{\pi^*_h}_h(s) \ge \mathrm{V}^{\pi}_h(s)\) for every state \(s\) and every policy \(\pi\)
Optimal policy \(\pi^*\)
\(\mathrm{V}^{\uparrow}_{\infty}(s)\)
\(\mathrm{V}^{*}_{\infty}(s)\)
\(\ge\)
in every
state
For example, for Mario:
acting optimally, horizon \(h = 1\)
states and
one special transition:
rewards
\(\mathrm{V}^*_1(s)\)
from 3
\(a_0 = \,{\uparrow}\)
\(a_0 = \,{\leftarrow}\)
\(a_0 = \,{\downarrow}\)
\(a_0 = \,{\rightarrow}\)
reward
\(r_0 = 1\)
\(r_0 = 1\)
\(r_0 = 1\)
\(r_0 = 1\)
acting optimally, horizon \(h = 2\)
states and
one special transition:
rewards
\(\mathrm{V}^*_1(s)\)
\(\mathrm{V}^*_2(s)\)
from 3
\(a_0 = \,{\uparrow}\)
\(a_0 = \,{\leftarrow}\)
\(a_0 = \,{\downarrow}\)
\(a_0 = \,{\rightarrow}\)
reward
\(r_0 = 1\)
\(r_0 = 1\)
\(r_0 = 1\)
\(r_0 = 1\)
lands in
\(s_1 = 3\)
\(s_1 = 2\)
\(s_1 = 6\)
\(s_1 = 3\)
\(r_0 + 0.9\,\mathrm{V}^*_1(s_1)\)
\(1 + 0.9 \cdot \mathrm{V}^*_1(3) = 1.9\)
\(1 + 0.9 \cdot \mathrm{V}^*_1(2) = 1\)
\(1 + 0.9 \cdot \mathrm{V}^*_1(6) = -8\)
\(1 + 0.9 \cdot \mathrm{V}^*_1(3) = 1.9\)
\(\mathrm{V}_h^*(s) = \max_{a} \big[\mathrm{R}(s, a) + \gamma \sum_{s'} \mathrm{T}(s, a, s') \mathrm{V}_{h-1}^*(s') \big]\)
Policy \(\pi(s)\)
Transition \(\mathrm{T}\left(s, a, s^{\prime}\right)\)
Reward \(\mathrm{R}(s, a)\)
time
act optimally for \(h\) steps: \(\mathrm{V}_h^*(s)\)
then act optimally for the remaining \((h-1)\) steps: \(\mathrm{V}_{h-1}^*(s')\)
(new, eq. 4️⃣, for an optimal policy)
(recall, eq. 2️⃣, for any policy)
a first action \(a\),
chosen with the future in mind
\(s_1 = s'\)
\(\mathrm{Q}^*_h(s, a)\), the optimal state-action value function, i.e. the value for:
(eq. 5️⃣)
(eq. 4️⃣)
\(\mathrm{V}_h^*(s) = \max_{a} \big[\mathrm{R}(s, a) + \gamma \sum_{s'} \mathrm{T}(s, a, s') \mathrm{V}_{h-1}^*(s') \big]\)
\(= \max_{a} \kern0.38em \mathrm{Q}^*_h(s, a)\)
\(\mathrm{Q}^*_h(s, a) = \mathrm{R}(s, a) + \gamma \sum_{s'} \mathrm{T}(s, a, s') \max_{a'} \mathrm{Q}^*_{h-1}(s', a')\)
From eq. 4 to eq. 5:
\(\begin{aligned} \mathrm{Q}^*_h(s, a) &= \mathrm{R}(s, a) + \gamma \sum\nolimits_{s'} \mathrm{T}(s, a, s')\, \mathrm{V}_{h-1}^*(s') && \text{(the two green boxes in eq. 4 are equal)} \\ &= \mathrm{R}(s, a) + \gamma \sum\nolimits_{s'} \mathrm{T}(s, a, s') \max\nolimits_{a'} \mathrm{Q}^*_{h-1}(s', a') && \text{(eq. 4 at horizon } h-1 \text{: } \mathrm{V}_{h-1}^*(s') = \max\nolimits_{a'} \mathrm{Q}^*_{h-1}(s', a') \text{)} \end{aligned}\)
\(\mathrm{Q}^*_0\) and \(\mathrm{Q}^*_1\)
states and
one special transition:
rewards
eq. 5️⃣ with \(h = 1\):
\(\mathrm{Q}^*_1(s,a)\)
\(= \mathrm{R}(s,a)\) \(+\ \gamma \sum_{s^{\prime}} \mathrm{T}(s,a,s^{\prime}) \max_{a^{\prime}} \mathrm{Q}^*_0(s^{\prime},a^{\prime})\)
\(= \mathrm{R}(s,a)\) since \(\mathrm{Q}^*_0\) is all zeros
\(\mathrm{Q}^*_h(s, a)\): the value for
starting in state \(s\),
take action \(a\), for one step
act optimally thereafter for the remaining \((h-1)\) steps
by convention: with no
steps left, every value is 0
building \(\mathrm{Q}^*_2\): the entry \((3, \downarrow)\)
states and
one special transition:
rewards
receive \(\mathrm{R}(3,\downarrow)\)
then act optimally from \(s' = 6\)
get \(\max_{a'} \mathrm{Q}^*_1(6, a')\)
\(\mathrm{Q}_2^*(3, \downarrow) = \mathrm{R}(3,\downarrow) + \gamma \max _{a^{\prime}} \mathrm{Q}_{1}^*\left(6, a^{\prime}\right)\)
\( = 1 + .9\times -10\)
\( = -8\)
building \(\mathrm{Q}^*_2\): the entry \((6, \uparrow)\)
states and
one special transition:
rewards
\(\mathrm{Q}_2^*(6, \uparrow)=\mathrm{R}(6,\uparrow) + \gamma[.2 \max _{a^{\prime}} \mathrm{Q}_{1}^*\left(2, a^{\prime}\right)+ .8\max _{a^{\prime}} \mathrm{Q}_{1}^*\left(3, a^{\prime}\right)] \)
\(= -10 + .9 [.2 \times 0+ .8 \times 1] \Rightarrow -9.28\)
receive \(\mathrm{R}(6,\uparrow)\)
then act optimally from \(s'\):
20%: \(s' = 2\), get \(\max_{a'} \mathrm{Q}^*_1(2, a')\)
80%: \(s' = 3\), get \(\max_{a'} \mathrm{Q}^*_1(3, a')\)
The same process repeats for every other state and action.
\(\mathrm{V}^*\) is the max of \(\mathrm{Q}^*\)
\(\mathrm{Q}^*_2(s, a)\)
\(\mathrm{V}^*_2(s)\)
\(\max_a\)
building \(\mathrm{Q}^*_3\): the entry \((6, \uparrow)\)
states and
one special transition:
rewards
\(\mathrm{Q}_3^*(6, \uparrow)=\mathrm{R}(6,\uparrow) + \gamma[.2 \max _{a^{\prime}} \mathrm{Q}_{2}^*\left(2, a^{\prime}\right)+ .8\max _{a^{\prime}} \mathrm{Q}_{2}^*\left(3, a^{\prime}\right)] \)
receive \(\mathrm{R}(6,\uparrow)\)
then act optimally from \(s'\):
20%: \(s' = 2\), get \(\max_{a'} \mathrm{Q}^*_2(2, a')\)
80%: \(s' = 3\), get \(\max_{a'} \mathrm{Q}^*_2(3, a')\)
\(= -10 + .9 [.2 \times 0.9 + .8 \times 1.9] \Rightarrow -8.47\)
The same process repeats for every other state and action.
Value Iteration
if we run this block \(h\) times and then break, it returns exactly \(\mathrm{Q}^*_h\)
\(\mathrm{Q}^*_{\infty}(s, a)\)
Value iteration: build \(\mathrm{Q}^*\) one horizon at a time (eq. 5️⃣)
so \(\mathrm{Q}_{\text{old}} = \mathrm{Q}^*_0\)
if \(\mathrm{Q}_{\text{old}} = \mathrm{Q}^*_{h-1}\), then \(\mathrm{Q}_{\text{new}} = \mathrm{Q}^*_h\)
So, how do we act? Read it off \(\mathrm{Q}^*\): 6️⃣
e.g. the best actions to take in state 5
first action
remaining \(h-1\) steps
how we compute it
\(\mathrm{V}^{\pi}_h(s)\)
\(\pi(s)\), the policy's pick
keep following \(\pi\)
definition 1️⃣, recursion 2️⃣
as \(h \rightarrow \infty\), equation 3️⃣
\(\mathrm{V}^{*}_h(s)\)
the optimal first action
act optimally
\(\max_a \mathrm{Q}^*_h(s, a)\)
from the principle of optimality 4️⃣
\(\mathrm{Q}^{*}_h(s, a)\)
any action \(a\) we name
act optimally
value iteration 5️⃣
MDPs are also called stochastic optimal control, stochastic shortest path, or sequential decision making under uncertainty.