Markov Decision
Processes
Deterministic Search
Real World
Stylized Example: Grid World
state: location of the bot
+1 reward for the diamond, -1 reward for the fire
noisy movement: actions may not go as planned
Markov Decision Processes: Ingredients
States: \(S\)
Policy: map of states to actions
Q-values: expected future utility from a state+action pair
Actions: \(A\)
Transitions: \(T(s^\prime | s,a)\)
Rewards: \(R(s,a,s^\prime)\)
Start state: \(s_0\)
Utility: sum of discounted rewards
Values: expected future utility from a state
Discount: \(\gamma\)
How do you combine rewards?
\(r_1, r_2, r_3, \ldots\)
We can define some function \(f\)
\(f(r_1, r_2, r_3, \ldots)\)
that aggregates these rewards.
e.g, \(\sum_{i}r_i\)
or, e.g, \(\sum_{i}r_i \gamma^i\)
(rewards sooner are better than later)
How do you combine rewards?
\(\begin{gathered}\left[a_1, a_2, \ldots\right]\succ\left[b_1, b_2, \ldots\right] \\ \text{iff} \\ {\left[r, a_1, a_2, \ldots\right] \succ\left[r, b_1, b_2, \ldots\right]}\end{gathered}\)
e.g, \(\sum_{i}r_i\)
or, e.g, \(\sum_{i}r_i \gamma^i\)
(rewards sooner are better than later)
(stationary preference assumption)
Discounting
Exit action has a reward of 10
Exit action has a reward of 1
R
L
R
L
R
L
R
L
Assume actions are deterministic
\(r_1 + \gamma r_2 + \gamma^2 r_3 + \cdots \)
What if \(\gamma = 1\)?
What if \(\gamma = 0.1\)?
Discounting
Exit action has a reward of 10
Exit action has a reward of 1
R
L
R
L
R
L
R
L
Assume actions are deterministic
\(r_1 + \gamma r_2 + \gamma^2 r_3 + \cdots \)
What value of \(\gamma\) makes both L and R equally good when in the blue state?
Markov Decision Processes: Policies
In deterministic single-agent search problems,
we wanted an optimal plan, or sequence of actions, from start to a goal
For MDPs, we want an optimal policy,
or a map from \(S \rightarrow A\).
Optimal policies are denoted \(\pi^\star\).
Markov Decision Processes: Example
Markov Decision Processes: Example
When you decide to move N:
with 80% probability, you move N
with 10% probability, you move E
with 10% probability, you move W
Markov Decision Processes: Example
When you decide to move S:
with 80% probability, you move S
with 10% probability, you move E
with 10% probability, you move W
Markov Decision Processes: Example
When you decide to move W:
with 80% probability, you move W
with 10% probability, you move N
with 10% probability, you move S
Markov Decision Processes: Example
When you decide to move E:
with 80% probability, you move E
with 10% probability, you move N
with 10% probability, you move S
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Example
Markov Decision Processes: Racing
A robot car wants to travel far, quickly.
Three states:
Cool, Warm, Overheated
Two actions:
Slow Down; Go Fast
Infinite Rewards?
What if the game lasts forever?
Do we get infinite rewards?
Discounting
Finite Horizon
(Gives non-stationary policies;
\(\pi\) will depend on time left.)
\(U\left(\left[r_0, \ldots r_{\infty}\right]\right)=\sum_{t=0}^{\infty} \gamma^t r_t \\ \leq R_{\max } /(1-\gamma)\)
Absorbing
State
Markov Decision Processes
The Optimization Problem
Find:
\(\pi^*=\max _\pi, \forall s \in \mathcal{S}, V^\pi(s)\)
Given:
\(\left(\mathcal{S}, \mathcal{A}, \mathcal{P}_{s s^{\prime}}^a, \mathcal{R}_{s s^{\prime}}^a, \gamma\right)\)
Optimal Quanitities
The value (utility) of a state s: \(\mathrm{V}^*(\mathrm{~s})=\) expected utility starting in \(s\) and acting optimally
The value (utility) of a q-state \((s, a)\) :
\(Q^*(s, a)=\) expected utility starting out having taken action a from state \(s\) and (thereafter) acting optimally
The optimal policy: \(\pi^*(s)=\) optimal action from state \(s\)
Optimality Strategy
1. Take the first correct action
2. Keep making optimal choices
Bellman Equations
\(V^*(s)=\max _a Q^*(s, a)\)
\(Q^*(s, a)=\sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V^*\left(s^{\prime}\right)\right]\)
\(V^*(s)=\max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V^*\left(s^{\prime}\right)\right]\)
\(s\)
\((s,a)\)
\(s^\prime\)
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
\(V^*(s)=\max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V^*\left(s^{\prime}\right)\right]\)
\(V^*(s)=\max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V^*\left(s^{\prime}\right)\right]\)
Value Iteration: Example
\(V_{1}\)
\(V_{0}\)
\(V_{2}\)
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
\(V_k\)
\(V_{k+1}\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
\(V_k\)
\(V_{k+1}\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
\(V_{0}(s) \leftarrow 0\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
\(V_{0}(s) \leftarrow 0\)
\(V_{1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{0}\left(s^{\prime}\right)\right]\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
\(V_{0}(s) \leftarrow 0\)
\(V_{1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)\right]\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
\(V_{0}(s) \leftarrow 0\)
\(V_{1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)\right]\)
\(V_{2}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_1\left(s^{\prime}\right)\right]\)
Value Iteration: Convergence Intuition
\(V_{k+1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_k\left(s^{\prime}\right)\right]\)
What is the difference between \(V_k\) and \(V_{k+1}\)?
\(V_{k}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V_{k-1}\left(s^{\prime}\right)\right]\)
\(V_{0}(s) \leftarrow 0\)
\(V_{1}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)\right]\)
\(V_{2}(s) \leftarrow \max _a \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)\right] + T\left(s, a, s^{\prime}\right)\gamma {\color{IndianRed}V_{1}\left(s^{\prime}\right)}\)
\(V_k\) and \(V_{k+1}\) are at most \(\gamma^k \max |R|\) different
The values do converge as \(k\) increases if \(0 \leqslant \gamma < 1\).
Policy Evaluation
Policy Evaluation
\(\mathrm{V}^\pi(\mathrm{s})=\) expected total discounted rewards starting in s and following \(\pi\)
\(V^\pi(s)=\sum_{s^{\prime}} T\left(s, \pi(s), s^{\prime}\right)\left[R\left(s, \pi(s), s^{\prime}\right)+\gamma V^\pi\left(s^{\prime}\right)\right]\)
\(V_0^\pi(s)=0\)
\(V_{k+1}^\pi(s) \leftarrow \sum_{s^{\prime}} T\left(s, \pi(s), s^{\prime}\right)\left[R\left(s, \pi(s), s^{\prime}\right)+\gamma V_k^\pi\left(s^{\prime}\right)\right]\)
Policy Iteration
Start with an arbitrary policy.
\(V_{k+1}^{\pi_i}(s) \leftarrow \sum_{s^{\prime}} T\left(s, \pi_i(s), s^{\prime}\right)\left[R\left(s, \pi_i(s), s^{\prime}\right)+\gamma V_k^{\pi_i}\left(s^{\prime}\right)\right]\)
Evaluation
Improvement
\(\pi_{i+1}(s)=\underset{{\color{IndianRed}a}}{{\color{IndianRed}\arg \max} } \sum_{s^{\prime}} T\left(s, a, s^{\prime}\right)\left[R\left(s, a, s^{\prime}\right)+\gamma V^{\pi_i}\left(s^{\prime}\right)\right]\)
Policy Iteration v. Value Iteration
Both value iteration and policy iteration compute the same thing (all optimal values)
Every iteration updates both the values and (implicitly) the policy
We don't track the policy, but taking the max over actions
implicitly recomputes it
We do several passes that update utilities with fixed policy (each pass is fast because we consider only one action, not all of them)
After the policy is evaluated, a new policy is chosen
(slow like a value iteration pass)
The new policy will be better (or we're done)
\(Q^\pi(s, a)=E\left[\sum_{t=1}^{\infty} \gamma^{t-1} r_t \mid s_0=s, a_0=a, \pi\right]\)
Markov Decision Processes
The Q-Function
& the Value Function
The \(\gamma\) accounts for discounting.
\(V^\pi(s)=Q^\pi(s, \pi(s))\)