Outline

• Recap: Value iteration
• Tabular Q-learning
• Fitted/deep Q-learning

• Policy-based methods
  • High-level idea
  • Policy gradient

Recall:

\(\gamma = 0.9\)

States and one special transition:

• act optimally for one more timestep, at the next state \(s^{\prime}\) 

• receive \(\mathrm{R}(6,\uparrow)\)

\(\mathrm{Q}^h(s, a)\): the expected sum of discounted rewards for

• starting in state \(s\),
• take action \(a\), for one step
• act optimally there afterwards for the remaining \((h-1)\) steps

\(Q^2(6, \uparrow) =\mathrm{R}(6,\uparrow)  + \gamma[.2 \max _{a^{\prime}} Q^{1}\left(2, a^{\prime}\right)+ .8\max _{a^{\prime}} Q^{1}\left(3, a^{\prime}\right)] \)

Recall:

\(\gamma = 0.9\)

States and one special transition:

\(Q^h(s, a)\): the expected sum of discounted rewards for

• starting in state \(s\),
• take action \(a\), for one step
• act optimally there afterwards for the remaining \((h-1)\) steps

Outline

• Recap: Value iteration
• Tabular Q-learning
• Fitted/deep Q-learning

• Policy-based methods
  • High-level idea
  • Policy gradient 

• Without \(\mathrm{R}\) and \(\mathrm{T}\), how about we approximate like so:
  • pick an \((s,a)\) pair
  • execute \((s,a)\)
  • observe \(r\) and \(s'\)
  • update:

States and unknown transition:

Game Set up

Try using

unknown rewards:

States and unknown transition:

Try out

• execute \((6, \uparrow)\)

• update \(\mathrm{Q}(6, \uparrow)\) as:

•  suppose, we observe a reward \(r=-10\), the next state \(s'=3\)

\(\gamma = 0.9\) 

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

States and unknown transition:

• execute \((6, \uparrow)\) again

• update \(\mathrm{Q}(6, \uparrow)\) as:

•  suppose, we observe a reward \(r=-10\), the next state \(s'=2\)

\(\gamma = 0.9\) 

Try out

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

States and unknown transition:

Try out

• execute \((6, \uparrow)\) again

• update \(\mathrm{Q}(6, \uparrow)\) as:

•  suppose, we observe a reward \(r=-10\), the next state \(s'=3\)

\(\gamma = 0.9\) 

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

States and unknown transition:

• execute \((6, \uparrow)\) again

• update \(\mathrm{Q}(6, \uparrow)\) as:

•  suppose, we observe a reward \(r=-10\), the next state \(s'=2\)

\(\gamma = 0.9\) 

Try out

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

States and unknown transition:

Try out

• execute \((6, \uparrow)\) again

• update \(\mathrm{Q}(6, \uparrow)\) as:

\(-10 + 0.9 \max _{a^{\prime}} \mathrm{Q}_{\text {old }}\left(3, a^{\prime}\right)\)

= -10 + 0.9 = -9.1

To update the estimate of \(\mathrm{Q}(6, \uparrow)\):

•  suppose, we observe a reward \(r=-10\), the next state \(s'=3\)

\(\gamma = 0.9\) 

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

• Indeed, value iteration relied on having full access to \(\mathrm{R}\) and \(\mathrm{T}\)

• Without \(\mathrm{R}\) and \(\mathrm{T}\), perhaps we could execute \((s,a)\), observe \(r\) and \(s'\), and use

• But target keeps "washing away" the old progress.

• Indeed, value iteration relied on having full access to \(\mathrm{R}\) and \(\mathrm{T}\)

• Without \(\mathrm{R}\) and \(\mathrm{T}\), perhaps we could execute \((s,a)\), observe \(r\) and \(s'\), and use

• execute \((6, \uparrow)\)

• update \(\mathrm{Q}(6, \uparrow)\) as:

•  suppose, we observe a reward \(r=-10\), the next state \(s'=3\)

States and unknown transition:

Better idea:

\(\gamma = 0.9\) 

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

To update the estimate of \(\mathrm{Q}(6, \uparrow)\):

• execute \((6, \uparrow)\) again

\((-10\)

•  suppose, we observe a reward \(r=-10\), the next state \(s'=2\)

States and unknown transition:

Better idea:

\(\gamma = 0.9\) 

pick learning rate \(\alpha =0.5\)

+ 0.5 

\(\mathrm{Q}_\text{old}(s, a)\)

\(\mathrm{Q}_{\text{new}}(s, a)\)

To update the estimate of \(\mathrm{Q}(6, \uparrow)\):

• update \(\mathrm{Q}(6, \uparrow)\) as:

\(+ 0.9 \max _{a^{\prime}} \mathrm{Q}_{\text {old }}\left(2, a^{\prime}\right))\)

"learning"

Q-Learning \(\left(\mathcal{S}, \mathcal{A}, \gamma, \alpha, s_0\right. \text{max-iter})\)

1. \(i=0\)
2. for \(s \in \mathcal{S}, a \in \mathcal{A}:\)
3.       \({\mathrm{Q}_\text{old}}(s, a) = 0\)
4. \(s \leftarrow s_0\)
5. while \(i < \text{max-iter}:\)
6.       \(a \gets \text{select}\_\text{action}(s, {\mathrm{Q}_\text{old}}(s, a))\)
7.       \(r,s' \gets \text{execute}(a)\)
8.      \({\mathrm{Q}}_{\text{new}}(s, a)  \leftarrow (1-\alpha){\mathrm{Q}}_{\text{old}}(s, a) + \alpha(r + \gamma \max_{a'}{\mathrm{Q}}_{\text{old}}(s', a'))\)
9.      \(s  \leftarrow s'\)
10.    \(i  \leftarrow (i+1)\)
11.      \(\mathrm{Q}_{\text{old}}  \leftarrow \mathrm{Q}_{\text{new}}\)
12. return \(\mathrm{Q}_{\text{new}}\)

• Remarkably, 👈 can converge to the true infinite-horizon Q-values\(^1\). 

• But the convergence can be extremely slow.

• During learning, especially in early stages, we'd like to explore, and observe diverse \((s,a\)) consequences.

"learning"

Q-Learning \(\left(\mathcal{S}, \mathcal{A}, \gamma, \alpha, s_0\right. \text{max-iter})\)

1. \(i=0\)
2. for \(s \in \mathcal{S}, a \in \mathcal{A}:\)
3.       \({\mathrm{Q}_\text{old}}(s, a) = 0\)
4. \(s \leftarrow s_0\)
5. while \(i < \text{max-iter}:\)
6.       \(a \gets \text{select}\_\text{action}(s, {\mathrm{Q}_\text{old}}(s, a))\)
7.       \(r,s' \gets \text{execute}(a)\)
8.      \({\mathrm{Q}}_{\text{new}}(s, a)  \leftarrow (1-\alpha){\mathrm{Q}}_{\text{old}}(s, a) + \alpha(r + \gamma \max_{a'}{\mathrm{Q}}_{\text{old}}(s', a'))\)
9.      \(s  \leftarrow s'\)
10.    \(i  \leftarrow (i+1)\)
11.      \(\mathrm{Q}_{\text{old}}  \leftarrow \mathrm{Q}_{\text{new}}\)
12. return \(\mathrm{Q}_{\text{new}}\)

Outline

• Recap: Value iteration
• Tabular Q-learning
• Fitted/deep Q-learning

• Policy-based methods
  • High-level idea
  • Policy gradient

Tabular Q-learning limitations

• What do we do if \(\mathcal{S}\) and/or \(\mathcal{A}\) are large, or even continuous?

• Notice that the key update line in Q-learning algorithm:

• Reminds us of: when minimizing \((\text{target} - \text{guess}_{\theta})^2\)

\(\mathrm{Q}_{\text {new}}(s, a) \leftarrow\mathrm{Q}_{\text {old }}(s, a)+\alpha\left([r+\gamma \max _{a^{\prime}} \mathrm{Q}_{\text {old}}(s', a')] - \mathrm{Q}_{\text {old }}(s, a)\right)\)

new belief

\(\leftarrow\)

old belief

learning rate

target

old belief

1. parameterize \(\mathrm{Q}_{\theta}(s,a)\) 
2. collect data \((r, s')\) to construct the target
3. update \(\theta\) via gradient-descent methods to minimize                    

  Fitted Q-learning limitations

 Q-learning limitations

  Fitted Q-learning limitations

General value-based method limitations

Quick Summary

• In reinforcement learning, we assume we are interacting with an unknown MDP, but we still want to find a good policy.  We will do so via estimating the Q value function.
• One problem is how to select actions to gain good reward while learning.  This “exploration vs exploitation” problem is important.
• Q-learning, for discrete-state problems, will converge to the optimal value function (with enough exploration).
• “Deep Q learning” can be applied to continuous-state or large discrete-state problems by using a parameterized function to represent the Q-values. 

Outline

• Recap: Value iteration
• Tabular Q-learning
• Fitted/deep Q-learning

• Policy-based methods
  • High-level idea
  • Policy gradient

• Often \(\pi\) can be simpler than \(Q\) or \(V\)
  • e.g. lots of \(\pi\) are roughly good
• \(V(s)\) : doesn't prescribe actions
• Q: need to be able to efficiently solve \(\arg \max_a Q(s, a)\), can also be challenging for continuous / high-dimensional action spaces
• Maybe makes sense to direct optimize policy end-to-end
• So how do we do this?

• Parameterize policy by \(\theta\) and directly try \(\max _\theta \mathbb{E}\left[\sum \gamma^t R\left(s_t, a_t\right) \mid \pi_\theta\right]\)
• Stochastic policy class \(\pi_\theta(a \mid s)\) : probability of action a in state s
  • Discrete \(\mathcal{A}\) : e.g. \(\pi_\theta(a \mid s)\) softmax
  • Continuous \(\mathcal{A}\) : e.g. \(\pi_\theta(a \mid s)\) Gaussian with mean/variance parameterized by \(\theta\)
• Smoothes out the optimization problem
• Also encourages exploration

Outline

• Recap: Value iteration
• Tabular Q-learning
• Fitted/deep Q-learning

• Policy-based methods
  • High-level idea
  • Policy gradient

Policy Gradient Derivation

• We overload notation:
• Let \(\tau\) denote a state-action sequence: \(\tau=s_0, a_0, s_1, a_1, \ldots\)
• Let \(R(\tau)\) denote the sum of discounted rewards on \(\tau: R(\tau)=\sum_t \gamma^t R\left(s_t, a_t\right)\)
• W.l.o.g. assume \(R(\tau)\) is deterministic in \(\tau\)
• Let \(P(\tau ; \theta)\) denote the probability of trajectory \(\tau\) induced by \(\pi_\theta\)
• Let \(U(\theta)\) denote the objective: \(U(\theta)=\mathbb{E}\left[\sum_t \gamma^t R\left(s_t, a_t\right) \mid \pi_\theta\right]\)
• Our goal is to find \[\theta: \max _\theta U(\theta)=\max _\theta \sum_\tau P(\tau ; \theta) R(\tau)\]

Policy Gradient Derivation

Policy Gradient Derivation

\(=\sum_\tau P(\tau ; \theta) \nabla_\theta \log P(\tau ; \theta) R(\tau)\)

Policy Gradient Derivation

\(=\sum_\tau P(\tau ; \theta) \nabla_\theta \log P(\tau ; \theta) R(\tau)\)

Approximate with the empirical estimate for \(m\) sample traj. under policy \(\pi_\theta\)

\(\nabla_\theta U(\theta) \approx \hat{g}=\frac{1}{m} \sum_{i=1}^m \nabla_\theta \log P\left(\tau^{(i)} ; \theta\right) R\left(\tau^{(i)}\right)\)

Policy Gradient Derivation

\(\nabla_\theta U(\theta) \approx \hat{g}=\frac{1}{m} \sum_{i=1}^m \nabla_\theta \log P\left(\tau^{(i)} ; \theta\right) R\left(\tau^{(i)}\right)\)

Policy Gradient Derivation

\(=\sum_{t=0} \underbrace{\nabla_\theta \log \pi_\theta\left(a_t \mid s_t\right)}_{\text {no transition model required, }}\)

\(\nabla_\theta \log P(\tau ; \theta)\)

\(\nabla_\theta U(\theta) \approx \hat{g}=\frac{1}{m} \sum_{i=1}^m \nabla_\theta \log P\left(\tau^{(i)} ; \theta\right) R\left(\tau^{(i)}\right)\)

The following expression provides us with an unbiased estimate of the gradient, and we can compute it without access to the world model:

where