Reminders

• Homework
  • PA 3 due Friday 3/31
  • PSet 4 due Wednesday Friday
  • 5789 Paper Reviews due weekly on Mondays
    • Hard deadline for 3 reviews by Friday
• My Tuesday office hours moved this week to 3-4pm

Agenda

1. Recap

2. Policy Optimization

3. with Trajectories

4. with Value

Recap: SGA

• Rather than exact gradients, SGA uses unbiased estimates of the gradient $g_i$, i.e. $$\mathbb E[g_i|\theta_i] = \nabla J(\theta_i)$$

Algorithm: SGA

• Initialize $\theta_0$; For $i=0,1,...$:
  • $\theta_{i+1} = \theta_i + \alpha g_i$

$\theta_1$

$\theta_2$

$\theta_1$

$\theta_2$

• gradient ascent
• stochastic gradient ascent

2D quadratic function

level sets of quadratic

Recap: DFO

• Random Search
  • Based on finite difference approximation
  • $g_i=\frac{1}{2\delta} (J(\theta_i+\delta v) - J(\theta_i - \delta v))v$ for $v\sim \mathcal N(0,I)$
• Montecarlo / Importance Weighting
  • Suppose that $J(\theta) = \mathbb E_{z\sim P_\theta}[h(z)]$
  • $g_i=\left[\nabla_{\theta}\log P_\theta(z)\right]_{\theta=\theta_i} h(z)$ for $z\sim P_{\theta_i}$

$\nabla J(\theta)$$\approx \frac{1}{2\delta} (J(\textcolor{cyan}{\theta}+{\delta v}) - J(\textcolor{cyan}{\theta}-{\delta v}))\textcolor{LimeGreen}{v}$

$J(\theta) = -\theta^2 - 1$

$\theta$

Recap: Random Search Example

• start with $\theta$ positive
• suppose $v$ is positive
• then $J(\theta+\delta v)<J(\theta-\delta v)$
• therefore $g$ is negative
• indeed, $\nabla J(\theta) = -2\theta<0$ when $\theta>0$

Recap: Montecarlo Example

$\nabla J(\theta)$$\approx \nabla_\theta \log(P_\theta(z)) h(z)$

$J(\theta) = \mathbb E_{z\sim P_\theta}[h(z)]$

$z$

• start with $\theta$ positive
• suppose $z>\theta$
• then score is positive
• therefore $g$ is negative (since $h(z)<0$)

Agenda

1. Recap

2. Policy Optimization

3. with Trajectories

4. with Value

$\mathcal M = \{\mathcal{S}, \mathcal{A}, r, P, \gamma, \mu_0\}$

• Goal: achieve high expected cumulative reward:

  $$\max_\pi ~~\mathbb E \left[\sum_{t=0}^\infty \gamma^t r(s_t, a_t)\mid s_0\sim \mu_0, s_{t+1}\sim P(s_t, a_t), a_t\sim \pi(s_t)\right ]$$

• Recall notation for a trajectory $\tau = (s_0, a_0, s_1, a_1, \dots)$ and probability of a trajectory $\mathbb P^{\pi}_{\mu_0}$
• Define cumulative reward $R(\tau) = \sum_{t=0}^\infty \gamma^t r(s_t, a_t)$
• For parametric (e.g. deep) policy $\pi_\theta$, the objective is: $$J(\theta) = \mathop{\mathbb E}_{\tau\sim \mathbb P^{\pi_\theta}_{\mu_0}}\left[R(\tau)\right]$$

Policy Optimization Setting

$\mathcal M = \{\mathcal{S}, \mathcal{A}, r, P, \gamma, \mu_0\}$

• Goal: achieve high expected cumulative reward:

  $$\max_\theta ~~J(\theta)= \mathop{\mathbb E}_{\tau\sim \mathbb P^{\pi_\theta}_{\mu_0}}\left[R(\tau)\right]$$

• Assume that we can "rollout" policy $\pi_\theta$ to observe:

  • a sample $\tau$ from $\mathbb P^{\pi_\theta}_{\mu_0}$

  • the resulting cumulative reward $R(\tau)$

• Note: we do not need to know $P$ or $r$!

Policy Optimization Setting

Meta-Algorithm: Policy Optimization

• Initialize $\theta_0$
• For $i=0,1,...$:
  • Rollout policy
  • Estimate $\nabla J(\theta_i)$ as $g_i$ using rollouts
  • Update $\theta_{i+1} = \theta_i + \alpha g_i$

Policy Optimization

• In today's lecture, we review four ways to construct the estimates $g_i$ such that $$\mathbb E[g_i| \theta_i] = \nabla J(\theta_i)$$
• We will have
  • two estimates that use trajectories $\tau$
  • two estimates that also use Q/Value functions
• We consider infinite length trajectories $\tau$ without worrying about computational feasibility
  • note that we could use a similar trick as in Lecture 12-13 to ensure finite sampling size/time

Policy Optimization Overview

Agenda

1. Recap

2. Policy Optimization

3. with Trajectories

4. with Value

$0$

$1$

stay: $1$

switch: $1$

stay: $p_1$

switch: $1-p_2$

stay: $1-p_1$

switch: $p_2$

Example

• Parametrized policy: $\pi_\theta(0)=$stay, $\pi_\theta(\mathsf{stay}|1) = \theta^{(1)}$ and $\pi_\theta(\mathsf{switch}|1) = \theta^{(2)}$.

• Initialize $\theta^{(1)}_0=\theta^{(2)}_0=1/2$

  • try perturbation in favor of "switch", then in favor of "stay"

  • update in direction of policy which receives more cumulative reward

reward: $+1$ if $s=0$ and $-\frac{1}{2}$ if $a=$ switch

Policy Gradient (REINFORCE)

Algorithm: REINFORCE

• Given $\alpha$. Initialize $\theta_0$
• For $i=0,1,...$:
  • Rollout policy $\pi_{\theta_i}$ and observe trajectory $\tau$
  • Estimate $g_i = \sum_{t=0}^\infty \nabla_\theta[\log \pi_\theta(a_t|s_t)]_{\theta=\theta_i}R(\tau)$
  • Update $\theta_{i+1} = \theta_i + \alpha g_i$

$0$

$1$

stay: $1$

switch: $1$

stay: $p_1$

switch: $1-p_2$

stay: $1-p_1$

switch: $p_2$

Example

• Parametrized policy: $\pi_\theta(0)=$stay, $\pi_\theta(\mathsf{stay}|1) = \theta^{(1)}$ and $\pi_\theta(\mathsf{switch}|1) = \theta^{(2)}$.
• Compute the score PollEV
  • $\nabla_\theta \log \pi_\theta(a|s)=\begin{bmatrix} 1/\theta^{(1)} \cdot \mathbb 1\{a=\mathsf{stay}\} \\ 1/\theta^{(2)}  \cdot 1\{a=\mathsf{switch}\}\end{bmatrix}$

• Initialize $\theta^{(1)}_0=\theta^{(2)}_0=1/2$

  • rollout, then sum score over trajectory $$g_0 \propto \begin{bmatrix}  \text{\# times } s=1,a=\mathsf{stay} \\ \text{\# times } s=1,a=\mathsf{switch} \end{bmatrix}$$

• Direction of update depends on empirical action frequency, size depends on $R(\tau)$

reward: $+1$ if $s=0$ and $-\frac{1}{2}$ if $a=$ switch

Agenda

1. Recap

2. Policy Optimization

3. with Trajectories

4. with Value

• So far, methods depend on entire trajectory rollout
• This leads to high variance estimates
• Incorporating (Q) Value function can reduce variance
• In practice, can only use estimates of Q/Value
  • results in bias (Lecture 15)
  • we ignore this issue today

Motivation: PG with Value

Policy Gradient with Advantage

Algorithm: Idealized Actor Critic with Advantage

• Same as previous slide, except estimation step
  • Estimate $g_i = \frac{1}{1-\gamma} \nabla_\theta[\log \pi_\theta(a|s)]_{\theta=\theta_i}A^{\pi_{\theta_i}}(s,a)$

• Claim: $\mathbb E_{a\sim \pi_\theta(s)}\left[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)\right] = 0$
• General principle: subtracting any action-independent "baseline" does not affect expected value
• Proof of claim:
  • $\mathbb E_{a\sim \pi_\theta(s)}\left[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)\right]$
    • $=\sum_{a\in\mathcal A} \pi_\theta(a|s)\left[\nabla_\theta \log \pi_\theta(a|s) \cdot b(s)\right]$
    • $=\sum_{a\in\mathcal A} \pi_\theta(a|s) \frac{\nabla_\theta \pi_\theta(a|s)}{\pi_\theta(a|s)} \cdot b(s)$
    • $=\nabla_\theta  \sum_{a\in\mathcal A}\pi_\theta(a|s) \cdot b(s)$
    • $=\nabla_\theta b(s) = 0$

PG with "Baselines"

Recap

• PSet due Wed Fri
• PA due Fri

• PG with rollouts: random search and REINFORCE
• PG with value: Actor-Critic and baselines

• Next lecture: Trust Regions