CS 4/5789: Introduction to Reinforcement Learning

Lecture 3: Policy Gradient

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

  • Homework
    • PSet 1 due Wednesday
    • Project 1 due 9/11
  • Enrollment: join waitlist, come to lecture, do assignments, there should be plenty of space

Agenda

 

1. Setting & Estimation

2. Policy Gradient

3. Example

 

Given data, how do we update the policy to improve the expected reward?

\begin{align*} Var_{\theta'}(w_ir_i)&=\mathbb E_{\theta'}[w_i^2 r_i^2] - J(\theta)^2\\ &= \mathbb E_{\theta}[w_i r_i^2] - J(\theta)^2\\ &= \mathbb E_{\theta}[w_i r_i^2] - \mathbb E_{\theta}[r_i^2] + \mathbb E_{\theta}[ r_i^2] - J(\theta)^2\\ &= \mathbb E_{\theta}[(w_i-1) r_i^2] + Var_{\theta}(r_i) \end{align*}

Data \(\{(x_i,a_i,r_i)\}\) collected with \(\pi_{\theta'}\), weights \(w_i = \frac{\pi_\theta(a_i|x_i)}{\pi_{\theta'}(a_i|x_i)}\)

Shorthand: \(\mathbb E_{x\sim\mu,a\sim\pi_\theta(x),r\sim R(x,a)}[\cdot ] = \mathbb E_\theta[\cdot ]\) and same for \(Var\)

Explain each step of the following:

Think/Pair/Share