CS 4/5789: Introduction to Reinforcement Learning

Lecture 3: Policy Gradient

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

• Homework
  • PSet 1 due Wednesday
  • Project 1 due 9/11
• Enrollment: join waitlist, come to lecture, do assignments, there should be plenty of space

Agenda

1. Setting & Estimation

2. Policy Gradient

3. Example

Given data, how do we update the policy to improve the expected reward?

Data \(\{(x_i,a_i,r_i)\}\) collected with \(\pi_{\theta'}\), weights \(w_i = \frac{\pi_\theta(a_i|x_i)}{\pi_{\theta'}(a_i|x_i)}\)

Shorthand: \(\mathbb E_{x\sim\mu,a\sim\pi_\theta(x),r\sim R(x,a)}[\cdot ] = \mathbb E_\theta[\cdot ]\) and same for \(Var\)

Explain each step of the following:

Think/Pair/Share