CS 4/5789: Introduction to Reinforcement Learning

Lecture 4: Trust Regions

Prof. Sarah Dean

MW 1:25-2:40pm
155 Olin Hall

Reminders

  • Homework
    • PSet 1 due tonight
    • Project 1 due 9/11
  • Enrollment: join waitlist, come to lecture, do assignments, there should be plenty of space
  • Unofficial audit: email for Canvas access

Agenda

 

1. Why Gradients?

2. KL & Trust Regions

3. NPG Derivation

4. PPO Derivation

 

How should we update the policy parameters to improve the expected reward?

\begin{align*} \arg\max_{\|u\|_2\leq \epsilon} c + v^\top u \end{align*}

What is the solution to the following optimization problem?

Think/Pair/Share