CS 4/5789: Introduction to Reinforcement Learning

Lecture 12

Prof. Sarah Dean

MW 2:45-4pm
110 Hollister Hall

Agenda

 

0. Announcements & Recap

1. Performance Difference Lemma

2. Supervision via Bellman Eq

3. Supervision via Bellman Opt

4. Function Approximation

Announcements

 

HW1 due tonight, HW2 released next Monday

 

5789 Paper Review Assignment (weekly pace suggested)

 

Prelim Tuesday 3/22 at 7:30pm in Phillips 101

 

OH cancelled Wednesday, instead Thursday 10:30-11:30am

Learning Theory Mentorship Workshop

with the Conference on Algorithmic Learning Theory (ALT)

Virtual, March 14-15, 2022

 

Application due March 10: https://let-all.com/alt22.html

Recap

Meta-Algorithm for Policy Iteration in Unknown MDP

Approximate Policy Iteration

Greedy Improvement:

\(\pi^{t+1}(s) = \arg\max_a \widehat Q^{t}(s, a)\)

Could oscillate!

Conservative Policy Iteration

Incremental Improvement:

\(\pi'(s) = \arg\max_a \widehat Q^{t}(s, a)\)

\(\pi^{t+1}(a\mid s) = (1-\alpha)\pi^{t}(a\mid s) + \alpha \pi'(s\mid a)\)

Recap

Meta-Algorithm for Policy Iteration in Unknown MDP

  • Sample \(h_1=h\) w.p. \(\propto \gamma^h\): \((s_{h_1}, a_{h_1}) = (s_i,a_i) \sim d^\pi_{\mu_0}\)
  • Sample \(h_2=h\) w.p. \(\propto \gamma^h\): \(y_i = \sum_{t=h_1}^{h_1+h_2} r_t\)

Supervision with Rollout (MC):

\(\mathbb{E}[y_i] = Q^\pi(s_i, a_i)\)

\(\widehat Q\) via ERM on \(\{(s_i, a_i, y_i)\}_{1}^N\)

Rollout:

\(s_t\)

\(a_t\sim \pi(s_t)\)

\(r_t\sim r(s_t, a_t)\)

\(s_{t+1}\sim P(s_t, a_t)\)

\(a_{t+1}\sim \pi(s_{t+1})\)

...