## CS 4/5789: Introduction to Reinforcement Learning

### Lecture 12

Prof. Sarah Dean

MW 2:45-4pm
110 Hollister Hall

## Agenda

0. Announcements & Recap

1. Performance Difference Lemma

2. Supervision via Bellman Eq

3. Supervision via Bellman Opt

4. Function Approximation

## Announcements

HW1 due tonight, HW2 released next Monday

5789 Paper Review Assignment (weekly pace suggested)

Prelim Tuesday 3/22 at 7:30pm in Phillips 101

OH cancelled Wednesday, instead Thursday 10:30-11:30am

Learning Theory Mentorship Workshop

with the Conference on Algorithmic Learning Theory (ALT)

Virtual, March 14-15, 2022

Application due March 10: https://let-all.com/alt22.html

## Recap

Meta-Algorithm for Policy Iteration in Unknown MDP

Approximate Policy Iteration

Greedy Improvement:

$$\pi^{t+1}(s) = \arg\max_a \widehat Q^{t}(s, a)$$

Could oscillate!

Conservative Policy Iteration

Incremental Improvement:

$$\pi'(s) = \arg\max_a \widehat Q^{t}(s, a)$$

$$\pi^{t+1}(a\mid s) = (1-\alpha)\pi^{t}(a\mid s) + \alpha \pi'(s\mid a)$$

## Recap

Meta-Algorithm for Policy Iteration in Unknown MDP

• Sample $$h_1=h$$ w.p. $$\propto \gamma^h$$: $$(s_{h_1}, a_{h_1}) = (s_i,a_i) \sim d^\pi_{\mu_0}$$
• Sample $$h_2=h$$ w.p. $$\propto \gamma^h$$: $$y_i = \sum_{t=h_1}^{h_1+h_2} r_t$$

Supervision with Rollout (MC):

$$\mathbb{E}[y_i] = Q^\pi(s_i, a_i)$$

$$\widehat Q$$ via ERM on $$\{(s_i, a_i, y_i)\}_{1}^N$$

Rollout:

$$s_t$$

$$a_t\sim \pi(s_t)$$

$$r_t\sim r(s_t, a_t)$$

$$s_{t+1}\sim P(s_t, a_t)$$

$$a_{t+1}\sim \pi(s_{t+1})$$

...

By Sarah Dean

Private