## CS 4/5789: Introduction to Reinforcement Learning

### Lecture 13

Prof. Sarah Dean

MW 2:45-4pm
110 Hollister Hall

## Agenda

0. Announcements & Recap

1. Q Function Approximation

4. Derivative-Free Optimization

## Announcements

HW2 released next Monday

5789 Paper Review Assignment (weekly pace suggested)

OH cancelled today, instead Thursday 10:30-11:30am

Learning Theory Mentorship Workshop

Application due March 10: https://let-all.com/alt22.html

Prelim Tuesday 3/22 at 7:30-9pm in Phillips 101

Closed-book, definition/equation sheet for reference will be provided

Focus: mainly Unit 1 (known models) but many lectures in Unit 2 revisit important key concepts

Study Materials: Lecture Notes 1-15, HW0&1

Lecture on 3/21 will be a review

## Recap

Meta-Algorithm for Policy Iteration in Unknown MDP

• Sample $$h_1=h$$ w.p. $$\propto \gamma^h$$: $$(s_{h_1}, a_{h_1}) = (s_i,a_i) \sim d^\pi_{\mu_0}$$
• Sample $$h_2=h$$ w.p. $$\propto \gamma^h$$: $$y_i = \sum_{t=h_1}^{h_1+h_2} r_t$$

Supervision with Rollout (MC):

$$\mathbb{E}[y_i] = Q^\pi(s_i, a_i)$$

$$\widehat Q$$ via ERM on $$\{(s_i, a_i, y_i)\}_{1}^N$$

Rollout:

$$s_t$$

$$a_t\sim \pi(s_t)$$

$$r_t\sim r(s_t, a_t)$$

$$s_{t+1}\sim P(s_t, a_t)$$

$$a_{t+1}\sim \pi(s_{t+1})$$

...

## Recap

• $$y_t =r_t + \gamma \max_a \widehat Q(s_{t+1},a)$$

Supervision with Bellman Exp (TD):

If $$\widehat Q = Q^\pi$$ then $$\mathbb{E}[y_t] = Q^\pi(s_t, a_t)$$

One step:

$$s_t$$

$$a_t\sim \pi(s_t)$$

$$r_t\sim r(s_t, a_t)$$

$$s_{t+1}\sim P(s_t, a_t)$$

$$a_{t+1}\sim \pi(s_{t+1})$$

Supervision with Bellman Opt (TD):

• $$y_t =r_t + \gamma \widehat Q(s_{t+1},a_{t+1})$$

If $$\widehat Q = Q^*$$ then $$\mathbb{E}[y_t] = Q^*(s_t, a_t)$$

SARSA and Q-learning are simple tabular algorithms

## Agenda

0. Announcements & Recap

1. Q Function Approximation