CS 4/5789: Introduction to Reinforcement Learning

Prof. Sarah Dean

MW 2:45-4pm
110 Hollister Hall

Agenda

0. Announcements & Recap

1. Supervision via Rollouts

2. Approximate Policy Iteration

3. Performance Different Lemma

4. Conservative Policy Iteration

HW0 grades and solutions released

HW1 due Monday 3/7

5789 Paper Review Assignment (weekly pace suggested)

Prelim Tuesday 3/22 at 7:30pm in Phillips 101

Office hours after lecture M (110 Hollister) and W (416A Gates)

Wednesday OH prioritize 1-1 questions/concerns over HW

Model-based RL (Meta-Algorithm)

Tabular MBRL

Using Simulation Lemma we saw that $\epsilon$ optimal policy requires $N\gtrsim \frac{S^2 A}{\epsilon^2}$

MBRL for LQR

Sample: $s_i\sim \mathcal N(0,\sigma^2 I)$ and $a_i\sim \mathcal N(0,\sigma^2I)$
Estimate: by least-squares linear regression
$\displaystyle \widehat A,\widehat B = \arg\min_{A,B} \sum_{i=1}^N\|s_i' - As_i-Ba_i\|_2^2$
Design: LQR dynamic programming

We will not cover the argument in detail, but $\epsilon$ optimal policy requires $N\gtrsim \frac{n_s+n_a}{\epsilon^2}$

Policy Evaluation uses knowledge of transition function $P$ and reward function $r$

initialize pi[0]
for t=1,2,...
    Q[t] = PolicyEvaluation(pi[t]) # Evaluation
    pi[t+1] = argmax_a(Q[t](:,a)) # Improvement

Can we estimate the Q function from sampled data?

By Sarah Dean

asst prof in CS at Cornell