CS 4/5789: Introduction to Reinforcement Learning
Lecture 13
Prof. Sarah Dean
MW 2:45-4pm
110 Hollister Hall
Agenda
0. Announcements & Recap
1. Q Function Approximation
2. Optimization & Gradient Descent
3. Stochastic Gradient Descent
4. Derivative-Free Optimization
Announcements
HW2 released next Monday
5789 Paper Review Assignment (weekly pace suggested)
OH cancelled today, instead Thursday 10:30-11:30am
Learning Theory Mentorship Workshop
Application due March 10: https://let-all.com/alt22.html
Prelim Tuesday 3/22 at 7:30-9pm in Phillips 101
Closed-book, definition/equation sheet for reference will be provided
Focus: mainly Unit 1 (known models) but many lectures in Unit 2 revisit important key concepts
Study Materials: Lecture Notes 1-15, HW0&1
Lecture on 3/21 will be a review
Prelim Exam
Recap
Meta-Algorithm for Policy Iteration in Unknown MDP
- Sample \(h_1=h\) w.p. \(\propto \gamma^h\): \((s_{h_1}, a_{h_1}) = (s_i,a_i) \sim d^\pi_{\mu_0}\)
- Sample \(h_2=h\) w.p. \(\propto \gamma^h\): \(y_i = \sum_{t=h_1}^{h_1+h_2} r_t\)
Supervision with Rollout (MC):
\(\mathbb{E}[y_i] = Q^\pi(s_i, a_i)\)
\(\widehat Q\) via ERM on \(\{(s_i, a_i, y_i)\}_{1}^N\)
Rollout:
\(s_t\)
\(a_t\sim \pi(s_t)\)
\(r_t\sim r(s_t, a_t)\)
\(s_{t+1}\sim P(s_t, a_t)\)
\(a_{t+1}\sim \pi(s_{t+1})\)
...
Recap
- \(y_t =r_t + \gamma \max_a \widehat Q(s_{t+1},a) \)
Supervision with Bellman Exp (TD):
If \(\widehat Q = Q^\pi\) then \(\mathbb{E}[y_t] = Q^\pi(s_t, a_t)\)
One step:
\(s_t\)
\(a_t\sim \pi(s_t)\)
\(r_t\sim r(s_t, a_t)\)
\(s_{t+1}\sim P(s_t, a_t)\)
\(a_{t+1}\sim \pi(s_{t+1})\)
Supervision with Bellman Opt (TD):
- \(y_t =r_t + \gamma \widehat Q(s_{t+1},a_{t+1}) \)
If \(\widehat Q = Q^*\) then \(\mathbb{E}[y_t] = Q^*(s_t, a_t)\)
SARSA and Q-learning are simple tabular algorithms
Agenda
0. Announcements & Recap
1. Q Function Approximation
2. Optimization & Gradient Descent
3. Stochastic Gradient Descent
4. Derivative-Free Optimization