Sarah Dean PRO
asst prof in CS at Cornell
Prof. Sarah Dean
MW 2:45-4pm
255 Olin Hall
1. Recap: Exploration
2. Imitation Learning
3. DAgger
4. Online Learning
action \(a_t\)
state \(s_t\)
reward \(r_t\)
policy
data \((s_t,a_t,r_t)\)
policy \(\pi\)
transitions \(P,f\)
experience
unknown
UCB-type Algorithms
1. Recap: Exploration
2. Imitation Learning
3. DAgger
4. Online Learning
Expert Demonstrations
Supervised ML Algorithm
Policy \(\pi\)
ex - SVM, Gaussian Process, Kernel Ridge Regression, Deep Networks
maps states to actions
Helicopter Acrobatics (Stanford)
Supervised Learning
Policy
Dataset of expert trajectory
\((x, y)\)
...
\(\pi\)( ) =
We further assume that \(\ell(\pi(s), a) \geq \mathbb 1\{\pi(s)\neq a\}\)
$$\displaystyle \mathbb E_{s\sim d_{\mu_0}^{\pi_\star}}[\mathbb 1\{\pi(s)\neq \pi_\star(s)\}]\leq \epsilon$$
\(U\)
\(D\)
\(U\)
\(U\)
\(D\)
\(D\)
\(1\)
\(0\)
\(2\)
1. Recap: Exploration
2. Imitation Learning
3. DAgger
4. Online Learning
expert trajectory
learned policy
No training data of "recovery" behavior
query expert
learned policy
and append trajectory
retrain
Idea: interact with expert to ask what they would do
Supervised Learning
Policy
Dataset
\(\mathcal D = (x_i, y_i)_{i=1}^M\)
...
\(\pi\)( ) =
Execute
Query Expert
\(\pi^*(s_0), \pi^*(s_1),...\)
\(s_0, s_1, s_2...\)
Aggregate
\((x_i = s_i, y_i = \pi^*(s_i))\)
[Pan et al, RSS 18]
Goal: map image to command
Approach: Use Model Predictive Controller as the expert!
\(\pi(\) \()=\) steering, throttle
DAgger
1. Recap: Exploration
2. Imitation Learning
3. DAgger
4. Online Learning
Online learning
Alg: FTRL
Supervised learning guarantee
\(\mathbb E_{s\sim d^{\pi^*}_\mu}[\mathbf 1\{\widehat \pi(s) - \pi^*(s)\}]\leq \epsilon\)
Online learning guarantee
\(\mathbb E_{s\sim d^{\pi^t}_\mu}[\mathbf 1\{ \pi^t(s) - \pi^*(s)\}]\leq \epsilon\)
Performance Guarantee
\(V_\mu^{\pi^*} - V_\mu^{\widehat \pi} \leq \frac{2\epsilon}{(1-\gamma)^2}\)
Performance Guarantee
\(V_\mu^{\pi^*} - V_\mu^{\pi^t} \leq \frac{\max_{s,a}|A^{\pi^*}(s,a)|}{1-\gamma}\epsilon\)
By Sarah Dean