CS 4/5789: Introduction to Reinforcement Learning

Lecture 23: Interactive Imitation Learning

Prof. Sarah Dean

MW 2:45-4pm
255 Olin Hall

Reminders

• Homework
  • 5789 Paper Reviews due weekly on Mondays
  • PSet 7 due tonight, PSet 8 (final one!) released tonight
    • Next week: midterm corrections
  • PA 4 due next Wednesday (May 3)
• Final exam is Saturday 5/13 at 2pm

Agenda

1. Recap: Exploration

2. Imitation Learning

3. DAgger

4. Online Learning

• Unit 2: Constructing labels for supervised learning, updating policy using learned quantities
• Unit 3: Design policy to both explore (collect useful data) and exploit (high reward)

Recap: Exploration

action \(a_t\)

state \(s_t\)

reward \(r_t\)

policy

data \((s_t,a_t,r_t)\)

policy \(\pi\)

transitions \(P,f\)

experience

unknown

Recap: UCB

UCB-type Algorithms

• Multi-armed bandits $$\arg\max_a \widehat \mu_{a,t} + \sqrt{C/N_{a,t}}$$
• Linear contextual bandits $$\arg\max_{a} \hat \theta_{a,t}^\top x_t + \sqrt{x_t A_{a,t}^{-1} x_t}$$
• Markov decision process (tabular) $$\arg\max_a \hat r_i(s,a)+H\sqrt{\frac{\alpha}{N_i(s,a)}}+\mathbb E_{s'\sim \hat P_i(s,a)}[\hat V^i_{t+1}(s')]$$

• Exploration becomes more difficult outside of tabular setting where a comprehensive search is possible
• Using expert data can sidestep exploration problem

Motivation for Imitation

Agenda

1. Recap: Exploration

2. Imitation Learning

3. DAgger

4. Online Learning

Imitation Learning

Helicopter Acrobatics (Stanford)

Behavioral Cloning

Behavioral Cloning

• Dataset from expert policy \(\pi_\star\): \(\{(s_i, a_i)\}_{i=1}^N \sim d_{\mu_0}^{\pi_\star} \)
  • Recall definition of discounted state distribution
• Supervised learning with empirical risk minimization (ERM) $$\min_{\pi\in\Pi} \sum_{i=1}^N \ell(\pi(s_i), a_i) $$
• In this class, we assume that supervised learning works!
  • i.e. we successfully optimize and generalize, so that the population loss is small: \(\displaystyle \mathbb E_{s,a\sim d_{\mu_0}^{\pi_\star}}[\ell(\pi(s), a)]\leq \epsilon\)

• We further assume that \(\ell(\pi(s), a) \geq \mathbb 1\{\pi(s)\neq a\}\)
  $$\displaystyle \mathbb E_{s\sim d_{\mu_0}^{\pi_\star}}[\mathbb 1\{\pi(s)\neq \pi_\star(s)\}]\leq \epsilon$$

Example

• Initial state \(s_0=0\) and reward \(r(s,a) = \mathbf 1\{s=1\}\)
  • Optimal policy \(\pi_\star(s) = U\)
• Discounted state distribution \(d_{0}^{\pi_\star} = \begin{bmatrix}  1-\gamma & \gamma & 0 \end{bmatrix}^\top \)
• Consider \(\hat\pi(1)=U\), \(\hat\pi(2)=D\), and $$\hat\pi(0) = \begin{cases}U & w.p. ~~1-\frac{\epsilon}{1-\gamma}\\ D & w.p. ~~\frac{\epsilon}{1-\gamma} \end{cases}$$
• PollEv What is the supervised learning error?
  • \( \mathbb E_{s\sim d_0^{\pi_\star}}\left[\mathbb E_{a\sim \hat \pi(s)}[\mathbf 1\{a\neq \pi_\star(s)\}]\right]=\epsilon\)
• Error in performance:
  • \(V^{\pi_\star}(0) = \frac{\gamma}{1-\gamma}\) vs. \(V^{\hat\pi}(0) =\frac{\gamma}{1-\gamma} - \frac{\epsilon\gamma}{(1-\gamma)^2}\)

• Assuming that SL works, how sub-optimal is \(\hat\pi\)?
  • Also assume \(r(s,a)\in[0,1]\)
• Recall Performance Difference Lemma on \(\pi^\star\) and \(\hat\pi\) $$ \mathbb E_{s\sim\mu_0} \left[ V^{\pi_\star}(s) - V^{\hat\pi}(s) \right] = \frac{1}{1-\gamma} \mathbb E_{s\sim d_{\mu_0}^{\pi\star}}\left[A^{\hat \pi}(s,\pi^\star(s))\right]$$
• The advantage of \(\pi^\star\) over \(\hat\pi\) depends on SL error
  • \(A^{\hat \pi}(s,\pi^\star(s)) \leq \frac{2}{1-\gamma}\mathbf 1 \{\pi^\star(s) \neq \hat \pi(s)\}\)
• Then the performance of BC is upper bounded: $$\mathbb E_{s\sim\mu_0} \left[ V^{\pi_\star}(s) - V^{\hat\pi}(s) \right]\leq \frac{2}{(1-\gamma)^2}\epsilon $$

BC Analysis

Agenda

1. Recap: Exploration

2. Imitation Learning

3. DAgger

4. Online Learning

query expert

learned policy

DAgger: Dataset Aggregation

Ex: Off-road driving

DAgger Setting

• Discounted Infinite Horizon MDP $$\mathcal M = \{\mathcal S, \mathcal A, P, r, \gamma\} $$
• \(P\) unknown and \(r\) unknown and maybe unobserved
• Access to an expert who knows \(\pi_\star\) and who we can query at any state \(s\) during training

DAgger Algorithm

Agenda

1. Recap: Exploration

2. Imitation Learning

3. DAgger

4. Online Learning

• Due to the active data collection, need a new framework beyond "supervised learning"
• Online learning is a general setting which captures the idea of learning from data over time

Online Learning

• Measure performance of online learning with average regret $$\frac{1}{T} R(T) = \frac{1}{T} \left(\sum_{i=1}^T \mathcal R_i(f_i) - \min_f \sum_{i=1}^T \mathcal R_i(f)\right) $$
• Define regret as the incurred risk compared with the best function in hindsight
• This baseline represents the usual offline supervised learning approach $$\min_f \frac{1}{T}\sum_{i=1}^T \mathcal R_i(f) = \min_f \mathbb E_{x,y\sim \bar{\mathcal D}} [\ell(f_i(x), y)]$$ where we define \(\bar{\mathcal D} = \frac{1}{T}\sum_{i=1}^T \mathcal D_i\)

Regret

• How should learner choose \(\theta_i\)?
• A good option is to solves a sequence of (regularized) supervised learning problems
  • \(d(f)\) regularizes the predictions, e.g. \(\|\theta\|_2^2\)

• Dataset aggregation: \( \sum_{k=1}^i \mathcal R_k(f) = \displaystyle \mathbb E_{x, y \sim \bar \mathcal D_i}[ \ell(f(x, y))]\)

Follow the Regularized Leader

• Theorem: For convex loss and strongly convex regularizer, $$ \max_{\mathcal R_1,...,\mathcal R_T} \frac{1}{T} \left( \sum_{i=1}^T \mathcal R_i(f_i) - \min_f \sum_{i=1}^T \mathcal R_i(f) \right) \leq O\left(1/\sqrt{T}\right)$$ i.e. regret is bounded for any sequence of \(\mathcal D_i\).
• Corollary: For DAgger with a noiseless expert, $$ \min_{1\leq i\leq T} \mathbb E_{s\sim d^{\pi^i}_{\mu_0} } [\mathbf 1\{\pi^i(s)\neq \pi^\star(s)\}] \leq O(1/\sqrt{T}) =: \epsilon $$
• Proof Sketch
  • DAgger as FTRL: \(f_i=\pi^i\), \((x,y)=(s, \pi^\star(s))\), and \(\mathcal D_i = d_{\mu_0}^{\pi^i}\)
  • Minimum policy error is upper bounded by average regret (using that loss upper bounds indicator)

DAgger Analysis

• Corollary: For DAgger with a noiseless expert, $$ \min_{1\leq i\leq T} \mathbb E_{s\sim d^{\pi^i}_{\mu_0} } [\mathbf 1\{\pi^i(s)\neq \pi^\star(s)\}] \leq O(1/\sqrt{T}) =: \epsilon $$
• We can show an upper bound, starting with PDL $$ \mathbb E_{s\sim\mu_0} \left[ V^{\pi_\star}(s) - V^{\pi^i}(s) \right] = -\frac{1}{1-\gamma} \mathbb E_{s\sim d_{\mu_0}^{\pi^i}}\left[A^{ \pi^\star}(s,\pi^i(s))\right]$$
• The advantage of \(\hat\pi\) over \(\pi^\star\) (no \(\gamma\) dependence)
  • \(A^{ \pi^\star}(s,\pi^i(s)) \geq -\mathbf 1\{\pi^i(s)\neq \pi^\star(s)\} \cdot \max_{s,a} |A^{ \pi^\star}(s,\pi^i(s))| \)
• Then the performance of DAgger is upper bounded: $$\mathbb E_{s\sim\mu_0} \left[ V^{\pi_\star}(s) - V^{\pi^i}(s) \right]\leq \frac{\epsilon}{1-\gamma}\max_{s,a} |A^{ \pi^\star}(s,\pi^i(s))|  $$

DAgger Analysis

Summary: BC vs. DAgger

Recap

• PSet due tonight

• Pitfalls of BC
• DAgger Algorithm
• Online learning

• Next lecture: Inverse RL