CS 4/5789: Introduction to Reinforcement Learning

Lecture 24: Inverse Reinforcement Learning

Prof. Sarah Dean

MW 2:45-4pm
255 Olin Hall

Reminders

• Homework
  • 5789 Paper Reviews due weekly on Mondays
  • PSet 8 (final one!) due Monday
    • Next week: midterm corrections
  • PA 4 due next Wednesday (May 3)
• Final exam is Saturday 5/13 at 2pm
  • Length: 2 hours

Agenda

1. Recap: Imitation Learning

2. Inverse RL Setting

3. Entropy & Constrained Optimization

4. MaxEnt IRL & Soft VI

Recap: Imitation Learning

Expert Demonstrations

Supervised ML Algorithm

Policy \(\pi\)

ex - SVM, Gaussian Process, Kernel Ridge Regression, Deep Networks

maps states to actions

Recap: BC/DAgger

Supervised Learning

Policy

Recap: BC vs. DAgger

Supervised learning guarantee

\(\mathbb E_{s\sim d^{\pi^*}_\mu}[\mathbf 1\{\widehat \pi(s) - \pi^*(s)\}]\leq \epsilon\)

Online learning guarantee

\(\mathbb E_{s\sim d^{\pi^t}_\mu}[\mathbf 1\{ \pi^t(s) - \pi^*(s)\}]\leq \epsilon\)

Performance Guarantee

\(V_\mu^{\pi^*} - V_\mu^{\widehat \pi} \leq \frac{2\epsilon}{(1-\gamma)^2}\)

Performance Guarantee

\(V_\mu^{\pi^*} - V_\mu^{\pi^t} \leq \frac{\max_{s,a}|A^{\pi^*}(s,a)|}{1-\gamma}\epsilon\)

Agenda

1. Recap: Imitation Learning

2. Inverse RL Setting

3. Entropy & Constrained Optimization

4. MaxEnt IRL & Soft VI

Inverse RL: Motivation

• Like previous lecture, learning from expert demonstrations
• Unlike previous lectures, focus on learning reward rather than learning policy
• Motivations:
  • Scientific inquiry - modelling human or animal behavior
  • Imitation via reward function
    • reward may be more succinct & transferrable
  • Modelling other agents in multi-agent setting (adversarial or cooperative)

Inverse RL: Setting

• Finite horizon MDP $$\mathcal M =  \{\mathcal S,\mathcal A, P, r, H, \mu\} $$
• Transition \(P\) known but reward \(r\) unknown and signal \(r_t\) unobserved
• State/state-action distributions:
  • \(d_{\mu}^{\pi}(s) = \frac{1}{H}\sum_{t=0}^{H-1} d_{\mu,t}^\pi(s)\) average prob. of state
  • \(d_{\mu}^{\pi}(s,a) = \frac{1}{H}\sum_{t=0}^{H-1} d_{\mu,t}^\pi(s,a)\) average prob. of state, action
• Observe traces of expert policy $$\mathcal D=\{s_i^\star, a_i^\star\} \sim d_{\mu}^{\pi_\star}$$
• Assumption: reward function is linear in known features: $$r(s,a) = \theta_\star^\top  \varphi(s,a)$$

Example: Driving

\(s=\)

\(a\in \{\) north, south, east, west \(\}\)

(Kitani et al., 2012)

Consistency principle

• Consistent rewards: find  \(r\)  such that for all \(\pi\) $$ \mathbb E_{d^{\pi^*}_\mu}[r(s,a)] \geq \mathbb E_{d^{\pi}_\mu}[r(s,a)]$$

• Consistent policies: find \(\pi\) such that $$\mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] = \mathbb E_{d^{\pi}_\mu}[\varphi (s,a)]$$

• Consistent policies imply consistent rewards

• Estimate the feature distribution from expert data: $$\mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] \approx \frac{1}{N} \sum_{i=1}^N \varphi(s_i^*, a_i^*) $$

Ambiguity problem

• Our goal is to find a policy \(\pi\) that matches the feature distribution $$\mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] = \mathbb E_{d^{\pi}_\mu}[\varphi (s,a)]$$

• Depending on expert policy and the definition of features, more than one policy may satisfy this

• How should we choose between policies?

• Occam's razor: choose the policy which encodes the least information

Agenda

1. Recap: Imitation Learning

2. Inverse RL Setting

3. Entropy & Constrained Optimization

4. MaxEnt IRL & Soft VI

Entropy

• The mathematical concept entropy allows us to quantify "amount of information"

• Definition: The entropy of a distribution \(P\in\Delta(\mathcal X) \) is $$\mathsf{Ent}(P) = \mathbb E_P[-\log P(x)] = -\sum_{x\in\mathcal X} P(x) \log P(x)$$

• Fact: since \(P(x)\in[0,1]\), entropy is non-negative

• Entropy is a measure of uncertainty

  • Distributions with higher entropy are more uncertain, while lower entropy means less uncertainty

Entropy Examples

• Consider distributions over \(A\) actions \(\mathcal A = \{1,...,A\}\)

• Uniform distribution \(U(a) = \frac{1}{A}\) PollEV

  • \(\mathsf{Ent}(P) =  -\sum_{a\in\mathcal A} \frac{1}{A} \log \frac{1}{A} = \log A\)

• Deterministic action with \(a=1\) with probability \(1\)

  • \(\mathsf{Ent}(P) = -\sum_{a=1} 1\cdot \log 1 =0\)

• Uniform distribution over \(\{1,2\}\), \(U_1(a) = \frac{1}{2}\mathbf 1\{a\in\{1,2\}\}\)

  • \(\mathsf{Ent}(P) =  -\sum_{a=1}^2 \frac{1}{2} \log\frac{1}{2}=\log 2\)

Max Entropy Principle

• Principle: among distributions consistent with constraints (i.e. observed data, mean, variance) choose the one with the most uncertainty, i.e. the highest entropy
  
  
   

• Specifically, we define

  • \(\mathsf{Ent}(\pi) = \mathbb E_{s\sim d^{\pi}_\mu}[\mathsf{Ent}(\pi(s)) ] \)

    • \(= \mathbb E_{s\sim d^{\pi}_\mu}[\mathbb E_{a\sim\pi(s)}[-\log \pi(a|s)]] \)

    • \(= \mathbb E_{s,a\sim d^{\pi}_\mu}[\mathbb -\log \pi(a|s)] \)

Max Entropy Principle

• Principle: among distributions consistent with constraints (i.e. observed data, mean, variance) choose the one with the most uncertainty, i.e. the highest entropy
  
   

maximize    \(\mathbb E_{ d^{\pi}_\mu}[-\log \pi(a|s)]\)

s.t.    \(\mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] - \mathbb E_{d^{\pi}_\mu}[\varphi (s,a)] = 0\)

Constrained Optimization

• General form for constrained optimization $$x^* =\arg \max~~f(x)~~\text{s.t.}~~g(x)=0$$
• An equivalent formulation (Lagrange) $$ x^* =\arg \max_x \min_w ~~f(x)+w\cdot g(x)$$
• Optimality condition: $$\nabla_x[f(x)+w\cdot g(x) ] = \nabla_w[f(x)+w\cdot g(x) ] = 0 $$
• Exercise: \(f(x) = x_1+x_2\) and \(g(x) = x_1^2+x_2^2\)

• Weighted objective of reward and entropy: $$\max_{P\in\Delta(\mathcal A)} \mathbb E_{a\sim P}[\mu_a] + \tau \mathsf{Ent}(P)$$
• The condition that \(P\in\Delta(\mathcal A)\) is actually a constraint $$\max_{P(1),\dots, P(A)} \sum_{a\in\mathcal A} P(a)\mu_a + \mathsf{Ent}(P)\quad\text{s.t.}\quad \sum_a P(a) -1=0$$
• Lagrange formulation and optimality condition:
  • \(\nabla_{P}[\sum_{a=1}^K P(a)\widehat\mu_a - \tau P(a)\log P(a) + w\left(\sum_{a=1}^K P(a)-1\right)]=0\)
    • \(P_\star(a) = \exp(\mu_a/\tau)\cdot \exp(w/\tau-1) \)
  • \(\nabla_w[w\left(\sum_{a=1}^K P(a)-1\right)]=0\)
    • \( \exp(w/\tau-1) = \frac{1}{\sum_{a=1}^A \exp(\mu_a /\tau)}\)

Example: High Entropy Action

Constrained Optimization

• General form for constrained optimization $$x^* =\arg \max~~f(x)~~\text{s.t.}~~g(x)=0$$
• An equivalent formulation (Lagrange) $$ x^* =\arg \max_x \min_w ~~f(x)+w\cdot g(x)$$
• Iterative algorithm for constrained optimization
  • For \(t=0,\dots,T-1\):
    1. Best response: \(x_t = \arg\max f(x) + w_t g(x)\)
    2. Gradient descent: \(w_{t+1} = w_t - \eta g(x)\)
  • Return \(\bar x = \frac{1}{T} \sum_{t=0}^{T-1} x_t\)

Constrained Optimization

• General form for constrained optimization $$x^* =\arg \max~~f(x)~~\text{s.t.}~~g_1(x)=g_2(x)=...=g_d(x)=0$$
• An equivalent formulation (Lagrange) $$ x^* =\arg \max_x \min_{w_1,\dots,w_d} ~~f(x)+\Sigma_{i=1}^dw_i g_i(x)$$

Constrained Optimization

• General form for constrained optimization $$x^* =\arg \max~~f(x)~~\text{s.t.}~~g(x)=0$$
• An equivalent formulation (Lagrange) $$ x^* =\arg \max_x \min_{w} ~~f(x)+w^\top g(x)$$
• Iterative algorithm for constrained optimization
  • For \(t=0,\dots,T-1\):
    1. Best response: \(x_t = \arg\max f(x) + w_t^\top g(x)\)
    2. Gradient descent: \(w_{t+1} = w_t - \eta g(x)\)
  • Return \(\bar x = \frac{1}{T} \sum_{t=0}^{T-1} x_t\)

Agenda

1. Recap: Imitation Learning

2. Inverse RL Setting

3. Entropy & Constrained Optimization

4. MaxEnt IRL & Soft VI

Max-Ent IRL

• Use iterative constrained optimization algorithm on our maximum entropy IRL formulation: $$\max \mathbb E_{ d^{\pi}_\mu}[-\log \pi(a|s)]\quad\text{s.t.}\quad \mathbb E_{d^{\pi}_\mu}[\varphi (s,a)]- \mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] = 0$$

Soft-VI

• Best response step resembles MDP with reward \(r(s,a) = w_k^\top \varphi (s,a)\) as well as a policy-dependent  term  $$\arg\max_\pi \mathbb E_{ d^{\pi}_\mu}[w_k^\top \varphi (s,a) -\log \pi(a|s)] $$
• We can solve this with DP!
  • Initialize \(V_H^*(s) = 0\), then for \(h=H-1,\dots 0\):
    • \(Q_h^*(s,a) = r(s,a) + \mathbb E_{s'\sim P}[V_{h+1}(s')]\)
    • \(\pi_h^*(a|s)  = \arg\max_{\pi(s)\in\Delta(\mathcal A)} \mathbb E_{a\sim \pi(s)}[Q_h^*(s,a)] + \mathsf{Ent}(\pi(s))\)
      • \(\pi_h^*(a|s) \propto \exp(Q^*_h(s,a))\)
    • \(V_h^*(s) = \log\left(\sum_{a\in\mathcal A} \exp(Q^*_h(s,a) \right)\)

Max-Ent IRL

• For \(k=0,\dots,K-1\):
  1. \(\pi^k = \mathsf{SoftVI}(w_k^\top \varphi)\)
  2. \(w_{k+1} = w_k - \eta  (\mathbb E_{d^{\pi^k}_\mu}[\varphi (s,a)]-\mathbb E_{d^{\pi^*}_\mu}[\varphi (s,a)] )\)
• Return \(\bar \pi = \mathsf{Unif}(\pi^0,\dots \pi^{K-1})\)

• Input: reward function \(r\). Initialize \(V_H^*(s) = 0\)
• For \(h=H-1,\dots 0\):
  1. \(Q_h^*(s,a) = r(s,a) + \mathbb E_{s'\sim P}[V_{h+1}(s')]\)
  2. \(\pi_h^*(a|s) \propto \exp(Q^*_h(s,a))\)
  3. \(V_h^*(s) = \log\left(\sum_{a\in\mathcal A} \exp(Q^*_h(s,a) \right)\)

Soft-VI

Modelling Behavior with IRL

Recap: IL vs. IRL

Recap

• PSet due Monday

• Inverse RL
• Maximum Entropy Principle
• Constrained Optimization
• Soft VI

• Next week: case study and societal implications