Announcements

• Questions about waitlist/enrollment?
  • https://www.cs.cornell.edu/courseinfo/enrollment
• Homework released next week
  • Problem Set 1 due 1 week later
  • Programming Assignment 1 due 2 weeks later

Agenda

1. Recap: Markov Decision Process

2. Imitation Learning

3. Trajectories and Distributions

Markov Decision Process (MDP)

action $a_t$

state $s_t$

$\sim \pi(s_t)$

reward

$r_t\sim r(s_t, a_t)$

$s_{t+1}\sim P(s_t, a_t)$

1. Agent observes state of environment
2. Agent takes action
   • depending on state according to policy
3. Environment state updates according to transition function

state
$s_t$

Markovian Assumption: Conditioned on $s_t,a_t$, the reward $r_t$ and next state $s_{t+1}$ are independent of the past.

When state transition are stochastic, we will write either:

• $s'\sim P(s,a)$ as a random variable
• $P(s'|s,a)$ as a number between $0$ and $1$

Notation: Stochastic Maps

Notation: Stochastic Maps

$0$

$1$

• From state $0$, action always works
• From state $1$, staying works with probability $p_1$ and switching with probability $p_2$

• $P(0,$stay$)=\mathbf{1}_{0}$
• $P(1,$stay$)=\mathsf{Bernoulli}(p_1)$

• $P(0\mid 0,$stay$)=$
• $P(1\mid 0,$stay$)=$
• $P(0\mid 1,$stay$)=$
• $P(1\mid 1,$stay$)=$

• $P(0\mid 0,$switch$)=$
• $P(1\mid 0,$switch$)=$
• $P(0\mid 1,$switch$)=$
• $P(1\mid 1,$switch$)=$

• $1$
• $0$
• $1-p_1$
• $p_1$

• $0$
• $1$
• $p_2$
• $1-p_2$

• $P(0,$switch$)=\mathbf{1}_{1}$
• $P(1,$switch$)=\mathsf{Bernoulli}(1-p_2)$

Infinite Horizon Discounted MDP

• $\mathcal{S}, \mathcal{A}$ state and action space
• $r$ reward function: stochastic map from (state, action) to scalar reward
• $P$ transition function: stochastic map from current state and action to next state
• $\gamma$ discount factor between $0$ and $1$

$\mathcal M = \{\mathcal{S}, \mathcal{A}, r, P, \gamma\}$

Infinite Horizon Discounted MDP

maximize   $\displaystyle \mathbb E\left[\sum_{i=1}^\infty \gamma^t r(s_t, a_t)\right]$

s.t.   $s_{t+1}\sim P(s_t, a_t), ~~a_t\sim \pi(s_t)$

$\pi$

We will find policies using optimization & learning

Agenda

1. Recap: Markov Decision Process

2. Imitation Learning

3. Trajectories and Distributions

Helicopter Acrobatics (Stanford)

LittleDog Robot (LAIRLab at CMU)

An Autonomous Land Vehicle In A Neural Network [Pomerleau, NIPS ‘88]

Imitation Learning

Behavioral Cloning

Dataset from expert policy $\pi_\star$: $$\{(s_i, a_i)\}_{i=1}^N \sim \mathcal D_\star$$

Behavioral Cloning

1. Policy class $\Pi$: usually parametrized by some $w\in\mathbb R^d$, e.g. weights of deep network, SVM, etc
2. Loss function $\ell(\cdot,\cdot)$: quantify accuracy
3. Optimization Algorithm: gradient descent, interior point methods, sklearn, torch

minimize   $\sum_{i=1}^N \ell(\pi(s_i), a_i)$

$\pi\in\Pi$

Supervised learning with empirical risk minimization (ERM)

Behavioral Cloning

minimize   $\sum_{i=1}^N \ell(\pi(s_i), a_i)$

$\pi\in\Pi$

Supervised learning with empirical risk minimization (ERM)

Ex: Learning to Drive

Policy $\pi$

Input: Camera Image

Output: Steering Angle

Ex: Learning to Drive

Ex: Learning? to Drive

Ex: Learning? to Drive

An Autonomous Land Vehicle In A Neural Network [Pomerleau, NIPS ‘88]

“If the network is not presented with sufficient variability in its training exemplars to cover the conditions it is likely to encounter...[it] will perform poorly”

Agenda

1. Recap: Markov Decision Process

2. Imitation Learning

3. Trajectories and Distributions

Trajectory

• When we deploy or roll out a policy, we will observe a sequence of states and actions
  • define the initial state distribution $\mu_0\in\Delta(\mathcal S)$
• This sequence is called a trajectory: $$\tau = (s_0, a_0, s_1, a_1, ... )$$

• observe $s_0\sim \mu_0$
• play $a_0\sim \pi(s_0)$
• observe $s_1\sim P(s_0, a_0)$
• play $a_1\sim \pi(s_1)$
• ...

Trajectory Example

$0$

$1$

stay: $1$

switch: $1$

stay: $p_1$

switch: $1-p_2$

stay: $1-p_1$

switch: $p_2$

Probability of a Trajectory

State Distribution

State Distribution Transition

Transition Matrix

$1$

$1-p_1$

$p_1$

$0$

$1$

Example: $\pi(s)=$stay and $\mu_0$ is each state with probability $1/2$.

State Evolution Example

Recap

1. Markov Decision Process

2. Imitation Learning

3. Trajectories and Distributions