• Policy \(\pi\) chooses an action based on the current state so \(a_t=a\) with probability \(\pi(a|s_t)\)
  • Shorthand for deterministic policy: \(a_t=\pi(s_t)\)

• Probability of trajectory \(\tau =(s_0, a_0, s_1, ... s_t, a_t)\) $$ \mathbb{P}_{\mu_0}^\pi (\tau) = \mu_0(s_0)\pi(a_0 \mid s_0) \cdot \displaystyle\prod_{i=1}^t {P}(s_i \mid s_{i-1}, a_{i-1}) \pi(a_i \mid s_i) $$
• Probability of \(s\) at \(t\) $$ \mathbb{P}^\pi_t(s ; \mu_0) = \displaystyle\sum_{\substack{s_{0:t-1}\\ a_{0:t-1}}} \mathbb{P}^\pi_{\mu_0} (s_{0:t}, a_{0:t-1} \mid s_t = s) $$

• Probability vector of \(s\) at \(t\): \(d_{\mu_0,t}^\pi(s) = \mathbb{P}^\pi_t(s ; \mu_0)  \) evolves as $$ d_{\mu_0,t+1}^\pi=P_\pi^\top d_{\mu_0,t}^\pi $$ where \(P_\pi\) at row \(s\) and column \(s'\) is \(\mathbb E_{a\sim \pi(s)}[P(s'\mid s,a)]\)
• Discounted "steady-state" distribution (PSet 2) $$ d^\pi_{\mu_0} = (1 - \gamma) \displaystyle\sum_{t=0}^\infty \gamma^t d_{\mu_0,t}^\pi$$

Food for thought:

• How are these distributions different when
  • Initial states are different
  • Policies are different (Performance Difference Lemma)
  • Transitions are different (Simulation Lemma)

• Evaluate policy by cumulative reward
  • \(V^\pi(s) = \mathbb E[\sum_{t=0}^\infty \gamma^t r(s_t,a_t) | s_0=s,P,\pi]\)
  • \(Q^\pi(s, a) = \mathbb E[\sum_{t=0}^\infty \gamma^t r(s_t,a_t) | s_0=s, a_0=a,P,\pi]\)
• For finite horizon, for \(t=0,...H-1\),
  • \(V_t^\pi(s) = \mathbb E[\sum_{k=t}^{H-1} r(s_k,a_k) | s_t=s,P,\pi]\)
  • \(Q_t^\pi(s, a) = \mathbb E[\sum_{k=t}^{H-1} r(s_k,a_k) | s_t=s, a_t=a,P,\pi]\)

Recursive Bellman Expectation Equation:

• Discounted Infinite Horizon
  •  \(V^{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} \left[ r(s, a) + \gamma \mathbb{E}_{s' \sim P( s, a)} [V^\pi(s')] \right]\)
  • \(Q^{\pi}(s, a) = r(s, a) + \gamma \mathbb{E}_{s' \sim P( s, a)} \left[ V^\pi(s') \right]\)
• Finite Horizon,  for \(t=0,\dots H-1\),
  • \(V^{\pi}_t(s) = \mathbb{E}_{a \sim\pi_t(s) } \left[ r(s, a) + \mathbb{E}_{s' \sim P(s, a)} [V^\pi_{t+1}(s')] \right]\)
  • \(Q^{\pi}_t(s) =  r(s, a) + \mathbb{E}_{s' \sim P(s, a)} [V^\pi_{t+1}(s')] \)

• Recursive computation: \(V^{\pi} = R^{\pi} + \gamma P_{\pi} V^\pi\)
  • Exact Policy Evaluation: \(V^{\pi} = (I- \gamma P_{\pi} )^{-1}R^{\pi}\)
  • Iterative Policy Evaluation: \(V^{\pi}_{i+1} = R^{\pi} + \gamma P_{\pi} V^\pi_i\)
    • Converges: fixed point contraction
• Backwards-Iterative computation in finite horizon:
  • Initialize \(V^{\pi}_H = 0\)
  • For \(t=H-1, H-2, ... 0\)
    • \(V^{\pi}_t = R^{\pi} +P_{\pi} V^\pi_{t+1}\)

• An optimal policy \(\pi^*\) is one where \(V^{\pi^*}(s) \geq V^{\pi}(s)\) for all \(s\) and policies \(\pi\)
• Equivalent condition: Bellman Optimality
  • \(V^*(s) = \max_{a\in\mathcal A} \left[r(s, a) + \gamma \mathbb{E}_{s' \sim P(s, a)} \left[V^*(s') \right]\right]\)
  • \( Q^*(s, a) = r(s, a) + \gamma \mathbb{E}_{s' \sim P(s, a)} \left[ \max_{a'\in\mathcal A} Q^*(s', a') \right]\)
• Optimal policy \(\pi^*(s) = \argmax_{a\in \mathcal A} Q^*(s, a)\)

• Finite horizon: for \(t=0,\dots H-1\),
  • \(V_t^*(s) = \max_{a\in\mathcal A} \left[r(s, a) + \mathbb{E}_{s' \sim P(s, a)} \left[V_{t+1}^*(s') \right]\right]\)
  • \(Q_t^*(s, a) = r(s, a) + \mathbb{E}_{s' \sim P(s, a)} \left[ \max_{a'\in\mathcal A} Q_{t+1}^*(s', a') \right]\)
• Optimal policy \(\pi_t^*(s) = \argmax_{a\in \mathcal A} Q_t^*(s, a)\)
• Solve exactly with Dynamic Programming
  • Iterate backwards in time from \(V^*_{H}=0\)

• Infinite horizon: algorithms for recursion in the Bellman Optimality equation
• Value Iteration
  • Initialize \(V_0\). For \(i=0,1,\dots\),
    • \(V^{i+1}(s) =\max_{a\in\mathcal A} r(s, a) + \gamma\mathbb{E}_{s' \sim P(s, a)} \left[ V^i(s') \right]\)
• Policy Iteration
  • Initialize \(\pi_0\). For \(i=0,1,\dots\),
    • \(V^{i}= \) PolicyEval(\(\pi^i\))
    • \(\pi^{i+1}(s) = \argmax_{a\in\mathcal A} r(s, a) + \gamma\mathbb{E}_{s' \sim P(s, a)} \left[ V^i(s') \right] \)

• Linear Dynamics: $$s_{t+1} = A s_t + Ba_t$$
• Unrolled dynamics (PSet 3) $$ s_{t} = A^ts_0 + \sum_{k=0}^{t-1} A^k Ba_{t-k-1}$$
• Stability of \(s_{t+1}=As_t\):
  • stable if \(\max_i |\lambda_i(A)|< 1\)
  • unstable if \(\max_i |\lambda_i(A)| > 1\)
  • marginally unstable if \(\max_i |\lambda_i(A)|= 1\)

• Finite Horizon LQR: Application of Dynamic Programming $$c(s,a) = s^\top Qs + a^\top Ra \implies \pi^\star_t(s) = Ks,~~V^\star_t(s) = s^\top P_t s$$
• Basis for approximation-based algorithms (local linearization and iLQR)
• Food for thought:
  • What is \(V^\pi\) for a non-optimal linear policy?
  • What are policy/value for infinite horizon discounted LQR? (recursive use of Bellman equations)

• What do we want to learn? \(\mathcal M = \{\mathcal S,\mathcal A,P,r,\gamma\}\)
  • Unknown transitions \(P(s'|s,a)\) or reward function \(r(s,a)\)
  • Value/Q function
    • of policy \(V^\pi(s)\) or \(Q^\pi(s,a)\)
    • optimal \(V^*(s)\) or \(Q^*(s,a)\)
  • Optimal Policy \(\pi^*(s)\)
• Given a dataset with features \(x_i\) and labels \(y_i\), model:
  • Via counting: \(\widehat f(x) = \sum_{i=1}^N y_i \mathbf 1\{x=x_i\} / \sum_{i=1}^N\mathbf 1\{x=x_i\}  \)
  • Function approx: \(\widehat f(x) = \min_{f\in\mathcal F} \frac{1}{N} \sum_{i=1}^N (f(x_i)-y_i)^2   \)
    • e.g. closed-form linear regression solution

Model-Based RL

• Features are \((s,a)\) and label is \(s'\)
• Data collection
  • Query model: uniform exploration
  • Active exploration with reward bonus
• Tabular setting: \(\widehat P\) via counting
• Simulation Lemma
  • Translate error in \(\widehat P\) vs \(P\) into difference in performance \(\widehat V\) vs \(V\)

• Features are \((s_i,a_i)\)
  • \((s_i,a_i)  = (s_{h_1}, a_{h_1}) \sim d^\pi_{\mu_0}\)
• Labels constructed as:
  • Rollout based (MC): \(y_i = \sum_{t=h_1}^{h_1+h_2} r_t\)
  • Bellman Exp based (TD): \(y_t =r_t + \gamma \widehat Q(s_{t+1},a_{t+1}) \)
  • Bellman Opt based (TD): \(y_t =r_t + \gamma \max_a \widehat Q(s_{t+1},a) \)
• On vs. off policy with importance weighting (Recall PSet)
• \(\widehat Q =\arg\min \frac{1}{N}\sum_{i=1}^N (Q(s_i,a_i)-y_i)^2\)

• Value-based RL
  For \(i=0,1...\):
  1. Rollout \(\pi^i\) and construct dataset
  2. Learn \(\widehat Q^{i+1}\) from data
  3. Update \(\pi^{i+1}\) greedy, incremental, or \(\epsilon\) greedy
• Approximate vs. Conservative Policy Iteration
  • Greedy improvement \(\rightarrow\) oscillations, incremental is more stable
• Performance Difference Lemma

• \(J(\theta)=\) expected cumulative reward under policy \(\pi_\theta\)
• Estimate \(\nabla_\theta J(\theta)\) via rollouts \(\tau\), observed reward \(R(\tau)\)
  • Random Search: \(\theta \pm \delta v\) , \(g=\frac{1}{2\delta}(R(\tau_+) - R(\tau_-))v\)
  • REINFORCE: \(g=\sum_{t=0}^\infty \nabla_\theta \log \pi_\theta(a_t|s_t) R(\tau)\)
  • Actor-Critic: \(s,a\sim d^{\pi_\theta}_{\mu_0}\) ,
    \(g=\frac{1}{1-\gamma} \nabla_\theta \log \pi_\theta(a_t|s_t) (Q^{\pi_\theta}(s,a)-b(s)) \)

• Policy Gradient Meta-Algorithm
  for \(i=0,1,...\)
  1. collect rollouts using \(\theta_i\)
  2. estimate gradient with \(g_i\)
  3. \(\theta_{i+1} = \theta_i+ \alpha g_i\)
• Trust regions and Natural PG

• Multi-Arm and Contextual Bandits: MDP with no transitions!
• Regret: $$ R(T) = \mathbb E\left[\sum_{t=1}^T r(x_t, \pi^*(x_t))-r(x_t, a_t)  \right] = \sum_{t=1}^T \mathbb E[\mu^*(x_t) - \mu_{a_t}(x_t)]$$
• Linear regret: pure random, pure greedy, \(\epsilon\) greedy (PSet)
• Sublinear regret: Explore-then-commit, UCB, LinUCB
  • \( \arg\max_a \widehat \mu_a\)   vs   \( \arg\max_a \widehat \mu_t^a + \sqrt{C/N_t^a}\)

Imitation Learning with BC

Imitation Learning with DAgger

• Inverse RL: Principle of maximum entropy
  
  
   
• Max-Ent IRL with Soft-VI (entropy regularized)
  • replace \(\max\) with softmax
• Lagrange Formulation to constrained optimization
  • \(x^* =\arg \min~~f(x)~~\text{s.t.}~~g(x)=0\)
  • \(\displaystyle x^* =\arg \min_x \max_{w} ~~f(x)+w\cdot g(x)\) equivalent
  • Iterative algorithm or solve \(\nabla [f(x) + w^\top g(x)] = 0\)

1. Add and subtract: $$ \|f(x) - g(y)\| \leq  \|f(x)-f(y)\| +\|f(y)-g(y)\| $$
2. Contractions (induction) $$ \|x_{t+1}\|\leq \gamma \|x_t\| \implies \|x_t\|\leq \gamma^t\|x_0\|$$
3. Additive induction $$ \|x_{t+1}\| \leq \delta_t + \|x_t\| \implies \|x_t\|\leq \sum_{k=0}^{t-1} \delta_k + \|x_0\|  $$
4. Basic Inequalities (PSet 1): $$|\mathbb E[f(x)] - \mathbb E[g(x)]| \leq \mathbb E[|f(x)-g(x)|] $$ $$|\max f(x) - \max g(x)| \leq \max |f(x)-g(x)| $$ $$ \mathbb E[f(x)] \leq \max f(x) $$

1. Move on if stuck!
2. Write explanations and show steps for partial credit
3. Multipart questions: can be done mostly independently
   • ex: 1) show \(\|x_{t+1}\|\leq \gamma \|x_t\|\)
           2) give a bound on \(\|x_t\|\) in terms of \(\|x_0\|\)