• Policy $\pi$ chooses an action based on the current state so $a_t=a$ with probability $\pi(a|s_t)$
  • Shorthand for deterministic policy: $a_t=\pi(s_t)$

• Probability of trajectory $\tau =(s_0, a_0, s_1, ... s_t, a_t)$ $$\mathbb{P}_{\mu_0}^\pi (\tau) = \mu_0(s_0)\pi(a_0 \mid s_0) \cdot \displaystyle\prod_{i=1}^t {P}(s_i \mid s_{i-1}, a_{i-1}) \pi(a_i \mid s_i)$$
• Probability of $(s, a)$ at $t$ $$\mathbb{P}^\pi_t(s, a ; \mu_0) = \displaystyle\sum_{\substack{s_{0:t-1}\\ a_{0:t-1}}} \mathbb{P}^\pi_{\mu_0} (s_{0:t-1}, a_{0:t-1}, s_t, a_t \mid s_t = s, a_t = a)$$
• Discounted "steady-state" distribution $$d^\pi_{\mu_0}(s, a) = (1 - \gamma) \displaystyle\sum_{t=0}^\infty \gamma^t \mathbb{P}^\pi_t(s, a; \mu_0)$$

Food for thought:

• How do these distributions change under two different policies $\pi$ and $\pi'$? (HW2)
• How to write the distributions $\mathbb{P}^\pi_t$ and $d^\pi_{\mu_0}$ over the state only?

• Evaluate policy by cumulative reward
  • $V^\pi(s) = \mathbb E[\sum_{t=0}^\infty \gamma^t r_t | s_0=s]$
  • $Q^\pi(s, a) = \mathbb E[\sum_{t=0}^\infty \gamma^t r_t | s_0=s, a_0=a]$
• For finite horizon, for $t=0,...H-1$,
  • $V_t^\pi(s) = \mathbb E[\sum_{k=t}^{H-1} r_k | s_t=s]$
  • $Q_t^\pi(s, a) = \mathbb E[\sum_{k=t}^{H-1} r_t | s_t=s, a_t=a]$

Recursive Bellman Expectation Equation:

• Discounted Infinite Horizon
  •  $V^{\pi}(s) = \mathbb{E}_{a \sim \pi(s)} \left[ r(s, a) + \gamma \mathbb{E}_{s' \sim P( s, a)} [V^\pi(s')] \right]$
  • $Q^{\pi}(s, a) = r(s, a) + \gamma \mathbb{E}_{s' \sim P( s, a)} \left[ V^\pi(s') \right]$
• Finite Horizon,  for $t=0,\dots H-1$,
  • $V^{\pi}_t(s) = \mathbb{E}_{a \sim\pi_t(s) } \left[ r(s, a) + \mathbb{E}_{s' \sim P(s, a)} [V^\pi_{t+1}(s')] \right]$
  • $Q^{\pi}_t(s) =  r(s, a) + \mathbb{E}_{s' \sim P(s, a)} [V^\pi_{t+1}(s')]$

• Recursive computation: $V^{\pi} = R^{\pi} + \gamma P^{\pi} V^\pi$
  • Exact Policy Evaluation: $V^{\pi} = (I- \gamma P^{\pi} )^{-1}R^{\pi}$
  • Iterative Policy Evaluation: $V^{\pi}_{t+1} = R^{\pi} + \gamma P^{\pi} V^\pi_t$
• Backwards-Iterative computation in finite horizon:
  • Initialize $V^{\pi}_H = 0$
  • For $t=H-1, H-2, ... 0$
    • $V^{\pi}_t = R^{\pi} +P^{\pi} V^\pi_{t+1}$

• An optimal policy $\pi^*$ is one where $V^{\pi^*}(s) \geq V^{\pi}(s)$ for all $s$ and policies $\pi$
• Equivalent condition: Bellman Optimality
  • $V^*(s) = \max_{a\in\mathcal A} \left[r(s, a) + \gamma \mathbb{E}_{s' \sim P(s, a)} \left[V^*(s') \right]\right]$
  • $Q^*(s, a) = r(s, a) + \gamma \mathbb{E}_{s' \sim P(s, a)} \left[ \max_{a'\in\mathcal A} Q^*(s', a') \right]$
• Optimal policy $\pi^*(s) = \argmax_{a\in \mathcal A} Q^*(s, a)$

• Finite horizon, for $t=0,\dots H-1$,
  • $V_t^*(s) = \max_{a\in\mathcal A} \left[r(s, a) + \mathbb{E}_{s' \sim P(s, a)} \left[V_{t+1}^*(s') \right]\right]$
  • $Q_t^*(s, a) = r(s, a) + \mathbb{E}_{s' \sim P(s, a)} \left[ \max_{a'\in\mathcal A} Q_{t+1}^*(s', a') \right]$
• Optimal policy $\pi_t^*(s) = \argmax_{a\in \mathcal A} Q_t^*(s, a)$
• Can directly solve with Dynamic Programming
  • Iterate backwards in time from $V^*_{H}=0$

• Infinite horizon: algorithms for recursion in the Bellman Optimality equation
• Value Iteration
  • Initialize $Q_0$. For $t=0,1,\dots$,
    • $Q^{t+1}(s,a) =r(s, a) + \gamma\mathbb{E}_{s' \sim P(s, a)} \left[ \max_{a'\in\mathcal A} Q^{t}(s', a') \right]$
• Policy Iteration
  • Initialize $\pi_0$. For $t=0,1,\dots$,
    • $Q^{t}=$ PolicyEval($\pi^t$)
    • $\pi^{t+1}(s) = \argmax_{a\in\mathcal A} Q^{t}(s,a)$

• Value Iteration
  • Fixed point iteration (like Iterative Policy Iteration) from Bellman Q Optimality
  • Contraction in Q: $\|Q^{t+1} - Q^*\|_\infty \leq \gamma \|Q^t - Q^*\|_\infty$
• Policy Iteration
  • Monotone Improvement: $Q^{t+1}(s,a) \geq Q^{t}(s,a)$
  • Contraction in V: $\|V^{t+1} - V^*\|_\infty \leq \gamma \|V^t - V^*\|_\infty$

• Linear Dynamics: $$s_{t+1} = A s_t + Ba_t + w_t,\quad w_t\sim \mathcal N(0,\sigma^2 I)$$
• Unrolled dynamics $$s_{t} = A^ts_0 + \sum_{k=0}^{t-1} A^k (Ba_{t-k-1} + w_{t-k-1})$$
• Stability of uncontrolled $s_{t+1}=As_t$:
  • stable if $\rho(A)< 1$
  • unstable if $\rho(A) > 1$
  • marginally unstable if $\rho(A) = 1$

Finite Horizon LQR: Application of Dynamic Programming

• Initialize $V^{\pi}_H(s) = 0$
• For $t=H-1, H-2, ... 0$
  • $Q^{*}_t(s,a) = c(s,a) +\mathbb E_{s'\sim P(s,a)}[ V^*_{t+1}(s')]$
  • $\pi^*(s) = \argmin_{a\in\mathcal A} Q^{*}_t(s,a)$
  • $V^*_t =  Q^{*}_t(s,\pi^*(s))$

1. Add and subtract: $$\|f(x) - g(y)\| \leq  \|f(x)-f(y)\| +\|f(y)-g(y)\|$$
2. Contractions (induction) $$\|x_{t+1}\|\leq \gamma \|x_t\| \implies \|x_t\|\leq \gamma^t\|x_0\|$$
3. Additive induction $$\|x_{t+1}\| \leq \delta_t + \|x_t\| \implies \|x_t\|\leq \sum_{k=0}^{t-1} \delta_k + \|x_0\|$$
4. Basic Inequalities (HW0) $$|\mathbb E[f(x)] - \mathbb E[g(x)]| \leq \mathbb E[|f(x)-g(x)|]$$ $$|\max f(x) - \max g(x)| \leq \max |f(x)-g(x)|$$ $$\mathbb E[f(x)] \leq \max f(x)$$

1. Move on if stuck!
2. Write explanations and show steps for partial credit
3. Multipart questions: can be done mostly independently
   • ex: 1) show $\|x_{t+1}\|\leq \gamma \|x_t\|$
           2) give a bound on $\|x_t\|$ in terms of $\|x_0\|$