Reminders

• Office hours this week moved to Friday 9-10am
  • cancelled next week due to travel
• Feedback on final project proposal
• Upcoming paper presentations starting next week
• Project midterm update due 11/11

Policies via convex programming

Optimal Control on Arbitrary Horizons

• In many cases, task horizon is long or not pre-defined
• Allow $T\to\infty$
  • average cost $$\min_{\pi} ~~\mathbb E_w\Big[\lim_{t\to\infty} \frac{1}{T}\sum_{k=0}^{T} c(s_k, \pi(s_k)) \Big]$$
  • discounted average cost $$\min_{\pi} ~~\mathbb E_w\Big[\sum_{k=0}^{\infty} \gamma^k c(s_k, \pi(s_k)) \Big]$$
• Policy is stationary (no longer depends on time) $\pi:\mathcal S\to\mathcal A$

Steady State LQR

Infinite Horizon LQR Problem

$$\min_{\pi} ~~\lim_{T\to\infty}\mathbb E_w\Big[\frac{1}{T}\sum_{k=0}^{T} s_k^\top Qs_k + a_k^\top Ra_k \Big]\quad \text{s.t}\quad s_{k+1} = A s_k+ Ba_k+w_k$$

Infinite Horizon Optimal Control

Reference: Ch 1 in Dynamic Programming & Optimal Control, Vol. I by Bertsekas

LQR Example

$$s_{t+1} = \begin{bmatrix} 0.9 & 0.1\\ & 0.9 \end{bmatrix}s_t + \begin{bmatrix}0\\1\end{bmatrix}a_t + w_t$$

The state is position & velocity $s=[\theta,\omega]$, input is a force $a\in\mathbb R$.

Goal: stay near origin and be energy efficient

• $c(s,a) = s^\top \begin{bmatrix} 10 & \\ & 0.1 \end{bmatrix}s + 5a^2$

Steady State System Response

$s_{t+1} = As_{t}+Ba_{t}+w_{t}$

$a_t = K s_t$

• Cost depends on the (semi-)infinite sequence $\mathbf s = (s_0, s_1, s_2,\dots)$
• Generated by (semi-)infinite operator $\mathbf \Phi_s = (\Phi_s^0, \Phi_s^1,\dots)$ acting on disturbance sequence $\mathbf w = (w_{-1}, w_0, w_1,\dots)$
  • the operation is a convolution $s_{t} = \sum_{k=1}^{t+1} \Phi_s^{k-1}w_{t-k}$
• We represent this operation with the notation $\mathbf s = \mathbf \Phi_s\mathbf w$
• Concretely,
  • semi-infinite vectors and Toeplitz matrices
  • frequency domain

Sequences & Operators

• $\mathbf s = (s_0, s_1, s_2,\dots)$, $\mathbf w = (w_{-1}, w_0, w_1,\dots)$, and $\mathbf \Phi_s = (\Phi_s^0, \Phi_s^1,\dots)$ $$\mathbf s = \mathbf \Phi_s\mathbf w$$
• Concretely,
  • semi-infinite vectors and Toeplitz matrices $$\begin{bmatrix} s_{0}\\\vdots \\s_t\\\vdots \end{bmatrix} = \begin{bmatrix} \Phi_s^{0}\\ \Phi_s^{ 1}& \Phi_s^{0}\\ \vdots  & \ddots & \ddots \\ \Phi_s^{t} & \Phi_s^{t-1} & \dots & \Phi_s^{0} \\ \vdots & & \ddots &&\ddots \end{bmatrix} \begin{bmatrix} s_0\\w_0\\ \vdots \\w_{t-1} \\\vdots \end{bmatrix}$$
  • frequency domain

Sequences & Operators

• $\mathbf s = (s_0, s_1, s_2,\dots)$, $\mathbf w = (w_{-1}, w_0, w_1,\dots)$, and $\mathbf \Phi_s = (\Phi_s^0, \Phi_s^1,\dots)$ $$\mathbf s = \mathbf \Phi_s\mathbf w$$
• Concretely,
  • semi-infinite vectors and Toeplitz matrices
  • frequency domain
    • define time shift operator $z$ such that $$z(s_0, s_1,s_2 \dots) = (s_1, s_2,\dots)$$
    • represent $\mathbf s(z) = \sum_{t=0}^\infty z^{-t}s_t$ and $\mathbf \Phi_s(z) = \sum_{t=0}^\infty z^{-t}\Phi_s^t$
    • multiplication of polynomials: $$\mathbf \Phi_s(z) \mathbf w(z) = (\sum_{t=-1}^\infty z^{-t}w_{t})(\sum_{t=0}^\infty z^{-t}\Phi_s^t) = \sum_{t=0}^\infty z^{-t} \sum_{k=1}^{t+1} \Phi_s^{k-1} w_{t-k}$$

Sequences & Operators

LQR Example

$$s_{t+1} = \begin{bmatrix} 0.9 & 0.1\\ & 0.9 \end{bmatrix}s_t + \begin{bmatrix}0\\1\end{bmatrix}a_t + w_t$$

The state is position & velocity $s=[\theta,\omega]$, input is a force $a\in\mathbb R$.

$\pi_\star(s) \approx -\begin{bmatrix} 7.0\times 10^{-2}& 3.7\times 10^{-2}\end{bmatrix} s$

$\begin{bmatrix} \mathbf s\\ \mathbf a\end{bmatrix} = \begin{bmatrix} \mathbf \Phi_s\\ \mathbf \Phi_a\end{bmatrix}\mathbf w$

$\underset{\color{teal}\mathbf{\Phi}}{\min}$$\left\| \begin{bmatrix}Q^{1/2} &\\& R^{1/2}\end{bmatrix} \begin{bmatrix}\color{teal} \mathbf{\Phi}_s \\ \color{teal} \mathbf{\Phi}_a \end{bmatrix} \right\|_{\mathcal H_2}^2$

$\text{s.t.}~~ \begin{bmatrix} zI -  A & - B\end{bmatrix} \begin{bmatrix}\color{teal} \mathbf{\Phi}_s \\ \color{teal} \mathbf{\Phi}_a \end{bmatrix}= I$

Infinite Horizon LQR

Where we use the  norm:

$$\|\mathbf \Phi\|_{\mathcal H_2}^2 = \sum_{t=0}^\infty \|\Phi^t\|_F^2$$

Recap: LQR

• Goal: minimize quadratic cost ($Q,R$) in a system with linear dynamics ($A,B$)
• Classic approach: Dynamic programming/Bellman optimality
  • $P = \mathrm{DARE}(A,B,Q,R)$ and $K_\star = -(R+B^\top PB)^{-1}B^\top QPA$
• System level synthesis: Convex optimization

  • $\underset{\color{teal}\mathbf{\Phi}}{\min}$$\left\| \begin{bmatrix}Q^{1/2} &\\& R^{1/2}\end{bmatrix} \begin{bmatrix}\color{teal} \mathbf{\Phi}_s \\ \color{teal} \mathbf{\Phi}_a \end{bmatrix} \right\|_{\mathcal H_2}^2~~\text{s.t.}~~ \begin{bmatrix} zI -  A & - B\end{bmatrix} \begin{bmatrix}\color{teal} \mathbf{\Phi}_s \\ \color{teal} \mathbf{\Phi}_a \end{bmatrix}= I$

• Both require knowledge of dynamics and costs!

policy

$\pi_t:\mathcal S\to\mathcal A$

observation

$s_t$

accumulate

$\{(s_t, a_t, c_t)\}$

Action in an unknown dynamic world

Goal: select actions $a_t$ to bring environment to low-cost states

action

$a_{t}$

Setting: dynamics (and cost) functions are not known, but we have data $\{s_k, a_k, c_k\}_{k=0}^N$. Approaches include a focus on:

1. Model: learn dynamics/costs from data, then do policy design
   • For LQR: estimate $\hat A,\hat B,\hat Q,\hat R$ then design $\hat K$
   • "model based"
2. Bellman: learn value or state-action function
   • For LQR: estimate $\hat J$ then determine $\hat K$ as $\argmin$
   • "model free"
3. Policy: estimate gradients and update policy directly
   • For LQR: $\hat K \leftarrow \hat K -\alpha\widehat{\nabla J}(\hat K)$
   • "model free"

Data-driven Policy Design

Setting: dynamics $A,B$ are not known, but we have data $\{s_k, a_k\}_{k=0}^N$

1. Learn Model:
   • estimate $\hat A,\hat B$ via least-squares $$\hat A,\hat B = \arg\min_{A,B} \sum_{k=0}^{N-1} \|s_{k+1}-As_k-Ba_k\|_2^2$$
   • error bounds  $$\|A-\hat A\|_2\leq  \varepsilon_A,\quad \|B-\hat B\|_2\leq \varepsilon_B$$
   • system identification guarantees $\max\{\varepsilon_A, \varepsilon_B\}\lesssim \sqrt{\frac{m+n}{N}}$
2. Design Policy:
   • nominal or certainty equivalent approach uses $\hat A, \hat B$
   • robust approach uses $\hat A, \hat B, \varepsilon_A, \varepsilon_B$

Model-based LQR

The state is position & velocity $s=[\theta,\omega]$, input is a force $a\in\mathbb R$.

Goal: be energy efficient

• $c(s,a) = s^\top \begin{bmatrix} 0.01& \\ & 0.01 \end{bmatrix}s + 100a^2$

LQR Example

Robust design is worst-case

$\underset{\mathbf a=\mathbf{Ks}}{\min}$  $\underset{\|A-\widehat A\|\leq \varepsilon_A \atop \|B-\widehat B\|\leq \varepsilon_B}{\max}$ $\mathbb{E}\left[\lim_{T\to\infty} \frac{1}{T}\sum_{t=0}^T s_t^\top Q s_t + a_t^\top R a_t\right]$

s.t.  $s_{t+1} = As_t + Ba_t + w_t$

Lemma: if the system response variables satisfy

• the nominal system constraint \( \begin{bmatrix} zI -  \hat A & - \hat B\end{bmatrix} \begin{bmatrix} \hat\mathbf{\Phi}_s \\  \hat\mathbf{\Phi}_a \end{bmatrix}= I  \)
• then if the inverse exists,  \( \begin{bmatrix} zI -   A & -  B\end{bmatrix} \begin{bmatrix} \hat\mathbf{\Phi}_s \\  \hat\mathbf{\Phi}_a \end{bmatrix} (I-\mathbf \Delta)^{-1}= I \) where $\mathbf \Delta = (\underbrace{A-\hat A}_{\Delta_A})\mathbf \Phi_s + (\underbrace{B-\hat B}_{\Delta_B})\mathbf \Phi_a$

Robust synthesis with SLS

Robust synthesis with SLS

Theorem (Anderson et al., 2019): A policy designed from systems responses satisfying  \(\begin{bmatrix} zI -  \hat A & - \hat B\end{bmatrix} \begin{bmatrix} \hat\mathbf{\Phi}_s \\  \hat\mathbf{\Phi}_a \end{bmatrix}= I\) will achieve response \(\begin{bmatrix} \hat\mathbf{\Phi}_s \\  \hat\mathbf{\Phi}_a \end{bmatrix} (I-\mathbf \Delta)^{-1}\)

where $\mathbf \Delta = (\underbrace{A-\hat A}_{\Delta_A})\mathbf \Phi_s + (\underbrace{B-\hat B}_{\Delta_B})\mathbf \Phi_a$ if the inverse exists.

Informal Theorem (Suboptimality):

For \(\hat\mathbf{\Phi}\) synthesized as above and $\mathbf\Phi_\star$ the true optimal system response,

$$J(\hat{\mathbf \Phi}) -  J(\mathbf \Phi_\star) \lesssim J(\mathbf \Phi_\star)\left\|\begin{bmatrix} \varepsilon_A & \\ & \varepsilon_B\end{bmatrix} \mathbf \Phi_\star\right\|_{\mathcal H_\infty}$$

Robust synthesis with SLS

1. Learn Model:
   • estimate $\hat A,\hat B$ via least-squares, guarantee $\max\{\varepsilon_A, \varepsilon_B\}\lesssim \sqrt{\frac{m+n}{N}}$
2. Design Policy:
   • robust approach uses $\hat A, \hat B, \varepsilon_A, \varepsilon_B$

     • $J(\hat{\mathbf \Phi}) -  J(\mathbf \Phi_\star) \lesssim J(\mathbf \Phi_\star)\left\|\mathbf \Phi_\star\right\|_{\mathcal H_\infty} \sqrt{\frac{m+n}{N}}$

   • nominal or certainty equivalent approach uses $\hat A, \hat B$
     • for small enough $\varepsilon$, can show that $J(\hat{\mathbf \Phi}) -  J(\mathbf \Phi_\star) \lesssim \varepsilon^2$
     • thus faster rate, $J(\hat{\mathbf \Phi}) -  J(\mathbf \Phi_\star) \lesssim \frac{m+n}{N}$

Model-based LQR

Using an explore then commit algorithm, we have $$R(T) = R_{\text{explore}}(N) + R_{\text{commit}}(N, T)$$

• Robust: $R(T) \leq C_1 N + C_2 \frac{T}{\sqrt{N}}$
  • $N\propto T^{2/3}\implies R(T)\lesssim O(T^{2/3})$
  • stability guaranteed
• Certainty equivalent: $R(T) \leq C_1 N + C_2 \frac{T}{N}$
  • $N\propto \sqrt{T}\implies R(T)\lesssim O(\sqrt{T})$
  • only holds for $T$ large enough that estimation errors are small

Online Model-based LQR

Recap

• Steady-state controllers and infinite horizons
  • $\pi^\star(s) = Ks$
• Taxonomy of RL
  • policy, value, model
• Model-based LQR & Robustness

References: System Level Synthesis by Anderson, Doyle, Low, Matni and Ch 2-3 in Machine Learning in Feedback Systems by Sarah Dean