Stochastic Infinite Horizon Optimal Control Problem

$$\min_{\pi} ~~\lim_{t\to\infty} \mathbb  E_w\Big[\frac{1}{T}\sum_{k=0}^{T} c(s_k, \pi(s_k)) \Big]\quad \text{s.t}\quad s_0~~\text{given},~~s_{k+1} = F(s_k, \pi(s_k),w_k)$$

$\underbrace{\qquad\qquad}_{J^\pi(s_0)}$

Bellman Optimality Equation

• $\underbrace{J^\star (s)}_{\text{value function}} = \min_{a\in\mathcal A} \underbrace{c(s, a)+\mathbb E_w[J^\star (F(s,a,w))]}_{\text{state-action function}}$
   
• Minimizing argument is $\pi^\star(s)$

Approximate Policy Iteration [KTR19]

• estimate quadratic state-action function with LSTD $$q^\top f + q^\top \phi(s_t, a_t) = c(s_t,a_t)+\mathbb E_{w_t}[q^\top \phi(s_{t+1}, \pi(s_{t+1})]$$
• update policy as $$\hat Ks = \arg\min_a \hat q^\top \phi(s, a) = \arg\min_a \begin{bmatrix}s\\ a\end{bmatrix}^\top \mathrm{mat}(q) \begin{bmatrix}s\\ a\end{bmatrix}$$
• guarantee $\|\hat K - K_\star \|_2 \leq \epsilon$ for $NT \gtrsim \frac{(m+n)^3}{\epsilon^2}$

Policy Gradient [FGKM18]

• estimate $\widehat{\nabla J}(K)$ with finite differencing
• update policy as $$K \leftarrow  K -\alpha\widehat{\nabla J}( K)$$
• guarantee $J(\hat K) - J(K_\star ) \leq\epsilon$ for $N\gtrsim poly(n,m,1/\epsilon)$

How to learn when data gradually reacts to your model [IZY22]

• Learner chooses $\theta_t$
• Population reacts as $\rho_{t} = \mathcal D(\theta_t, \rho_{t-1})$
• For fixed $\theta$, the limiting distribution $\rho_\star(\theta) = \lim_{t\to\infty} \rho_t$
• Goal for learner: minimize $\mathcal L^\star(\theta) = \mathbb E_{z\sim\rho_\star(\theta)} [\ell(z, \theta)]$
  • Similar to goal of minimizing $\lim_{T\to\infty} \frac{1}{T} \sum_{t=0}^T \mathbb E_{z\sim\rho_t} [\ell(z, \theta)]$
• Approach: sample-based gradient descent.
  • estimate parameters of $\rho_t$ and parametric model of $\partial_t \mathcal D$, use to estimate gradient of $\mathcal L^\star$

Is low cost all we want?

• In the setting of performative prediction, we may learn that homogeneous populations are easier to make predictions about
  • recall retention dynamics of Hashimoto et al
• Controlling autonomous vehicles or robots may involve avoiding obstacles or staying on the road

A trajectory of states $(s_0,\dots,s_t)$ is safe if $\mathcal s_k\in\mathcal S_\mathrm{safe}$ for all $0\leq k\leq t$.

(we can analogously define $\mathcal A_\mathrm{safe}\subseteq \mathcal A$ and require that $\mathcal a_k\in\mathcal A_\mathrm{safe}$ for all $0\leq k\leq t$)

A state $s$ is safe if $\mathcal s\in\mathcal S_\mathrm{safe}$.

Safety constraint on position $|\theta|\leq 1$

Are trajectories safe as long as $|\theta_0|<1$?

• no! Exercise: what is the necessary condition on $\omega_0$?

A system $s_{t+1}=F(s_t)$ is safe if some $\mathcal S_\mathrm{inv}\subseteq \mathcal S_{\mathrm{safe}}$ is invariant, i.e.

• for all $s\in\mathcal S_\mathrm{inv}$, $F(s)\in\mathcal S_\mathrm{inv}$

Exercise: Prove that if $\mathcal S_\mathrm{inv}$ is invariant for dynamics $F$, then $s_0\in \mathcal S_\mathrm{inv} \implies s_t\in\mathcal S_\mathrm{inv}$ for all $t$.

Example: An invariant set for
$s=[\theta,\omega]$ with $s_{t+1} = \begin{bmatrix} 0.9 & 0.1\\ & 0.9 \end{bmatrix}s_t$

Claim: if $V(s)$ is a Lyapunov function for $F$ then any sublevel set $\{V(s)\leq c\}$ is invariant.

Definition: A Lyapunov function $V:\mathcal S\to \mathbb R$ for $F$ is continuous and

• (positive definite) $V(0)=0$ and $V(0)>0$ for all $s\in\mathcal S - \{0\}$
• (decreasing) $V(F(s)) - V(s) \leq 0$ for all $s\in\mathcal S$

$\begin{bmatrix} \mathbf s\\ \mathbf a\end{bmatrix} = \begin{bmatrix} \mathbf \Phi_s\\ \mathbf \Phi_a\end{bmatrix}\mathbf w$

$\mathbf w = \begin{bmatrix}s_0\\ 0\\ \vdots \\0 \end{bmatrix}$

Claim: Suppose that for all $t$, the policy satisfies

• $C(F(s_t, \pi(s_t)))\leq \gamma C(s_t)$ for some $0\leq \gamma\leq 1$.
• Then $\{s\mid C(s)\leq 0\}$ is an invariant set.

$$\pi(s_t) = \text{find}\quad a\quad\text{s.t.}\quad  C(F(s_t, a)) \leq   \gamma C(s_t)$$

$C(F(s, a))-C(s) \leq   -(1-\gamma) C(s)$

$$a_t = \arg\min_{a\in\mathcal A_\mathrm{safe} } \|a-Ks_t\|_2 \quad \text{s.t.}\quad  C(As_t+Ba_t) \leq  \gamma C(s_t)$$

Exercise: If $C$ is a quadratic function, when is the above optimization problem feasible for some $a\in\mathbb R^m$?

Adversarial perspective is common when dealing with disturbances.

• $\mathcal S_\mathrm{inv}$ is robustly invariant if for all $s\in\mathcal S_\mathrm{inv}$ and $w\in\mathcal W$, $F(s, w)\in\mathcal S_\mathrm{inv}$
  • ex: $F(s,w) = \gamma s+w$ and $|w|\leq B$ then $|s|\leq B/(1-\gamma)$ is invariant.
• Robust constraints: $$\mathbf \Phi_s\mathbf w \in\mathcal S_\mathrm{safe}^T\quad\text{for all}\quad w\in\mathcal W$$
• Robust safety filter: $$\pi(s_t) = \text{find}\quad a\quad\text{s.t.}\quad  C(F(s_t, a, w)) \leq   \gamma C(s_t)~~\forall ~~ w\in\mathcal W$$