Reminders

• Homework
  • Grades released: regrades today-Friday
  • PSet due TONIGHT
  • PA due on 3/1 -- extension to 3/3
• My office hours:
  • Tuesdays 10:30-11:30am in Gates 416A
    • cancelled 2/28 (February break)
  • Wednesdays 4-4:50pm in Olin 255 (right after lecture)

Agenda

1. Recap: Local LQR

2. Iterative LQR

3. PID Control

4. Limitations to Control

Recap: LQR

• General form:  $f_t(s_t,a_t) = A_ts_t + B_t a_t +c_t$ and $$c_t(s,a) = s^\top Q_ts+a^\top R_ta+a^\top M_ts + q_t^\top s + r_t^\top a+ v_t$$
• General solution: $\pi^\star_t(s) = K_t s+ k_t$ where $$\{K_t,k_t\}_{t=0}^{H-1} = \mathsf{LQR}(\{A_t,B_t,c_t, Q_t, R_t, M_t, q_t, r_t, v_t\}_{t=0}^{H-1})$$

Recap: Example

• Setting: hovering UAV over a target
  • $s = [\mathsf{pos},\mathsf{vel}]$
• Action: imperfect thrust right/left
• $s_{t+1}=\begin{bmatrix}\mathsf{pos}_{t}+ \mathsf{vel}_{t} \\  \mathsf{vel}_{t} + e^{- (\mathsf{vel}_t^2+a_t^2)} a_t\end{bmatrix}$
  • $\approx \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}s_t + \begin{bmatrix}0\\ 1\end{bmatrix}a_t$ near $(0,0)$
• $c(s,a) =(1-e^{-\mathsf{pos}^2}) +\lambda a^2$
  • $\approx \mathsf{pos}^2 + \lambda a^2$ near $(0,0)$

$a_t$

Recap: Example

• Setting: hovering UAV over a target
• Action: imperfect thrust right/left
• LQR$\left(\begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix},\begin{bmatrix}1&0\\ 0&0\end{bmatrix},\frac{1}{2}\right)$

$a_t$

Recap: Example

• Setting: hovering UAV over a target
• Action: imperfect thrust right/left
• Local control $\pi_t^\star(s) = \begin{bmatrix}{ \gamma^\mathsf{pos}_t }& {\gamma_t^\mathsf{vel}} \end{bmatrix}s$

$a_t$

Agenda

1. Recap: Local LQR

2. Iterative LQR

3. PID Control

4. Limitations to Control

Approximate with Trajectory

• Rather than approximate around single point $(s_0,a_0)$
  • local approximations for trajectory $\tau=(s_t,a_t)_{t=0}^{H-1}$
• Leads to time-varying approximation of dynamics & costs
  • For each $t$, linearize $f$ around $(s_t,a_t)$: $\{A_t,B_t,c_t\}_{t=0}^{H-1}$
  • For each $t$, approx $c$ as quadratic: $\{Q_t,R_t,M_t,q_t,r_t,v_t\}_{t=0}^{H-1}$
• But what trajectory should we use?

$s_0\sim\mu_0$

iLQR

• Initialize $\bar a_0^0,\dots \bar a_{H-1}^0$ and $\bar s_0^0\sim \mu_0$
• Generate initial trajectory $\tau_0 = \{(\bar s_t^0, \bar a_t^0)\}_{t=0}^{H-1}$
  • by $\bar s^0_{t+1} =f(\bar s_t^0, \bar a_t^0)$ for $t\in[0,H]$
• For $i=0,1,\dots$:
  • $\{A_t, B_t, v_t, Q_t, R_t, q_t, r_t, c_t\}_{t=0}^{H-1}=$Approx$(f, c, \tau_i)$
  • $\{K^\star_t, k^\star_t\}_{t=0}^{H-1}=$LQR$(\{A_t, B_t, v_t, Q_t, R_t, q_t, r_t, c_t\}_{t=0}^{H-1})$
  • generate $\tau_{i+1} = \{(\bar s_t^{i+1}, \bar a_t^{i+1})\}_{t=0}^{H-1}$
    • by $\bar s_{t+1}^{i+1} = f(\bar s_{t}^{i+1},\underbrace{ K^\star_t\bar s_{t}^{i+1} + k^\star_t}_{\bar a_t^{i+1}})$ for $t\in[0,H]$

Linearize around a trajectory. What trajectory? Iterate!

Black lines: $\tau_{i-1}$, red arrows: trajectory if linearization was true, blue dashed lines: $\tau_i$

Agenda

1. Recap: Local LQR

2. Iterative LQR

3. PID Control

4. Limitations to Control

PID Control

• Type of policy which may not be optimal, but is open used in practice (especially for low-level stabilization)
• Applicable when:
  • There is an observation $o_t\in\mathbb R$ and desired setpoint $o^\star_t$
  • The action $a_t\in\mathbb R$ is "correlated" with $o_t$, i.e. positive actions tend to increase $o_t$
• Actions are determined by errors $e_t = o^\star_t - o_t$ $$a_t = K_P e_t + K_I \sum_{k=0}^t e_k + K_D (e_t-e_{t-1})$$

PID Control

• Actions are determined by errors $e_t = o^\star_t - o_t$ $$a_t = K_P e_t + K_I \sum_{k=0}^t e_k + K_D (e_t-e_{t-1})$$
• Policy depends on history of errors $e_{t},e_{t-1},\dots e_{0}$
• Tuning parameters is a heuristic process

$t$

Agenda

1. Recap: Local LQR

2. Iterative LQR

3. PID Control

4. Limitations to Control

• How good can an optimal policy be?
• Are there inherent properties of a system that limit performance?

Limitations to Control

1. Finite and Deterministic MDP with $r(s,a) = 1$ if $s=0$ and $0$ otherwise
   
   
   
   
    
2. Linear Dynamics with cost $\|s_t\|_2^2$ $$s_{t+1} = \begin{bmatrix} 2 & 0 \\ 0 & 1\end{bmatrix} s_t + \begin{bmatrix}0\\1\end{bmatrix}a_t$$

Motivating examples

Definition:

• A state $s'$ is reachable from a state $s$ if there exists a sequence of actions $a_0,\dots,a_{T-1}$ for a finite $T$ such that $$\mathbb P\{s_T=s'\mid s_0=s,a_0,\dots,a_{T-1}\}>0$$
• MDP is reachable (also controllable) if all states are reachable from any other state

Reachability

Discrete Reachability

Proof:

• Since the graph is strongly connected, there exists a directed path from $s'$ to $s$ for any $s$ and $s'$
  • Let $T$ be its length
• By construction, each edge along this path corresponds to at least one action $a_i$ and some nonzero transition probability $p_i$
• $\mathbb P\{s_T=s'\mid s_0=s,a_0,\dots,a_{T-1}\}>\prod_{i=0}^{T-1} p_i >0$

Discrete Reachability

Linear Deterministic Reachability

Theorem: The linear dynamics $s_{t+1}=As_t+Ba_t$ are controllable if the controllability grammian $\mathcal C$ is full rank. $$\mathrm{rank}\Big(\underbrace{\begin{bmatrix}B & AB & A^2 B & \dots & A^{n_s-1}B\end{bmatrix}}_{\mathcal C}\Big) = n_s$$

Linear Deterministic Reachability

Example

• Setting: hovering UAV over a target
• Action: thrust right/left
• $s_{t+1}=\begin{bmatrix}1 & 1\\  0 & 1\end{bmatrix} s_t +\begin{bmatrix}0\\  1\end{bmatrix}a_t$
• Controllability grammian $$\mathcal C = \begin{bmatrix}0&1 \\1& 1\end{bmatrix}$$

$a_t$

• Our focus is mostly optimization rather than design
  • Design includes building a system and then modelling it as an MDP
• In real applications, design is just as (if not more) important
  • If reachability is an issue, maybe we can add another actuator
  • If robustness is an issue, maybe we can tweak the cost or reward function

Optimization vs Design

Recap

• PSet due TONIGHT
• PA due after Feb break

• Iterative Nonlinear Control
• PID Control
• Reachability

• Happy Feb break!
• Next lecture: Model-Based RL