Reminders

• Homework this week
  • Programming Assignment 1 due TONIGHT
  • Next PSet and PA released tonight
    • PSet due next Wednesday
    • PA due in 2 weeks
• My office hours:
  • Tuesdays 10:30-11:30am in Gates 416A
  • Wednesdays 4-4:50pm in Olin 255 (right after lecture)

Agenda

1. Recap

2. Linear Control

3. Linear Quadratic Regulator

Recap: Optimal Control

• Continuous $\mathcal S = \mathbb R^{n_s}$ and $\mathcal A = \mathbb R^{n_a}$
• Cost to be minimized $c=(c_0,\dots, c_{H-1}, c_H)$
• Deterministic transitions described by dynamics function $$s_{t+1} = f(s_t, a_t)$$
• Finite horizon $H$

$\mathcal M = \{\mathcal{S}, \mathcal{A}, c, f, H\}$

Recap: Linear Dynamics

• The dynamics function $f$ has a linear form $$s_{t+1} = As_t + Ba_t$$
• $A$ describes the evolution of the state when there is no action (internal dynamics) $$s_{t+1}=As_t$$

Recap: Trajectories and Stability

$\mathbb C$

$\mathcal R(\lambda)$

$\mathcal I(\lambda)$

Trajectory is determined by the eigenstructure of $A$

Recap: Trajectories and Stability

$\mathbb C$

$\mathcal R(\lambda)$

$\mathcal I(\lambda)$

Trajectory is determined by the eigenstructure of $A$

$\lambda = \alpha \pm i \beta$

$\mathbb C$

$\mathcal R(\lambda)$

$\mathcal I(\lambda)$

Trajectory is determined by the eigenstructure of $A$

$\lambda = \alpha \pm i \beta$

Recap: Trajectories and Stability

$\mathbb C$

$\mathcal R(\lambda)$

$\mathcal I(\lambda)$

Trajectory is determined by the eigenstructure of $A$

Recap: Trajectories and Stability

$\lambda_1 = \lambda_2=\lambda$

$\mathbb C$

$\mathcal R(\lambda)$

$\mathcal I(\lambda)$

Trajectory is determined by the eigenstructure of $A$

• depends on if $A$ is diagonalizable

$0<\lambda<1$

Recap: Trajectories and Stability

$\lambda_1 = \lambda_2=\lambda$

Recap: Stability Theorem

Agenda

1. Recap

2. Linear Control

3. Linear Quadratic Regulator

Controlled Trajectories

• Full dynamics depend on actions $$s_{t+1} = As_t+Ba_t$$

• The trajectories can be written as (PSet 3) $$s_{t} = A^t s_0 + \sum_{k=0}^{t-1}A^k Ba_{t-k-1}$$
• The internal dynamics $A$ determines the long term effects of actions

Example

• Setting: hovering UAV over a target $$s_{t+1} = \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}s_t + \begin{bmatrix}0\\ 1\end{bmatrix}a_t$$
• Initially at rest, then one rightward thrust followed by one leftward thrust $$a_0=1,\quad a_{t_0}=-1,\quad a_k=0~~k\notin\{0,t_0\}$$

$a_t$

• $s_{t} = \displaystyle \begin{bmatrix}1 & t \\ 0 & 1\end{bmatrix}\begin{bmatrix}\mathsf{pos}_0  \\ 0 \end{bmatrix}+ \sum_{k=0}^{t-1} \begin{bmatrix}1 & k\\ 0 & 1\end{bmatrix} \begin{bmatrix}0\\ 1\end{bmatrix}a_{t-k-1}$
• $s_{t} = \displaystyle \begin{bmatrix}\mathsf{pos}_0  \\ 0 \end{bmatrix}+  \begin{bmatrix}1 & t-1\\ 0 & 1\end{bmatrix} \begin{bmatrix}0\\ 1\end{bmatrix}- \begin{bmatrix}1 & t-t_0-1\\ 0 & 1\end{bmatrix} \begin{bmatrix}0\\ 1\end{bmatrix}$
• for $t\leq t_0$, $s_{t} = \displaystyle \begin{bmatrix}\mathsf{pos}_0+ t-1 \\ 1 \end{bmatrix}$ and for $t\geq t_0$, $s_{t} = \displaystyle \begin{bmatrix}\mathsf{pos}_0+ t_0 \\ 0 \end{bmatrix}$

Linear Policy

• Linear policy defined by $a_t=Ks_t$: $$s_{t+1} = As_t+BKs_t = (A+BK)s_t$$

• The trajectories can be written as $$s_{t} = (A+BK)^t s_0$$
• The internal dynamics $A$ are modified depending on $B$ and $K$

Example

• Setting: hovering UAV over a target $$s_{t+1} = \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}s_t + \begin{bmatrix}0\\ 1\end{bmatrix}a_t$$
• Thrust according to distance from target $a_t = -(\mathsf{pos}_t- x)$

$a_t$

Example

• Setting: hovering UAV over a target $$s_{t+1} = \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}s_t + \begin{bmatrix}0\\ 1\end{bmatrix}a_t$$
• Thrust according to distance from target $a_t = -(\mathsf{pos}_t+\mathsf{vel}_t- x)$

$a_t$

Agenda

1. Recap

2. Linear Control

3. Linear Quadratic Regulator

Linear Quadratic Regulator

Special case of optimal control problem with

• Quadratic cost $$c_t(s,a) = s^\top Qs+ a^\top Ra,\quad c_H = s^\top Qs$$
• Linear dynamics $$s_{t+1} = As_t+ Ba_t$$

Example

• Setting: hovering UAV over a target
• Action: thrust right/left
• State is $s_t = \begin{bmatrix}\mathsf{position}_t - x\\ \mathsf{velocity}_t\end{bmatrix}$
• $c_t(s_t, a_t) = (\mathsf{position}_t-x)^2+\lambda a_t^2$
• $f(s_t, a_t) = \begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix}s_t + \begin{bmatrix}0\\ 1\end{bmatrix}a_t$

$a_t$

Example

• Setting: hovering UAV over a target
• Action: thrust right/left
• State: distance from target, velocity
• Consider $H=1$

$a_t$

DP for Optimal Control

Reformulating for optimal control, our general purpose dynamic programming algorithm is:

• Initialize $V^\star_H(s) = c_H(s)$
• For $t=H-1, H-2, ..., 0$:
  • $Q_t^\star(s,a) = c(s,a)+\mathbb E_{s'=f(s,a)}[V^\star_{t+1}(s')]$
  • $\pi_t^\star(s) = \arg\min_a Q_t^\star(s,a)$
  • $V^\star_{t}(s)=Q_t^\star(s,\pi_t^\star(s) )$
• Return $\pi^\star = (\pi^\star_0,\dots ,\pi^\star_{H-1})$

LQR via DP

• $V_H^\star(s) = s^\top Q s$
• $t=H-1$: $\quad \min_{a} s^\top Q s+a^\top Ra+ (As+Ba)^\top Q (As+Ba)$
  • $\quad \min_{a} s^\top (Q+A^\top QA) s+a^\top (R+B^\top QB) a+2s^\top A^\top Q Ba$
• General minimization: $\arg\min_a c + a^\top M a + 2m^\top a$
  • $2Ma_\star + 2m = 0 \implies a_\star = -M^{-1} m$
    • $\pi_{H-1}^\star(s)=-(R+B^\top QB)^{-1}B^\top QAs$
  • minimum is $c-m^\top M^{-1} m$
    • $V_{H-1}^\star(s) = s^\top (Q+A^\top QA - A^\top QB(R+B^\top QB)^{-1}B^\top QA) s$

DP: $V_t^\star (s) = \min_{a} c(s, a)+V_{t+1}^\star (f(s,a))$

Example

• Setting: hovering UAV over a target
• Action: thrust right/left
• State: distance from target, velocity
• LQR$\left(\begin{bmatrix}1 & 1 \\ 0 & 1\end{bmatrix},\begin{bmatrix}0\\ 1\end{bmatrix},\begin{bmatrix}1&0\\ 0&0\end{bmatrix},\frac{1}{2}\right)$

$a_t$

LQR Extensions

• The same dynamic programming method extends in a straightforward manner when:
  1. Dynamics and costs are time varying
  2. Affine term in the dynamics, cross terms in the costs
• General form: $$c_t(s,a) = s^\top Q_t s+a^\top R_t a + s^\top M_ta + m_t$$ $$f_t(s_t,a_t) = A_ts_t + B_t a_t +c_t$$
• Many applications can be reformulated this way:
  • e.g. trajectory tracking $c_t(s,a) = \|s-\bar s_t\|_2^2 + \|a\|_2^2$ for given $\bar s_t$
  • Next lecture: general (nonlinear) dynamics and costs

Recap

• PA 1 due TONIGHT

• Linear Control
• LQR

• Next lecture: Nonlinear Control