Model-Based RL:
LQR, iLQR
Curricula
- Linear Quadratic Regulator
- iterative LQR (iLQR)
- The case of unknown dynamics
LQR - motivation
We are aimed to optimize
J=∑t=0Tr(st,at)s.t.st+1=f(st,at)
J = \sum_{t=0}^{T} r(s_t, a_t) \;\;\; s.t. \; \; s_{t+1} = f(s_t, a_t)
for now assume deterministic dynamics
What if r and f are known and are "nice":
- we can find maxar(s,a) analytically
- the composition of r and f is also "nice"
Then we can express the optimal aT action as a function of sT:
aT=argmaxr(sT,⋅)=πT(sT)
a_T = \arg\max r(s_T, \cdot) = \pi_T(s_T)
Now we can write the value of the last time-step
V(sT)=r(sT,πT(sT))
V(s_T) = r(s_T, \pi_T(s_T))
LQR - motivation
Going one step backward:
Apply recursively until t=0 where the state s0 is known!
Then we can go forward:
V(sT−1)=Q(sT−1,πT−1(sT))
V(s_{T-1}) = Q(s_{T-1}, \pi_{T-1}(s_T))
Q(sT−1,aT−1)=r(sT−1,aT−1)+V(sT)
Q(s_{T-1}, a_{T-1}) = r(s_{T-1}, a_{T-1}) + V(s_T)
Q(sT−1,aT−1)=r(sT−1,aT−1)+V(f(sT−1,aT−1))
Q(s_{T-1}, a_{T-1}) = r(s_{T-1}, a_{T-1}) + V(f(s_{T-1}, a_{T-1}))
Using the rule of "nice" compositions:
πT−1(sT−1)=argmaxQ(sT−1,⋅)
\pi_{T-1}(s_{T-1}) = \arg\max Q (s_{T-1}, \cdot)
st+1=f(st,at)
s_{t+1} = f(s_t, a_t)
at=πt(st)
a_t = \pi_t(s_t)
Linear dynamics, Quadratic rewards
ft(st,at)=Ft[stat]+ft
f_t(s_t, a_t) = F_t \begin{bmatrix} s_t \\ a_t \end{bmatrix} + f_t
rt(st,at)=21[stat]TRt[stat]+[stat]Trt
r_t(s_t, a_t) =\frac{1}{2}\begin{bmatrix} s_t \\ a_t \end{bmatrix}^T R_t \begin{bmatrix} s_t \\ a_t \end{bmatrix} + \begin{bmatrix} s_t \\ a_t \end{bmatrix} ^T r_t
Dynamics
Rewards
LQR - backward pass
Q(sT,aT)=21[sTaT]TRT[sTaT]+[sTaT]TrT
Q(s_T, a_T) = \frac{1}{2}\begin{bmatrix} s_T \\ a_T \end{bmatrix}^T R_T \begin{bmatrix} s_T \\ a_T \end{bmatrix} + \begin{bmatrix} s_T \\ a_T \end{bmatrix} ^T r_T
=[sTaT]TQT[sTaT]+[sTaT]TqT
= \begin{bmatrix} s_T \\ a_T \end{bmatrix}^T Q_T \begin{bmatrix} s_T \\ a_T \end{bmatrix} + \begin{bmatrix} s_T \\ a_T \end{bmatrix} ^T q_T
∇Q(sT,aT)=QTassT+QTaaaT+qTa=0
\nabla Q(s_T, a_T) = Q_{{T}_{as}} s_T + Q_{T_{aa}} a_T + q_{T_a} = 0
πT(sT)=−QTaa−1(QTassT+qTa)
\pi_T(s_T) = - Q_{T_{aa}}^{-1} ( Q_{T_{as}} s_T + q_{T_{a}})
The last Q:
Equate the gradient to zero:
Get the optimal last time-step behavior
LQR - backward pass
(check it yourself slide)
V(sT)=Q(sT,πT(sT))=sTTVTsT+sTTvT
V(s_T) = Q(s_T, \pi_T(s_T)) = s_T^T V_T s_T + s_T^T v_T
πt(st)=−Qtaa−1(Qtasst+qta)
\pi_t(s_t) = - Q_{t_{aa}}^{-1} ( Q_{t_{as}} s_t + q_{t_{a}})
The last time-step value is also quadratic in sT
The Q function at T−1 is again quadratic in both sT−1 and aT−1:
Thus you can get an analytical formula for policy at each time-step:
Q(sT−1,aT−1)=21[sT−1aT−1]TRT−1[sT−1aT−1]+[sT−1aT−1]TrT−1+V(f(sT−1,aT−1))
Q(s_{T-1}, a_{T-1}) = \frac{1}{2}\begin{bmatrix} s_{T-1} \\ a_{T-1} \end{bmatrix}^T R_{T-1} \begin{bmatrix} s_{T-1} \\ a_{T-1} \end{bmatrix} + \begin{bmatrix} s_{T-1} \\ a_{T-1} \end{bmatrix} ^T r_{T-1} + V(f(s_{T-1}, a_{T-1}))
LQR - algorithm
Given: s0,Ft,ft,Rt,rt for all t
- Calculate Qt,qt for all t going backward
- Calculate at=−Qtaa−1(Qtasst+qta) for all t going forward
- Calculate st+1=ft(st,at)
LQR - stochastic dynamics
Closed-loop
Planning
Tutorial
LQR - algorithm (stochastic dynamics)
Given: s0,Ft,ft,Rt,rt for all t
- Calculate Qt,qt for all t going backward
- Calculate a0=−Q0aa−1(Q0ass0+q0a) only for t=0
- Apply a0 in the real environment
- Observe s1∼p(s1∣s0,a0)
- Start from the beginning!
Curricula
- Linear Quadratic Regulator
- iterative LQR (iLQR)
- The case of unknown dynamics
The dynamics is not L
the rewards are not Q :(
Tailor's expansion:
f(st,at)≈f(s^t,a^t)+∇f(s^t,a^t)[st−s^tat−a^t]
r(st,at)≈r(s^t,a^t)+∇r(s^t,a^t)[st−s^tat−a^t]+21[st−s^tat−a^t]T∇2r(s^t,a^t)[st−s^tat−a^t]
Simplify a bit....
f~t(st,at)= Ft[stat]+ft
r~t(st,at)=21[st at]TRt[st at]+[stat]Trt
Iterative LQR (iLQR)
Algorithm:
Initialize (s^0,a^0,s^1,…) somehow
- Ft=∇f(s^t,a^t)
- ft=…
- Rt=∇2r(s^t,a^t)
- rt=…
- (a0,s1,a1,…)=LQR()
- (s^0,a^0,s^1,a^1,…)←(s0,a0,s1,a1,…)
- Go to 1.
Curricula
- Linear Quadratic Regulator
- iterative LQR (iLQR)
- The case of unknown dynamics
Model-Based RL: LQR, iLQR
MB-RL: LQR, iLQR, DDP
By cydoroga
MB-RL: LQR, iLQR, DDP
- 557
