Sarah Dean PRO
asst prof in CS at Cornell
Model Predictive Control
ML in Feedback Sys #21
Prof Sarah Dean
Reminders/etc
- Project midterm update due 11/11
- Scribing feedback to come by next week!
- Upcoming paper presentations starting next week
- Participation includes attending presentations
policy
πt:S→A
observation
st
accumulate
{(st,at,ct)}
Safe action in a dynamic world
Goal: select actions at to bring environment to low-cost states
while avoiding unsafe states
action
at
F
s
Motivation for constraints
Recap: Invariant Sets
- A set Sinv is invariant under dynamics st+1=F(st) if for all s∈Sinv, F(s)∈Sinv
- If Sinv is invariant for dynamics F, then s0∈Sinv⟹st∈Sinv for all t.
- Example: sublevel set of Lyapunov function
- {s∣V(s)≤c}
Recap: Safety
- We define safety in terms of the "safe set" Ssafe⊆S.
-
A state s is safe if s∈Ssafe.
-
A trajectory of states (s0,…,st) is safe if sk∈Ssafe for all 0≤k≤t.
-
A system st+1=F(st) is safe if some Sinv⊆Ssafe is invariant and s0∈Ssafe.
Recap: Constrained Control
at=Ktst
amin t=0∑Tst⊤Qst+at⊤Rat
s.t. st+1=Ast+Bat
st∈Ssafe, at∈Asafe
[sa]=[ΦsΦa]w
w=s00⋮0
- open loop problem is convex if costs convex, dynamics linear, and Ssafe and Asafe are convex
- closed-loop linear problem can be reformulated as convex using System Level Synthesis
Φmin[Qˉ1/2Rˉ1/2][ΦsΦa]w22
s.t. [I−ZAˉ−ZBˉ][ΦsΦa]=I
Φsw∈SsafeT, Φaw∈AsafeT
Recap: Control Barrier Function
Claim: Suppose that for all t, the policy satisfies
- C(F(st,π(st)))≤γC(st) for some 0≤γ≤1.
- Then {s∣C(s)≤0} is an invariant set.
π(st)=argamin∥a−πunc⋆(st)∥22s.t. C(F(st,a))≤ γC(st)
size of s
size of a
safety constraint
C(s)=0
Today: Receding Horizon Control
Instead of optimizing for open loop control...
a0,…,aTmin t=0∑Tc(st,at)
s.t. s0 given, st+1=F(st,at)
st∈Ssafe, at∈Asafe
- Observe state
- Optimize plan
- For t=0,1,...
- Apply planned action
- For t=0,1,…
- Observe state
- Optimize plan
- Apply first planned action
...re-optimize to close the loop
model predicts the trajectory during planning
Also called Model Predictive Control
Figure from slides by Borelli, Jones, Morari
Plan:
- minimize lap time
- subject to
- car dynamics
- staying within lane
- bounded acceleration
Example: racing
- For t=0,1,…
- Observe state
- Optimize plan
- Apply first planned action
Receding Horizon Leads to Feedback
time
Do
Plan
Do
Plan
Do
Plan
- For t=0,1,…
- Observe state
- Optimize plan
- Apply first planned action
at
F
s
st
at
a0,…,aHmin k=0∑Hc(sk,ak)
s.t. s0 given, sk+1=F(sk,ak)
sk∈Ssafe, ak∈Asafe
We can:
- replan if state st+1 does not evolve as expected (e.g. due to disturbances)
- easily incorporate time-varying dynamics, constraints (e.g. obstables), costs (e.g. new incentives)
- plan over a shorter horizon than we operate (e.g. H≤T)
- important for realtime computation!
Benefits of Replanning
π(st)=u0⋆(st)
u0,…,uHmink=0∑Hc(xk,uk)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe
Notation: distinguish real states and actions st and at from the planned optimization variables xk and uk.
[u0⋆,…,uH⋆](st)=arg
The MPC Policy
u0,…,uHmink=0∑Hc(xk,uk)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe
Notation: distinguish real states and actions st and at from the planned optimization variables xk and uk.
The MPC Policy
F
s
st
at=u0⋆(st)
Example: receding horizon LQR
Infinite Horizon LQR Problem
min T→∞limT1t=0∑Tst⊤Qst+at⊤Rats.tst+1=Ast+Bat
We know that at⋆=π⋆(st) where π⋆(s)=Ks and
- K=−(R+B⊤PB)−1B⊤QPA
- P=DARE(A,B,Q,R)
Finite LQR Problem
min k=0∑Hxk⊤Qxk+uk⊤Ruks.tx0=s,xk+1=Axk+Buk
MPC Policy at=u0⋆(st) where
u0⋆(s)=K0s and
- K0=−(R+B⊤P1B)−1B⊤QP1A
- P1= backwards DP iteration (H step)
- P=Q+A⊤PA+A⊤PB(R+B⊤PB)−1B⊤PA
- Pt=Q+A⊤Pt+1A+A⊤Pt+1B(R+B⊤Pt+1B)−1B⊤Pt+1A
Example
The state is position & velocity s=[θ,ω] with st+1=[10.11]st+[0 1]at
Goal: stay near origin and be energy efficient
- c(s,a)=s⊤[10.5]s+a2
- Safety constraint ∣θ∣≤1 and actuation limit ∣a∣≤0.5
- Linear Program (LP): Linear costs and constraints, feasible set is polyhedron. zminc⊤z s.t. Gz≤h, Az=b
- Quadratic Program (QP): Quadratic cost and linear constraints, feasible set is polyhedron. Convex if P⪰0. zminz⊤Pz+q⊤z s.t. Gz≤h, Az=b
Some types of optimization problems
Figures from slides by Goulart, Borelli
- Linear Program (LP): Linear costs and constraints, feasible set is polyhedron. zminc⊤z s.t. Gz≤h, Az=b
-
Quadratic Program (QP): Quadratic cost and linear constraints, feasible set is polyhedron. Convex if P⪰0. zminz⊤Pz+q⊤z s.t. Gz≤h, Az=b
- Nonconvex if P⋡0.
Some types of optimization problems
Figures from slides by Goulart, Borelli
- Linear Program (LP): Linear costs and constraints, feasible set is polyhedron. zminc⊤z s.t. Gz≤h, Az=b
-
Quadratic Program (QP): Quadratic cost and linear constraints, feasible set is polyhedron. Convex if P⪰0. zminz⊤Pz+q⊤z s.t. Gz≤h, Az=b
- Nonconvex if P⋡0.
- Mixed Integer Linear Program (MILP): LP with discrete constraints. zminc⊤z s.t. Gz≤h, Az=b, z∈{0,1}n
Some types of optimization problems
Figures from slides by Goulart, Borelli
- Pros
- Flexibility: any model and any objective
- Guarantees safety by definition*
- *over planning horizon
- Cons
- Computationally demanding
- How to guarantee long term stability/feasibility?
Pros and Cons of MPC
Example: infeasibility
The state is position & velocity s=[θ,ω] with st+1=[10.11]st+[0 1]at
Goal: stay near origin and be energy efficient
- Safety constraint ∣θ∣≤1 and actuation limit ∣a∣≤0.5
Recap
- Recap of safety
- invariance, constraints, barrier functions
- Receding Horizon Control
- MPC Policy and Optimization
- Feasibility Problem
References: Predictive Control by Borrelli, Bemporad, Morari
Model Predictive Control ML in Feedback Sys #21 Prof Sarah Dean
21 - Model Predictive Control - ML in Feedback Sys
By Sarah Dean
