Sarah Dean PRO
asst prof in CS at Cornell
Model Predictive Control
ML in Feedback Sys #21
Prof Sarah Dean
Reminders/etc
- Project midterm update due Friday!
- Scribing feedback to come by next week
-
Upcoming paper presentations starting next week
- RB17 Frank, Wei-Han, Minjae
- DSA+20 Jerry, Wendy, Yueying
- Participation includes attending presentations
- Relevant AI Seminar tomorrow 12:15-1:15 in Gates 122: Brandon Amos, Learning with differentiable and amortized optimization
policy
πt:S→A
observation
st
accumulate
{(st,at,ct)}
Safe action in a dynamic world
Goal: select actions at to bring environment to low-cost states
while avoiding unsafe states
action
at
F
s
Recap: Invariant Sets
- A set Sinv is invariant under dynamics st+1=F(st) if for all s∈Sinv, F(s)∈Sinv
- If Sinv is invariant for dynamics F, then s0∈Sinv⟹st∈Sinv for all t.
- Example: sublevel set of Lyapunov function
- {s∣V(s)≤c}
Recap: Receding Horizon
time
Do
Plan
Do
Plan
Do
Plan
- For t=0,1,…
- Observe state
- Optimize plan
- Apply first planned action
π(st)=u0⋆(st)
u0,…,uH−1mink=0∑H−1c(xk,uk)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe
Notation: distinguish real states and actions st and at from the planned optimization variables xk and uk.
[u0⋆,…,uH−1⋆](st)=arg
Recap: The MPC Policy
u0,…,uH−1mink=0∑H−1c(xk,uk)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe
Notation: distinguish real states and actions st and at from the planned optimization variables xk and uk.
Recap: The MPC Policy
F
s
st
at=u0⋆(st)
Example: infeasibility
The state is position & velocity s=[θ,ω] with st+1=[10.11]st+[0 1]at
Goal: stay near origin and be energy efficient
- Safety constraint ∣θ∣≤1 and actuation limit ∣a∣≤0.5
- Infeasibility = inability to guarantee safety
- also leads to loss of stability
- States that are initially feasible vs. states that remain feasible
- not different when plan is over H=T
Infeasibility Problem
- infeasible
- initially feasible
- remain feasible
- Infeasibility = inability to guarantee safety
- also leads to loss of stability
- States that are initially feasible vs. states that remain feasible
- not different when plan is over (H=T)
-
Definition: We call Sinv a control invariant set for dynamics
st+1=F(st,at) if for all s∈Sinv, there exists an a∈Asafe such that F(s,a)∈Sinv -
Definition: The region of attraction for dynamics F~(s) is the set of initial states s0 that converge to the origin.
- We consider the closed loop dynamics F~(s)=F(s,π(s))
Infeasibility Problem
u0,…,uH−1mink=0∑H−1c(xk,uk)+cH(xH)
s.t.x0=s,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH∈SH
- Let J(s;u0,…,uH−1) be the value of the objective for actions u0,…,uH−1
- Let J⋆(s)=J(s;u0⋆,…,uH−1⋆) be the optimal value.
- Assume that stage cost c(s,a) is positive definite, i.e. c(s,a)>0 for all s,a=0 and c(0,0)=0.
- Assume that 0∈Ssafe and 0∈Asafe
Terminal cost and constraints
- Receding horizon control is short sighted
- Additional terms more closely approximate infinite horizon problem
u0,…,uHmink=0∑H−1c(xk,uk)
s.t.x0=s,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH=0
- Assume that the origin is a fixed point of the uncontrolled dynamics
- i.e. F(0,0)=0
- We will prove that
- πMPC is recursively feasible
- F(s,πMPC(s)) is stable within region of attraction
Terminal cost and constraints
- Warm up: consider SH={0} and cH(0)=0
Recursive feasibility: feasible at st⟹ feasible at st+1
Proof:
- st feasible and solution to optimization problem is u0⋆,…,uH−1⋆ with corresponding states x0⋆,…,xH⋆
- After applying at=u0⋆, state moves to st+1=F(st,at)
- Notice that st+1=x1⋆
- Claim: u1⋆,…,uH−1⋆,0 is now a feasible solution.
- because xH⋆=0 and F(0,0)=0
- thus corresponding states x1⋆,…,xH⋆,0 satisfy constraints
Recursive Feasibility
u0,…,uHmink=0∑H−1c(xk,uk)s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH=0
Review: Lyapunov Stability
Definition: A Lyapunov function V:S→R for F is continuous and
- (positive definite) V(0)=0 and V(0)>0 for all s∈S−{0}
- (decreasing) V(F(s))−V(s)≤0 for all s∈S
- Optionally, (strict) V(F(s))−V(s)<0 for all s∈S−{0}
Theorem (1.2, 1.4): Suppose that F is locally Lipschitz, seq=0 is a fixed point, and V is a Lyapunov function for F,seq. Then, seq=0 is
- asymptotically stable if V is strictly decreasing
Proof:
J⋆(s) is positive definite and strictly decreasing. Therefore, the closed loop dynamics F(⋅,πMPC(⋅)) are asymptotically stable.
- Positive definite:
- if s=0, then optimal actions are 0 since F(0,0)=0 and stage cost is positive definite
- if s=0, J⋆(s)>0 since stage cost is positive definite
Stability
- Strictly decreasing: recall J⋆(st)=∑k=0H−1c(xk⋆,uk⋆)+cH(xH⋆)
- J⋆(st+1)≤ cost of feasible solution starting at x0=F(st,u0⋆)
- =J(st+1;u1⋆,…,uH−1⋆,0)=∑k=1H−1c(xk⋆,uk⋆) +c(xH⋆,0)+cH(0)
- =∑k=0H−1c(xk⋆,uk⋆) +c(xH⋆,0)+cH(0) −c(x0⋆,u0⋆)
- =J⋆(s)−c(x0⋆,u0⋆)<J⋆(s)
- J⋆(st+1)≤ cost of feasible solution starting at x0=F(st,u0⋆)
u0,…,uHmink=0∑H−1c(xk,uk)+cH(xH)
s.t.x0=s,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH∈SH
General terminal cost and constraints
Assumptions:
- The origin is uncontrolled fixed pointed F(0,0)=0
- The costs are positive definite and constraints contain 0
- The terminal set is contained in safe set and is control invariant
- The terminal cost satisfies cH(st+1)−cH(st)≤−c(st,u) for some u such that st+1=F(st,u)∈SH
Recursive feasibility: feasible at st⟹ feasible at st+1
Proof:
- st feasible and solution to optimization problem is u0⋆,…,uH−1⋆ with corresponding states x0⋆,…,xH⋆
- After applying at=u0⋆, state moves to st+1=F(st,at)
- Notice that st+1=x1⋆
- Claim: there exists a u such that u1⋆,…,uH−1⋆,u is a feasible solution.
- because xH⋆∈SH and SH is control invariant
- thus corresponding states x1⋆,…,xH⋆,F(xH⋆,u) satisfy constraints
Recursive Feasibility
Proof:
J⋆(s) is positive definite and strictly decreasing. Therefore, the closed loop dynamics F(⋅,πMPC(⋅)) are asymptotically stable.
Stability
- Positive definite: same argument as before
- Strictly decreasing: recall J⋆(st)=∑k=0H−1c(xk⋆,uk⋆)+cH(xH⋆)
- J⋆(st+1)≤ cost of feasible solution starting at x0=F(st,u0⋆)
- =J(st+1;u1⋆,…,uH−1⋆,u)
- =∑k=1H−1c(xk⋆,uk⋆) +c(xH⋆,u)+cH(F(xH⋆,u))
- =∑k=0H−1c(xk⋆,uk⋆)+cH(xH⋆)+c(xH⋆,u)+cH(F(xH⋆,u))−cH(xH⋆)−c(x0⋆,u0⋆)
- ≤ J⋆(s)+c(xH⋆,u)−c(xH⋆,u)−c(x0⋆,u0⋆)<J⋆(s)
- J⋆(st+1)≤ cost of feasible solution starting at x0=F(st,u0⋆)
Terminal cost and constraints for LQR
Based on unconstrained LQR policy where P=DARE(A,B,Q,R) K=−(B⊤PB+R)−1B⊤P
- Terminal cost as cH(s)=s⊤Ps
- Terminal set is any invariant set for closed loop s∈SH⟹(A+BK)s∈SH which also guarantees safety: SH⊆Ssafe,Ks∈Asafe∀ s∈SH
- ex: sublevel set of s⊤Ps
Constrained LQR Problem
min T→∞limT1t=0∑Tst⊤Qst+at⊤Rats.tst+1=Ast+BatGsst≤bs,Gaat≤ba
MPC Policy
min k=0∑H−1xk⊤Qxk+uk⊤Ruk+xH⊤PxHs.tx0=s, xk+1=Axk+BukGsxk≤bs,Gauk≤ba,xH∈SH
Terminal cost and constraints for LQR
This satisfies the assumptions:
- The origin is uncontrolled fixed pointed F(0,0)=0
- ✓ A⋅0+B⋅0=0
- The costs are positive definite and constraints contain 0
- ✓ Q,R,P are psd, assume bs,ba≥0
- The terminal set is contained in safe set and is control invariant
- ✓ by construction, have u=Ks guarantees invariance
- The terminal cost satisfies cH(st+1)−cH(st)≤−c(st,u) for some u such that st+1=F(st,u)∈SH
- Exercise: use the form of K and the DARE to show that ((A+BK)s)⊤P(A+BK)s−s⊤Ps=−s⊤Qs
- Recall the Bellman Optimality Equation:
- π⋆(s)=argmina∈Ac(s,a)+J⋆(F(s,a))
- For LQR, this means that
- π⋆(s)=argminas⊤Qs+a⊤Ra+(As+Ba)⊤P(As+Ba)
- π⋆(s)=argminux0⊤Qx0+u⊤Ru+x1⊤Px1 s.t. x0=s, x1=Ax0+Bu
-
This is MPC with H=1 and correct terminal cost!
Equivalence for unconstrained
Constrained LQR Problem
min T→∞limT1t=0∑Tst⊤Qst+at⊤Rats.tst+1=Ast+Bat
MPC Policy
min k=0∑H−1xk⊤Qxk+uk⊤Ruk+xH⊤PxHs.tx0=s, xk+1=Axk+Buk
u0,…,uH−1mink=0∑H−1c(xk,uk)+cH(xH)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH∈SH
- Terminal constraint not often used (instead: long horizon)
- Soft constraints
- xk+δ∈Ssafe and add penalty C∥δ∥22 to cost
- Accuracy of costs/dynamics vs. ease of optimization
- Sampling based optimization (cross entropy method)
MPC in practice
F
s
st
at=u0⋆(st)
u0,…,uH−1mink=0∑H−1c(xk,uk)+cH(xH)
s.t.x0=st,xk+1=F(xk,uk)
xk∈Ssafe, uk∈Asafe, xH∈SH
- Disturbances:
- optimize expectation, high probability, or worst-case
- Unknown dynamics/costs
- robust to uncertainty (worst case)
- learn from data
MPC extensions
F
s
st
at=u0⋆(st)
Recap
- Recap: MPC
- Feasibility problems
- Terminal sets and costs
- Proof of feasibility and stability
References: Predictive Control by Borrelli, Bemporad, Morari
Reminders
- Project update due Friday
- Upcoming paper presentations:
-
[RB17] Learning model predictive control for iterative tasks
-
[DSA+20] Fairness is not static
-
[FLD21] Algorithmic fairness and the situated dynamics of justice
-
Model Predictive Control ML in Feedback Sys #21 Prof Sarah Dean
22 - Model Predictive Control - ML in Feedback Sys
By Sarah Dean
