Convergence theory for minimizing movements schemes
Flavien Léger
joint works with Pierre-Cyril Aubin-Frankowski,
Gabriele Todeschi, François-Xavier Vialard
What I will present
Theory for minimizing movement schemes in infinite dimensions and in nonsmooth (nondifferentiable) settings, with a movement limiter given by a general cost function.
Main motivation: optimization on a space of measures \(\mathcal{P}(M)\):
minimize \(E\colon \mathcal{P}(M)\to\mathbb{R}\cup\{+\infty\}\)
Typical scheme:
\mu_{n+1}\in\operatorname*{argmin}_{\mu\in\mathcal{P}(M)} E(\mu)+\frac{1}{2\tau}D(\mu,\mu_n)
where \(D(\mu,\nu)=\)
transport cost: \(W_2^2(\mu,\nu)\), \(\mathcal{T}_c(\mu,\nu)\),...
Bregman divergence: \(\operatorname{KL}(\mu,\nu)\),...
Csiszár divergence: \(\int_M (\sqrt{\mu}-\sqrt{\nu})^2\),...
...
What I will present
Theory for minimizing movement schemes in infinite dimensions and in nonsmooth (nondifferentiable) settings, with a movement limiter given by a general cost function.
1. Formulations for implicit and explicit schemes in a general setting
More (not covered): forward–backward schemes, alternating minimization
2. Theory for rates of convergence based on convexity along specific paths, and generalized “\(L\)-smoothness” (“\(L\)-Lipschitz gradients”) for explicit scheme
Setting
\inf_{y\in Y}D(x,y)=0
Minimize \(E\colon X\to\mathbb{R}\cup\{+\infty\}\), where \(X\) is a set (set of measures, metric space...).
Use \(D\colon X\times Y\to\mathbb{R}\cup\{+\infty\}\), where \(Y\) is another set (often \(X=Y\)).
\begin{aligned}
x_n & \in\operatorname*{argmin}_{x\in X}E(x)+D(x,y_n)\\
y_{n+1}&\in\operatorname*{argmin}_{y\in Y} D(x_n,y)
\end{aligned}
Algorithm
(Implicit scheme)
Explicit minimizing movements
E(x)
D\colon X\times Y\to\mathbb{R}\cup\{+\infty\}
E\colon X\to\mathbb{R}\cup\{+\infty\}
\(\exists h\colon Y\to\mathbb{R}\cup\{+\infty\}\)
D(x,y)+h(y)
=\inf_{y\in Y}
Definition.
\(E\) is c-concave if
E
E
c-concave
not c-concave
D(x,y)+h(y)
generalizes “\(L\)-smoothness”
Explicit minimizing movements
\begin{aligned}
y_{n+1}&\in\operatorname*{argmin}_{y\in Y}D(x_n,y)+h(y)\\
x_{n+1} & \in\operatorname*{argmin}_{x\in X}D(x,y_{n+1})
\end{aligned}
E(x)
(majorize)
(minimize)
E
D\colon X\times Y\to\mathbb{R}\cup\{+\infty\}
E\colon X\to\mathbb{R}\cup\{+\infty\}
\(\exists h\colon Y\to\mathbb{R}\cup\{+\infty\}\)
D(x,y)+h(y)
=\inf_{y\in Y}
Definition.
\(E\) is c-concave if
Algorithm.
(Explicit scheme)
Assume \(E\) c-concave.
(L–Aubin-Frankowski '23)
D(x,y)+h(y)
Explicit minimizing movements
\(X,Y\) smooth manifolds, \(D\in C^1(X\times Y)\), \(E\in C^1(X)\) c-concave
\begin{aligned}
-\nabla_{\!x} D(x_n,y_{n+1})&=-\nabla E(x_n)\\
\nabla_{\!x} D(x_{n+1},y_{n+1})&=0
\end{aligned}
Under certain assumptions, the explicit scheme can be written as
x_{n+1}-x_n=-\tau\nabla f(x_n)
D(x,y)=\frac{1}{2\tau}\lVert x-y\rVert^2
x_{n+1}=\exp_{x_n}(-\tau\nabla f(x_n))
D(x,y)=\frac{1}{2\tau}d_M^2(x,y)
\nabla u(x_{n+1})-\nabla u(x_n)=-\nabla f(x_n)
D(x,y)=u(x)-u(y)-\nabla u(y)(x-y)
x_{n+1}-x_n=-\nabla^2u(x_n)^{-1}\nabla f(x_n)
D(x,y)=u(y)-u(x)-\nabla u(x)(y-x)
More: nonsmooth mirror descent, convergence rates for Newton
2. Convergence rates
EVI and convergence rates
Definition.
\forall x\in X,\quad E(x_n)+D(x_n,y_n)+(1+\mu)D(x,y_{n+1})\leq E(x)+D(x,y_n)
(Csiszár–Tusnády ’84)
(L–Aubin-Frankowski ’23)
Evolution Variational Inequality (or five-point property):
(\mu\geq 0)
x_n \in\operatorname*{argmin}_{x\in X}E(x)+D(x,y_n), \quad y_{n+1} \in\operatorname*{argmin}_{y\in Y} D(x_n,y)
If \((x_n,y_n)\) satisfy the EVI then
E(x_n)\leq E(x)+\frac{C(x,x_0,y_0)}{n}
sublinear rates when \(\mu=0\)
exponential rates when \(\mu>0\)
E(x_n)\leq E(x)+\frac{C(x,x_0,y_0,\mu)}{(1+\mu)^n-1}
Theorem.
(L–Aubin-Frankowski '23)
(Ambrosio–Gigli–Savaré ’05)
Variational c-segments and NNCC spaces
⏵ \(s\mapsto (x(s),\bar y)\) is a variational c-segment if \(D(x(s),\bar y)\) is finite and
⏵ \((X\times Y,D)\) is a space with nonnegative cross-curvature (NNCC space) if variational c-segments always exist.
\(X, Y\) two arbitrary sets, \(D\colon X\times Y\to\mathbb{R}\cup\{\pm\infty\}\).
(1-s)[D(x(0),\bar y)-D(x(0),y)]+s[D(x(1),\bar y)-D(x(1),y)].
\forall y\in Y\!,\quad D(x(s),\bar y)-D(x(s),y)\leq
x(0)
x(1)
\bar y
Definition.
(L–Todeschi–Vialard '24)
More: origins in regularity of optimal transport
(Ma–Trudinger–Wang ’05)
(Trudinger–Wang ’09)
(Kim–McCann ’10)
convexity of the set of c-concave functions
(Figalli–Kim–McCann '11)
Examples
Gromov–Wasserstein
Kullback–Leibler divergence, Hellinger, Fisher–Rao costs are NNCC
\mathcal{T}_c(\mu,\nu)=\inf_{\pi\in\Pi(\mu,\nu)}\int c(x,y)\,d\pi
Transport costs
\((\mathbb{G}\times\mathbb{G},\operatorname{GW}^2)\) is NCCC
\((\mathcal{P}(X)\times\mathcal{P}(Y),\mathcal{T}_c)\) NNCC \(\iff\) \((X\times Y,c)\) NNCC
(Polish spaces, lsc cost)
Ex: \(W_2^2\) on \(\mathbb{R}^n\), on \(\mathbb{S}^n\)...
\(\mathbf{X}=[X,f,\mu]\) and \(\mathbf{Y}=[Y,g,\nu]\in\mathbb{G}\)
\operatorname{GW}^2(\mathbf{X},\mathbf{Y})=\inf_{\pi\in\Pi(\mu,\nu)}\int\lvert f(x,x')-g(y,y')\rvert^2\,d\pi(x,y)\,d\pi(x',y')\,.
(L–Todeschi–Vialard '24)
Variational c-segments \(\approx\) generalized geodesics
Any Hilbert or Bregman cost is NNCC
Properties of NNCC spaces
Stable by products
Stable by quotients with “equidistant fibers”
Stable under Gromov–Hausdorff convergence
Metric cost \(c(x,y)=d^2(x,y)\) NNCC\(\implies\)PC
d^2(x(s),\bar y )\leq (1-s)d^2(x(0),\bar y)+s\,d^2(x(1),\bar y)-s(1-s)d^2(x(0),x(1))
(connect to: Ambrosio–Gigli–Savaré ’05)
(connect to: Kim–McCann '12)
(connect to: Loeper ’09)
(L–Todeschi–Vialard '24)
Convergence rates
Suppose that for each \(x\in X\) and \(n\geq 0\),
Then sublinear (\(\mu=0\)) or linear (\(\mu>0\)) convergence rates.
⏵ there exists a variational c-segment \(s\mapsto (x(s),y_n)\) on \((X\times Y,D)\) with \(x(0)=x_n\) and \(x(1)=x\)
⏵ \(s\mapsto E(x(s))-\mu \,D(x(s),y_{n+1})\) is convex
⏵ \(\displaystyle\lim_{s\to 0^+}\frac{D(x(s),y_{n+1})}{s}=0\)
Theorem.
(L–Aubin-Frankowski '23)
x_n \in\operatorname*{argmin}_{x\in X}E(x)+D(x,y_n), \quad y_{n+1} \in\operatorname*{argmin}_{y\in Y} D(x_n,y)
E(x_n)\leq E(x)+\frac{C(x,x_0,y_0)}{n}
E(x_n)\leq E(x)+\frac{C(x,x_0,y_0,\mu)}{(1+\mu)^n-1}
Thank you!
(Les Houches 2025-01-17)
By Flavien Léger
