First-order methods for structured optimization
Huang Fang, Department of Computer Science
Supervisor: Michael P. Friedlander
October 13th, 2021
- Optimization is everywhere: machine learning, operation research, data mining, theoretical computer science, etc.
x∈Cmin f(x)
\underset{x \in C}{\min} ~f(x)
- First-order methods:
x(t)=span{x(0),∇f(x(0)),∇f(x(1)),…,∇f(x(t−1))}
x^{(t)} = \text{span}\{ x^{(0)}, \nabla f( x^{(0)} ), \nabla f( x^{(1)} ), \ldots, \nabla f( x^{(t-1)} ) \}
- Why first-order methods?
Background and motivation
x(t)
x^{(t)}
x(t+1)
x^{(t+1)}
The literature
- Gradient descent can be traced back to Cauchy's work in1847.
- It is gaining increasing interest in the past 3 decades due to its empirical success.
- Fundamental works have been done by pioneering researchers (Bertsekas, Nesterov, etc.).
- There are still gaps between theory and practice.
- Coordinate optimization
- Mrrior descent
- Stochastic subgradient method
Coordinate Optimization
Coordinate Descent
For
t=0,1,2,…,
t = 0, 1, 2, \ldots,
- Select coordinate
i∈[d]
i \in [d]
- Update
xi(t+1)=xi(t)−ηt∇if(x(t))
x_i^{(t+1)} = x_i^{(t)} - \eta_t \nabla_i f( x^{(t)} )
Different coordinate selection rules:
- Random selection:
- Cyclic selection:
- Random permuted cyclic (the matrix AMGM inequality conjecture is false [LL20, S20])
- Greedy selection (Gauss-Southwell)
i∈argmaxj∈[d]∣∇jf(x(t))∣
i \in \arg\max_{ j \in [d] } | \nabla_j f( x^{(t)} ) |
i∼uniform{1,2,…,d}
i \sim \mathrm{uniform}\{1,2,\ldots, d\}
i=(t−1) mod d+1
i = (t-1) ~\mathrm{mod}~ d + 1
GCD for Sparse Optimization
x∈Rdmin F(x):=f(x)+g(x)
\underset{ x \in \mathbb{R}^d }{\min}~F(x) \coloneqq f(x) + g(x)
regularizer
- One-norm regularization or nonnegative constraint can promote a sparse solution.
- When initailzed at zero, greedy CD is observed to have an implicit screening ability to select variables that are nonzero at solution.
Iter 1
Iter 2
Iter 3
Iter 4
Iter
T
T
g(x)=λ∥x∥1
g(x) = \lambda \| x \|_1
g(x)=δ≥0(x)
g(x) = \delta_{\geq 0}(x)
data fitting
GCD for Sparse Optimization
[FFSF, AISTATS'20]
We provide a theoretical characterization of GCD's screening ability:
- GCD converges fast in first few iterations.
- The iterate is "close" the to solution when the iterate is still sparse, and sparsity pattern will not further expand anymore.
x(0)
x^{(0)}
x∗
x^{*}
x(100)
x^{(100)}
d=104,∥x∗∥0=10
d = 10^4, \| x^* \|_0 = 10
∥x(t)∥0≤110
\| x^{(t)} \|_0 \leq 110
for
t>100
t > 100
δ
\delta
From coordinate to atom
Learning a sparse representation of an atomic set :
A
\mathcal{A}
x∗=∑a∈Acaa
x^* = \sum_{a \in \mathcal{A}} c_a a
such that
ca≥0 ∀a∈A
c_a \geq 0~\forall a \in \mathcal{A}
- sparse vector:
A={±e1,±e2,…,±ed}
\mathcal{A} = \{ \pm e_1, \pm e_2, \ldots, \pm e_d \}
- Low-rank matrix:
A={uvT∣u∈Rn,v∈Rm,∥u∥=∥v∥=1}
\mathcal{A} = \{ uv^T \mid u \in \mathbb{R}^{n}, v \in \mathbb{R}^m, \|u\| = \|v\| = 1 \}
Our contribution: how to identify the atoms with nonzero coefficients at solution during the optimization process.
[FFF21] Submitted
Online Mirror Descent
Online Convex Optimization
Play a game for rounds, for
T
T
- Propose a point
x(t)∈X.
x^{(t)} \in \mathcal{X}.
- Suffer loss
ft(x(t)).
f_t( x^{(t)} ).
t=1,2,…,T
t = 1,2,\ldots, T
The goal of online learning algorithm: obtain sublinear regret
Regret(T,z):=t=1∑Tft(x(t))−t=1∑Tft(z).
\displaystyle \mathrm{Regret} (T, z) \coloneqq \sum_{t=1}^T f_t( x^{(t)} ) - \sum_{t=1}^T f_t(z).
Regret(T,z)=o(T)→Regret(T,z)/T→0.
\mathrm{Regret} (T, z) = o(T) \to \mathrm{Regret} (T, z)/T \to 0.
player's loss
competitor's loss
Mirror Descent (MD) and Dual Averaging (DA)
- MD and DA are parameterized by mirror map, they have advantages over the vanilla projected subgradient method.
ft(x)=ctTxandX={∑i=1dxi=1,xi≥0}, ∥ct∥∞≤1.
f_t(x) = c_t^T x \quad \text{and} \quad \mathcal{X} = \left\{ \sum_{i=1}^d x_i = 1, x_i \geq 0 \right\}, ~\|c_t\|_{\infty} \leq 1.
- When is known in advance or , both MD and DA guarantee regret
T
T
O(T).
\mathcal{O}( \sqrt{T} ).
- When is unknown in advance and , then MD has rate while DA still guarantees regret.
T
T
supx,y∈XDΦ(x,y)=+∞
\sup_{x, y \in \mathcal{X}} D_\Phi(x, y) = +\infty
Ω(T)
\Omega ( T )
O(T)
\mathcal{O}( \sqrt{T} )
supx,y∈XDΦ(x,y)<+∞
\sup_{x, y \in \mathcal{X}} D_\Phi(x, y) < +\infty
Our contribution: fix the divergence issue of MD and obtain
regret.
O(T)
\mathcal{O}( \sqrt{T} )
OMD Algorithm
[FHPF, ICML'20]
Primal
Dual
x(t)
x^{(t)}
x^(t)
\hat{x}^{(t)}
∇Φ
\nabla \Phi
y^(t)
\hat{y}^{(t)}
−ηt∇ft(x(t))
-\eta_t \nabla f_t( x^{(t)} )
y(t)
y^{(t)}
x(t+1)
x^{(t+1)}
Bregman projection
∇Φ∗
\nabla \Phi^*
Figure accredited to Victor Portella
Stabilized OMD
[FHPF, ICML'20]
Primal
Dual
x(t)
x^{(t)}
x^(t)
\hat{x}^{(t)}
∇Φ
\nabla \Phi
y^(t)
\hat{y}^{(t)}
−ηt∇ft(x(t))
-\eta_t \nabla f_t( x^{(t)} )
y(t)
y^{(t)}
x(t+1)
x^{(t+1)}
Bregman projection
∇Φ∗
\nabla \Phi^*
x^(0)
\hat{x}^{(0)}
w^(t)
\hat{w}^{(t)}
}
γt
\gamma_t
With stabilization, OMD can obtain regret.
O(T)
\mathcal{O}( \sqrt{T} )
(Stochastic) Subgradient Method
Smooth v.s. Nonsmooth Minimization
- Consider minimizing a convex function
- The iteration complexity of gradient descent (GD) and subgradient descent (subGD) for smooth and nonsmooth objectives:
x∈Rdmin f(x).
\underset{ x \in \mathbb{R}^d }{\min}~ f(x).
- when is smooth:
f
f
O(1/ϵ).
\mathcal{O}(1/\epsilon).
- when is nonsmooth:
f
f
O(1/ϵ2).
\mathcal{O}(1/\epsilon^2).
smooth
nonsmooth
The empirical observation
Some discrepancies between theory and practice:
- Nonsmoothness from the model does not slow down our training in practice.
- The learning rate schedule can yield optimal iteration complexity but seldom used in practice.
ηt∝1/t
\eta_t \propto 1/\sqrt{t}
Filling the gap
Our assumption:
- The objective satisfy certain structure:
x∈Rdmin f(x)=n1∑i=1nfi(x):=n1∑i=1nℓ(hi(x)),
\underset{x \in \mathbb{R}^d}{\min}~ f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x) \coloneqq \frac{1}{n} \sum_{i=1}^n \ell(h_i(x)),
where is a nonnegative, , convex, 1-smooth loss function, 's are Lipschitz continuous.
ℓ:R→R≥0
\ell : \mathbb{R} \to \mathbb{R}_{\geq 0}
- The interpolation condition: there exist such that
x∗
x^*
f(x∗)=0.
f(x^*) = 0 .
hi
h_i
We prove
[FFF, ICLR'21]
- Convex objective:
- Strongly convex objective:
O(1/ϵ2)→O(1/ϵ).
\mathcal{O}(1/\epsilon^2) \to \mathcal{O}(1/\epsilon).
O(1/ϵ)→O(log(1/ϵ)).
\mathcal{O}(1/\epsilon) \to \mathcal{O}(\log(1/\epsilon)).
Optimal
infℓ=0
\inf \ell = 0
- The above rates match the rate of SGD for smooth objectives.
Acknowledgment
Advisor
Committee Members
Collaborators
University Examiners
External Reviewer
Thank you! Questions?
First-order methods for structured optimization Huang Fang, Department of Computer Science Supervisor: Michael P. Friedlander October 13th, 2021