First-order methods for structured optimization
Huang Fang, Department of Computer Science
Supervisor: Michael P. Friedlander
October 13th, 2021
- Optimization is everywhere: machine learning, operation research, data mining, theoretical computer science, etc.
\underset{x \in C}{\min} ~f(x)
- First-order methods:
x^{(t)} = \text{span}\{ x^{(0)}, \nabla f( x^{(0)} ), \nabla f( x^{(1)} ), \ldots, \nabla f( x^{(t-1)} ) \}
- Why first-order methods?
Background and motivation
x^{(t)}
x^{(t+1)}
The literature
- Gradient descent can be traced back to Cauchy's work in1847.
- It is gaining increasing interest in the past 3 decades due to its empirical success.
- Fundamental works have been done by pioneering researchers (Bertsekas, Nesterov, etc.).
- There are still gaps between theory and practice.
- Coordinate optimization
- Mrrior descent
- Stochastic subgradient method
Coordinate Optimization
Coordinate Descent
For
t = 0, 1, 2, \ldots,
- Select coordinate
i \in [d]
- Update
x_i^{(t+1)} = x_i^{(t)} - \eta_t \nabla_i f( x^{(t)} )
Different coordinate selection rules:
- Random selection:
- Cyclic selection:
- Random permuted cyclic (the matrix AMGM inequality conjecture is false [LL20, S20])
- Greedy selection (Gauss-Southwell)
i \in \arg\max_{ j \in [d] } | \nabla_j f( x^{(t)} ) |
i \sim \mathrm{uniform}\{1,2,\ldots, d\}
i = (t-1) ~\mathrm{mod}~ d + 1
GCD for Sparse Optimization
\underset{ x \in \mathbb{R}^d }{\min}~F(x) \coloneqq f(x) + g(x)
regularizer
- One-norm regularization or nonnegative constraint can promote a sparse solution.
- When initailzed at zero, greedy CD is observed to have an implicit screening ability to select variables that are nonzero at solution.
Iter 1
Iter 2
Iter 3
Iter 4
Iter
T
g(x) = \lambda \| x \|_1
g(x) = \delta_{\geq 0}(x)
data fitting
GCD for Sparse Optimization
[FFSF, AISTATS'20]
We provide a theoretical characterization of GCD's screening ability:
- GCD converges fast in first few iterations.
- The iterate is "close" the to solution when the iterate is still sparse, and sparsity pattern will not further expand anymore.
x^{(0)}
x^{*}
x^{(100)}
d = 10^4, \| x^* \|_0 = 10
\| x^{(t)} \|_0 \leq 110
for
t > 100
\delta
From coordinate to atom
Learning a sparse representation of an atomic set :
\mathcal{A}
x^* = \sum_{a \in \mathcal{A}} c_a a
such that
c_a \geq 0~\forall a \in \mathcal{A}
- sparse vector:
\mathcal{A} = \{ \pm e_1, \pm e_2, \ldots, \pm e_d \}
- Low-rank matrix:
\mathcal{A} = \{ uv^T \mid u \in \mathbb{R}^{n}, v \in \mathbb{R}^m, \|u\| = \|v\| = 1 \}
Our contribution: how to identify the atoms with nonzero coefficients at solution during the optimization process.
[FFF21] Submitted
Online Mirror Descent
Online Convex Optimization
Play a game for rounds, for
T
- Propose a point
x^{(t)} \in \mathcal{X}.
- Suffer loss
f_t( x^{(t)} ).
t = 1,2,\ldots, T
The goal of online learning algorithm: obtain sublinear regret
\displaystyle \mathrm{Regret} (T, z) \coloneqq \sum_{t=1}^T f_t( x^{(t)} ) - \sum_{t=1}^T f_t(z).
\mathrm{Regret} (T, z) = o(T) \to \mathrm{Regret} (T, z)/T \to 0.
player's loss
competitor's loss
Mirror Descent (MD) and Dual Averaging (DA)
- MD and DA are parameterized by mirror map, they have advantages over the vanilla projected subgradient method.
f_t(x) = c_t^T x \quad \text{and} \quad \mathcal{X} = \left\{ \sum_{i=1}^d x_i = 1, x_i \geq 0 \right\}, ~\|c_t\|_{\infty} \leq 1.
- When is known in advance or , both MD and DA guarantee regret
T
\mathcal{O}( \sqrt{T} ).
- When is unknown in advance and , then MD has rate while DA still guarantees regret.
T
\sup_{x, y \in \mathcal{X}} D_\Phi(x, y) = +\infty
\Omega ( T )
\mathcal{O}( \sqrt{T} )
\sup_{x, y \in \mathcal{X}} D_\Phi(x, y) < +\infty
Our contribution: fix the divergence issue of MD and obtain
regret.
\mathcal{O}( \sqrt{T} )
OMD Algorithm
[FHPF, ICML'20]
Primal
Dual
x^{(t)}
\hat{x}^{(t)}
\nabla \Phi
\hat{y}^{(t)}
-\eta_t \nabla f_t( x^{(t)} )
y^{(t)}
x^{(t+1)}
Bregman projection
\nabla \Phi^*
Figure accredited to Victor Portella
Stabilized OMD
[FHPF, ICML'20]
Primal
Dual
x^{(t)}
\hat{x}^{(t)}
\nabla \Phi
\hat{y}^{(t)}
-\eta_t \nabla f_t( x^{(t)} )
y^{(t)}
x^{(t+1)}
Bregman projection
\nabla \Phi^*
\hat{x}^{(0)}
\hat{w}^{(t)}
}
\gamma_t
With stabilization, OMD can obtain regret.
\mathcal{O}( \sqrt{T} )
(Stochastic) Subgradient Method
Smooth v.s. Nonsmooth Minimization
- Consider minimizing a convex function
- The iteration complexity of gradient descent (GD) and subgradient descent (subGD) for smooth and nonsmooth objectives:
\underset{ x \in \mathbb{R}^d }{\min}~ f(x).
- when is smooth:
f
\mathcal{O}(1/\epsilon).
- when is nonsmooth:
f
\mathcal{O}(1/\epsilon^2).
smooth
nonsmooth
The empirical observation
Some discrepancies between theory and practice:
- Nonsmoothness from the model does not slow down our training in practice.
- The learning rate schedule can yield optimal iteration complexity but seldom used in practice.
\eta_t \propto 1/\sqrt{t}
Filling the gap
Our assumption:
- The objective satisfy certain structure:
\underset{x \in \mathbb{R}^d}{\min}~ f(x) = \frac{1}{n} \sum_{i=1}^n f_i(x) \coloneqq \frac{1}{n} \sum_{i=1}^n \ell(h_i(x)),
where is a nonnegative, , convex, 1-smooth loss function, 's are Lipschitz continuous.
\ell : \mathbb{R} \to \mathbb{R}_{\geq 0}
- The interpolation condition: there exist such that
x^*
f(x^*) = 0 .
h_i
We prove
[FFF, ICLR'21]
- Convex objective:
- Strongly convex objective:
\mathcal{O}(1/\epsilon^2) \to \mathcal{O}(1/\epsilon).
\mathcal{O}(1/\epsilon) \to \mathcal{O}(\log(1/\epsilon)).
Optimal
\inf \ell = 0
- The above rates match the rate of SGD for smooth objectives.
Acknowledgment
Advisor
Committee Members
Collaborators
University Examiners
External Reviewer
Thank you! Questions?
Backup slides
Backup slides for CD
GCD for Sparse Optimization
The GS-s rule:
The GS-r rule:
The GS-q rule:
j \in \arg\max_{i \in [d]} \left\{ \min_{s \in \partial g_i(x_i)} | \nabla_i f(x) + s | \right\}
j \in \arg\max_{i \in [d]} \left\{ \left| x_i - \mathrm{prox}_{g_i/L} \left[ x_i - \frac{1}{L} \nabla_i f(x) \right] \right| \right\}
j \in \arg\max_{i \in [d]} \left\{ \min_d\left\{ \nabla_i f(x) d + \frac{Ld^2}{2} + g_i( x_i + d ) - g_i(x_i) \right\} \right\}
\displaystyle \mathrm{prox}_g (x) = \underset{y \in \mathbb{R}^d }{\arg\min}~ \frac{1}{2} \|y - x\|_2^2 + g(y)
Sparsity measure
\delta_i \coloneqq \min\{ -\nabla_i f(x^*) - l_i, u_i + \nabla_i f(x^*) \}
where .
\partial g_i(x^*_i) = [l_i, u_i]
A key property:
| \nabla_i f( x^{(t)} ) - \nabla_i f(x^*) | \leq \delta_i,~x_i^{(t)} = 0,~x_i^* = 0
\Rightarrow \quad x_i^{(t+1)} = 0
The definition of
\delta_i
\|x\|_2 \leq \| x \|_1 \leq \sqrt{d} \| x \|_2
Norm inequality:
Sparsity measure
\text{\# inactive} = \sum_{i=1}^d \textbf{1}\left\{ | \nabla_i f(x^{(t)}) - \nabla_i f(x^*) | \leq \delta_i \right\}
F( x^{(t+1)} ) - F^* = (1 - \rho_t)\left( F(x^{(t)}) - F^* \right)
Matrix AMGM inequality
\displaystyle \left\| \frac{1}{n!} \sum_{ \sigma \in \mathrm{Perm}(n) } \prod_{j=1}^n A_{\sigma(j)} \right\| \leq \displaystyle \left\| \frac{1}{n^n} \sum_{ k_1,\ldots,k_n=1 }^n \prod_{j=1}^n A_{k_j} \right\|
for all PSD matrix
A_1, \ldots, A_n \in \mathbb{R}^{d \times d}.
The matrix AMGM inequality conjecture is false [LL20, S20])
Atom identification
\mathcal{F}_\mathcal{A} ( z, \epsilon ) = \{ a \in \mathcal{A} \mid \langle a, z \rangle \geq \sigma_\mathcal{A}(z) - \epsilon \}
\mathcal{F}_\mathcal{A} ( z ) = \{ a \in \mathcal{A} \mid \langle a, z \rangle \geq \sigma_\mathcal{A}(z) \}
where is the support function.
\sigma_{\mathcal{A}}(z) = \sup_{a \in \mathcal{A}} \langle a, z \rangle
x \to x^* \quad {\Rightarrow} \quad \mathrm{supp}(x) \to \mathrm{supp}(x^*)
y \to y^* \quad \Rightarrow \quad \mathcal{F}_{\mathcal{A}}(y, \epsilon) \to \mathcal{F}_{\mathcal{A}}(y^*)
The primal-dual relationship:
\mathrm{supp}(x^*) \subseteq \mathcal{F}_{\mathcal{A}}(y^*)
This allows us to do screening base on dual variable:
Backup slides for OMD
Basics
Bregman divergence
D_\Phi(x, y) \coloneqq \Phi(x) - \Phi(y) - \langle \nabla \Phi(y), x - y \rangle
Properties of mirror map
\Phi : \bar{\mathcal{D}} \to \mathbb{R}^d
- is nonempty,
\mathcal{D}
- is differentiable on , and
\Phi
\mathcal{D}
- , where is the boundary of , i.e.,
\partial \mathcal{D}
\lim_{x \to \partial \mathcal{D}} \| \nabla \Phi(x) \| = + \infty
\mathcal{D}
\partial \mathcal{D} \coloneqq \mathrm{cl} \mathcal{D} ~\backslash~ \mathcal{D}.
\Phi(x) = \frac{1}{2} \| x \|_2^2 ~\Rightarrow~ D_\Phi(x, y) = \frac{1}{2} \| x - y \|_2^2
Examples:
\Phi(x) = \sum_{i=1}^d x_i \log(x_i) ~\Rightarrow~ D_\Phi(x, y) = \sum_{i=1}^d x_i \log(x_i/y_i)
Backup slides for SGD
Basics
| smooth | nonsmooth | smooth+IC | nonsmooth + IC | |
|---|---|---|---|---|
| convex | ||||
| strong cvx |
\mathcal{O}(1/\epsilon)
\mathcal{O}(1/\epsilon^2)
\mathcal{O}(1/\epsilon^2)
\mathcal{O}(1/\epsilon)
\mathcal{O}(1/\epsilon)
\mathcal{O}(\log(1/\epsilon))
\mathcal{O}(1/\epsilon^2)
\mathcal{O}(1/\epsilon)
Semi-smooth properties
Assume , is -Lipschitz continuous and is nonnegative, convex, 1-smooth, minimum at 0, 1-dimensional function. Then
f(x) \coloneqq \ell( h(x) )
h
L
\ell
\| \partial f(x) \|^2 \leq 2 L^2 f(x)
(the generalized growth condition)
\| \partial f(x) - \partial f(y) \| \leq L^2 \| x - y \| + 2L \sqrt{2\min\{ f(x), f(y) \}}
Extension
High probability error bound
\mathcal{O}( \log(1/\delta)/\epsilon )
Conjecture: variance reduced SGD (SAG, SVRG, SAGA) + generalized Freedman inequality
\to
Simple proof of exponential tail bound
thesis
By Fang Huang
thesis
Slides for my oral defense
