Searching for
Optimal Per-Coordinate Step-sizes with
Victor Sanches Portella
September 2023
cs.ubc.ca/~victorsp
joint with Frederik Kunstner, Nick Harvey, and Mark Schmidt
Multidimensional Backtracking
Theory Student Seminar @ University of Toronto
Gradient Descent and Line Search
Why first-order optimization?
Training/Fitting a ML model is often cast a (uncontrained) optimization problem
Usually in ML, models tend to be BIG
x∈Rdmin f(x)
\displaystyle \min_{x \in \mathbb{R}^{d}}~f(x)
d is BIG
Running time and space O(d) is usually the most we can afford
First-order (i.e., gradient based) methods fit the bill
(stochastic even more so)
Usually O(d) time and space per iteration
Convex Optimization Setting
f(y)≥f(x)+⟨∇f(y),x−y⟩+2L∥x−y∥22
\displaystyle f(y) \geq f(x) + \langle \nabla f(y), x - y\rangle + \frac{L}{2}\lVert x - y \rVert_2^2
f is convex
Not the case with Neural Networks
Still quite useful in theory and practice
{
\displaystyle \Bigg\{
More conditions on f for rates of convergence
L-smooth
x∈Rdmin f(x)
\displaystyle \min_{x \in \mathbb{R}^{d}}~f(x)
μ-strongly convex
f(y)≤f(x)+⟨∇f(y),x−y⟩+2μ∥x−y∥22
\displaystyle f(y) \leq f(x) + \langle \nabla f(y), x - y\rangle + \frac{\mu}{2}\lVert x - y \rVert_2^2
Gradient Descent
xt+1=xt−α∇f(xt)
\displaystyle x_{t+1} = x_t - \alpha \nabla f(x_t)
Which step-size α should we pick?
⟹
\displaystyle \implies
f(xt)−f(x∗)≤(1−Lμ)t(f(x0)−f(x∗))
\displaystyle f(x_t) - f(x_*) \leq \left( 1 - \frac{\mu}{L} \right)^t (f(x_0) - f(x_*))
Condition number
α=L1
\displaystyle \alpha = \frac{1}{L}
κ=μL
\displaystyle \kappa = \frac{L}{\mu}
κ Big ⟹ hard function
What Step-Size to Pick?
If we know L, picking 1/L always works
and is worst-case optimal
What if we do not know L?
Locally flat ⟹ we can pick bigger step-sizes
xt+1=xt−L1∇f(xt)
\displaystyle x_{t+1} = x_t - \tfrac{1}{L} \nabla f(x_t)
If f is L smooth, we have
f(xt+1)≤f(xt)−L121∥∇f(xt)∥22
\displaystyle f(x_{t+1}) \leq f(x_t) - \tfrac{1}{L} \tfrac{1}{2} \lVert \nabla f(x_t) \rVert_2^2
"Descent Lemma"
Idea: Pick η big and see if the "descent condition" holds
(Locally 1/η-smooth)
Backtracking Line-Search
Backtracking Line-Search
Start with αmax>2L
α←αmax/2
If
f(xt+1)≤f(xt)−α21∥∇f(xt)∥22
\displaystyle f(x_{t+1}) \leq f(x_t) - \alpha \tfrac{1}{2} \lVert \nabla f(x_t) \rVert_2^2
t←t+1
Else
While t≤T
αmax←αmax/2
\displaystyle \alpha_{\max} \gets \alpha_{\max}/2
Halve candidate space
Guarantee: step-size will be at least 21⋅L1
Armijo Condition
Beyond Line-Search?
f(x)=xTAx
\displaystyle f(x) = x^T A x
A=(1000000.001)
\displaystyle A =
\begin{pmatrix}
1000 & 0 \\
0 & 0.001
\end{pmatrix}
κ=10−6
\displaystyle \kappa = 10^{-6}
xt+1=xt−(0.001001000)∇f(xt)
\displaystyle x_{t+1} = x_t -
\begin{pmatrix}
0.001 & 0 \\
0 & 1000
\end{pmatrix}
\nabla f(x_t)
Converges in 1 step
P
O(d) space and time ⟹ P diagonal (or sparse)
Can we find a good P automatically?
"Adapt to f"
Preconditioer P
"Adaptive" Optimization Methods
Adaptive and Parameter-Free Methods
xt+1=xt−Pt⋅∇f(xt)
\displaystyle x_{t+1} = x_t - P_t \cdot \nabla f(x_t)
Preconditioner at round t
AdaGrad from Online Learning
Pt=(i≤t∑∇f(xi)∇f(xi)T)1/2
\displaystyle P_t = \Big( \sum_{i \leq t} \nabla f(x_i) \nabla f(x_i)^T \Big)^{1/2}
Diag(i≤t∑∇f(xi)∇f(xi)T)1/2
\displaystyle \mathrm{Diag}\Big( \sum_{i \leq t} \nabla f(x_i) \nabla f(x_i)^T \Big)^{1/2}
or
Better guarantees if functions are easy
while preserving optimal worst-case guarantees in Online Learning
Attains linear rate in classical convex opt (proved later)
But... Online Learning is too adversarial, AdaGrad is "conservative"
In OL, functions change every iteration adversarially
"Fixing" AdaGrad
But... Online Learning is too adversarial, AdaGrad is "conservative"
"Fixes": Adam, RMSProp, and other workarounds
"AdaGrad inspired an incredible number of clones, most of them with similar, worse, or no regret guarantees.(...) Nowadays, [adaptive] seems to denote any kind of coordinate-wise learning rates that does not guarantee anything in particular."
Francesco Orabona in "A Modern Introduction to Online Learning", Sec. 4.3
Hypergradient Methods
Idea: look at step-size/preconditioner choice as an optimization problem
Gradient descent on the hyperparameters
How to pick the step-size of this? Well...
Little/ No theory
Unpredictable
... and popular?!
Second-order Methods
Pt=∇2f(xt)
P_t = \nabla^2 f(x_t)
Newton's method
is usually a great preconditioner
Superlinear convergence
...when ∥xt− x∗∥ small
Newton may diverge otherwise
Using step-size with Newton and QN method ensures convergence away from x∗
Worse than GD
f(xt)−f(x∗)≤(1−κ21)t(f(x0)−f(x∗))
\displaystyle f(x_t) - f(x_*) \leq \left( 1 - \frac{1}{\kappa^2} \right)^t (f(x_0) - f(x_*))
κ2
\displaystyle \phantom{\kappa}^2
∇2f(x) is usually expensive to compute
Pt≈∇2f(xt)
P_t \approx \nabla^2 f(x_t)
should also help
Quasi-Newton Methods, e.g. BFGS
State of Affairs
(Quasi-)Newton: needs Hessian, can be slower than GD
Hypergradient methods: purely heuristic, unstable
Online Learning Algorithms: Good but pessimistic theory
at least for smooth optimization it seems pessimistic...
Online Learning
Smooth Optimization
1 step-size
d step-sizes
(diagonal preconditioner )
Backtracking Line-search
Diagonal AdaGrad
Coordinate-wise
Coin Betting
(non-smooth opt?)
Multidimensional Backtracking
Scalar AdaGrad
Coin-Betting
What does it mean for a method to be adaptive?
Preconditioner Search
Optimal (Diagonal) Preconditioner
μI⪯∇2f(x)⪯LI
\displaystyle \mu I \preceq \nabla^2 f(x) \preceq L I
κ1P−1⪯∇2f(x)⪯P−1
\displaystyle \frac{1}{\kappa} P^{-1} \preceq \nabla^2 f(x) \preceq P^{-1}
Optimal step-size: biggest that guarantees progress
Optimal preconditioner: biggest (??) that guarantees progress
P=L1I
\displaystyle P = \tfrac{1}{L} I
κ=μL
\displaystyle \kappa = \tfrac{L}{\mu}
f(y)≥f(x)+⟨∇f(y),x−y⟩+2L∥x−y∥22
\displaystyle f(y) \geq f(x) + \langle \nabla f(y), x - y\rangle + \frac{L}{2}\lVert x - y \rVert_2^2
L-smooth
μ-strongly convex
f(y)≤f(x)+⟨∇f(y),x−y⟩+2μ∥x−y∥22
\displaystyle f(y) \leq f(x) + \langle \nabla f(y), x - y\rangle + \frac{\mu}{2}\lVert x - y \rVert_2^2
P∗
\displaystyle P_*
minimizes κ∗ such that
κ∗1P−1⪯∇2f(x)⪯P−1
\displaystyle \frac{1}{\kappa_*} P^{-1} \preceq \nabla^2 f(x) \preceq P^{-1}
Over diagonal matrices
From Line-search to Preconditioner Search
Line-search
step-size is at least 1/2 the optimum 1/L
# backtracks ≤
log(α0L)
\displaystyle \log\Big(\alpha_{0} L \Big)
f(xt)−f(x∗)≤(1−2⋅κ1)t(f(x0)−f(x∗))
\displaystyle f(x_t) - f(x_*) \leq\Big(1 - \frac{1}{2 \cdot \kappa}\Big)^t (f(x_0) - f(x_*))
Multidimensional Backtracking
Condition number is at least 1/2d the optimum
# backtracks ≲
d⋅log(α0⋅L)
\displaystyle d \cdot \log\Big( \alpha_0 \cdot L\Big)
f(xt)−f(x∗)≤(1−2d⋅κ∗1)t(f(x0)−f(x∗))
\displaystyle f(x_t) - f(x_*) \leq\Big(1 - \frac{1}{\sqrt{2d} \cdot \kappa_*}\Big)^t (f(x_0) - f(x_*))
{
\displaystyle \Bigg\{
{
\displaystyle \Bigg\{
Worth it if 2d κ∗≪2κ
Multidimensional Backtracking
Why Naive Search does not Work
Line-search: test if step-size αmax/2 makes enough progress:
f(xt+1)≤f(xt)−αmax21∥∇f(xt)∥22
\displaystyle f(x_{t+1}) \leq f(x_t) - \alpha_{\max} \tfrac{1}{2} \lVert \nabla f(x_t) \rVert_2^2
Armijo condition
If this fails, cut out everything bigger than αmax/2
Preconditioner search:
0
0
α0
\alpha_{0}
2α0
\tfrac{\alpha_{0}}{2}
L1
\tfrac{1}{L}
4α0
\tfrac{\alpha_{0}}{4}
Test if preconditioner P makes enough progress:
Candidate preconditioners S: diagonals in a box
f(xt+1)≤f(xt)
\displaystyle f(x_{t+1}) \leq f(x_t)
−21∥∇f(xt)∥P2
- \tfrac{1}{2} \lVert \nabla f(x_t) \rVert_P^2
If this fails, cut out everything bigger than P
⟨∇f(xt),P∇f(xt)⟩
\langle \nabla f(x_t), P \nabla f(x_t) \rangle
Why Naive Search does not Work
Preconditioner search:
Test if preconditioner P makes enough progress:
Candidate preconditioners S: diagonals in a box
f(xt+1)≤f(xt)
\displaystyle f(x_{t+1}) \leq f(x_t)
⟨∇f(xt),P∇f(xt)⟩
\langle \nabla f(x_t), P \nabla f(x_t) \rangle
−21∥∇f(xt)∥P2
- \tfrac{1}{2} \lVert \nabla f(x_t) \rVert_P^2
If this fails, cut out everything bigger than P
Convexity to the Rescue
P does not yield sufficient progress
Which preconditioners can be thrown out?
All Q such that P⪯Q works, but it is too weak
h(P):=f(x−P−1∇f(x))−f(x)+21∥∇f(x)∥P2
\displaystyle
h(P) \coloneqq f(x - P^{-1} \nabla f(x)) - f(x) + \tfrac{1}{2} \lVert \nabla f(x) \rVert_P^2
P does not yield sufficient progress ⟺ h(P)>0
h(Q)≥h(P)+⟨∇h(P),Q−P⟩
\displaystyle
h(Q) \geq h(P) + \langle \nabla h(P), Q - P \rangle
Convexity ⟹
h(P)+⟨∇h(P),Q−P⟩>0
\displaystyle
h(P) + \langle \nabla h(P), Q - P \rangle > 0
⟹ Q is invalid
A separating hyperplane!
P in this half-space
{
\displaystyle \Bigg\{
Hypergradient
Convexity to the Rescue
Box as Feasible Sets
How Deep to Query?
Ellipsoid Method to the Rescue
Smallest Axis-Aligned Ellipsoid
Contraction of 1/2d from boundary
⟹
\displaystyle
\implies
Constant volume contraction
Conclusions
Theoretically principled adaptive optimization method for strongly convex smooth optimization
A theoretically-informed use of "hypergradients"
ML Optimization meets Cutting Plane methods
Stochastic case?
Heuristics for non-convex case?
Other cutting-plane methods?
?
?
?
Thanks!
Backup Slides
Searching for Optimal Per-Coordinate Step-sizes with Victor Sanches Portella September 2023 cs.ubc.ca/~victorsp joint with Frederik Kunstner , Nick Harvey , and Mark Schmidt Multidimensional Backtracking Theory Student Seminar @ University of Toronto
Preconditioner Search
By Victor Sanches Portella
Preconditioner Search
- 281
