Householder-based algorithm for the general eigenvalue problem
Luis Manuel Román García
ITAM , 2018
Presentation Overview
- Problem description
- QZ traps and pitfalls
- The Hessenberg reduction step
- Conclusion
Problem description
What is a hard problem?
Ax = \lambda x
Ax=λx
A quick glance through the contents of the average 60 papers per year in SIMAX shows that roughly 40% of the papers are associated with eigenvalue problem research, and it is likely that this holds more or less for the many papers per year that focus on numerical linear algebra.
A harder problem
Ax = \lambda B x
Ax=λBx
The generalized eigenvalue problem
The generalized eigenvalue problem
Steps:
1.- Reduce B to upper triangular (QR)
2.- Reduce A to Hessenberg form
3.- Reduce A to quasitriangular form
4.- Compute the eigenvalues
5.- Compute the eigenvectors
QZ traps and pitfalls
QZ traps and pitfalls
1.- Infinite eigenvalues
2.- Vanishing entries
3.- Computationally intensive
The Hessenberg reduction step
Second order methods
w_{k+1} = w_k - \alpha H_{(|S|_k)} \frac{1}{|X|_k}\displaystyle\sum_{i=1}^{|X|_k}\nabla l(h_{w_k})
wk+1=wk−αH(∣S∣k)∣X∣k1i=1∑∣X∣k∇l(hwk)
|S|_k\leq |X|_k
∣S∣k≤∣X∣k
Under some regularity assumptions, the best we can expect is super linear - quadratic convergence:
Best case scenario
|X_k| \geq |X_0|\eta_k^k;\quad |X_0|\geq\bigg(\frac{6v\gamma M}{\hat{\mu}^2}\bigg), \eta_k > \eta_{k-1}, \eta_k \rightarrow\infty, \eta_1>1
∣Xk∣≥∣X0∣ηkk;∣X0∣≥(μ^26vγM),ηk>ηk−1,ηk→∞,η1>1
|S_k| > |S_{k-1}|;\quad \displaystyle\lim_{k\rightarrow\infty}|S_k|=\infty; \quad |S_0|\geq\bigg(\frac{4\sigma}{\hat{\mu}}\bigg)^2
∣Sk∣>∣Sk−1∣;k→∞lim∣Sk∣=∞;∣S0∣≥(μ^4σ)2
\|w_0-w^*\|\leq\frac{\hat{\mu}}{3\gamma M}
∥w0−w∗∥≤3γMμ^
\mathbb{E}[|w_k - w^*|]\leq\tau_k\quad\quad\displaystyle\lim_{k\rightarrow\infty}\frac{\tau_{k+1}}{\tau_k}\rightarrow 0
E[∣wk−w∗∣]≤τkk→∞limτkτk+1→0
In an online scenario regret grows O(log(T)):
Best case scenario
\gamma = \frac{1}{2}\min\{\frac{1}{4GD}, \alpha\}, \epsilon = \frac{1}{\gamma^2 D^2}
γ=21min{4GD1,α},ϵ=γ2D21
regret_T \leq 5(\frac{1}{\alpha}+ GD)n\log(T)
regretT≤5(α1+GD)nlog(T)
The multiple advantages of second order methods:
- Faster convergence rate
- Embarrassingly parallel
- Take into consideration curvature information
- Ideal for highly varying functions
- Higher per iteration cost
2010-2016
-
Martens is the first to successfully train a deep convolutional neural network with L-BFGS.
-
Sutskever successfully trains a recurrent neural network with a generalized Gauss-Newton algorithm
-
Bengio achieves state of the art results training recurrent networks with second order methods
Conclusion
Householder-based algorithm for the general eigenvalue problem Luis Manuel Román García ITAM , 2018
Cache Friendly QZ
By Luis Roman
Cache Friendly QZ
- 863
