Klas Modin PRO
Mathematician at Chalmers University of Technology and the University of Gothenburg
A Silly Method for
QR Factorization
Klas Modin
Outline
- Riemannian geometry
- Manifold of multivariate Gaussians
- Fisher-Rao metric
- Homogeneous space structure
- QR (and Cholesky)
- Gradient flow
- Convexity and convergence
- Numerical example
Manifold
Riemannian manifold
v
v
gx(v,v)=⟨v,v⟩x
g_x(v,v) = \langle v,v\rangle_x
L(γ)=∫01gγ(s)(γ′(s),γ′(s))dt
\displaystyle L(\gamma) = \int_0^1 \sqrt{g_{\gamma(s)}\big(\gamma'(s),\gamma'(s)\big)}\mathrm{d}t
x
x
A(γ)=∫01gγ(s)(γ′(s),γ′(s))dt
\displaystyle A(\gamma) = \int_0^1 g_{\gamma(s)}\big(\gamma'(s),\gamma'(s)\big)\mathrm{d}t
Information geometry
p(x;μ,Σ)=(2π)n∣Σ∣1exp(−21(x−μ)⊤Σ−1(x−μ))
\displaystyle p(x;\mu,\Sigma) = \frac{1}{\sqrt{(2\pi)^n|\Sigma|}}\exp(-\frac{1}{2}(x-\mu)^\top \Sigma^{-1}(x-\mu))
[Amari, ~80's]
Information geometry
p(x;Σ)=(2π)n∣Σ∣1exp(−21x⊤Σ−1x)
\displaystyle p(x;\Sigma) = \frac{1}{\sqrt{(2\pi)^n|\Sigma|}}\exp(-\frac{1}{2}x^\top \Sigma^{-1}x)
[Amari, ~80's]
Manifold of inverse covariance matrices
P(n)={W∈Rn×n∣W=W⊤,W>0}
P(n) = \{ W\in \mathbb{R}^{n\times n}\mid W=W^\top, W>0 \}
p(x;W−1)=(2π)n∣W∣exp(−21x⊤Wx)
\displaystyle p(x;W^{-1}) = \sqrt{\frac{|W|}{(2\pi)^n}}\exp(-\frac{1}{2}x^\top W x)
Fisher-Rao metric on P(n)
TWP(n)={U∈Rn×n∣U=U⊤}
T_W P(n) = \{ U\in \mathbb{R}^{n\times n}\mid U=U^\top \}
U
U
W
W
gW(U,U)=21tr(W−1UW−1U)
g_W(U,U) = \frac{1}{2}\mathrm{tr}(W^{-1}UW^{-1}U)
Geodesics on P(n)
W¨−W˙W−1W˙=0
\ddot W - \dot W W^{-1}\dot W = 0
Explicit distance function
d(W0,W1)2=21tr(log(W1W0−1)log(W1W0−1))
\displaystyle d(W_0,W_1)^2 = \frac{1}{2}\mathrm{tr}\big(\log(W_1W_0^{-1})\log(W_1W_0^{-1}) \big)
Geodesic equation
W0
W_0
W1
W_1
Where is QR?!
Homogeneous space structure
I
I
fiber
π
\pi
fiber
I
I
W1
W_1
P(n)
P(n)
A
A
Q
Q
GL(n)
\mathrm{GL}(n)
O(n)\GL(n)={[A]∣A∈GL(n),[A]=O(n)⋅A}
\mathrm{O}(n)\backslash \mathrm{GL}(n) = \{ [A] \mid A\in\mathrm{GL}(n), [A]=\mathrm{O}(n)\cdot A \}
O(n)\GL(n)≃P(n)byπ:A↦A⊤A
\mathrm{O}(n)\backslash \mathrm{GL}(n) \simeq P(n) \quad\text{by}\quad \pi\colon A\mapsto A^\top A
Principal bundle
Fisher-Rao invariance
U
U
W
W
gW(U,U)=21tr(W−1UW−1U)
g_W(U,U) = \frac{1}{2}\mathrm{tr}(W^{-1}UW^{-1}U)
GL(n)×P(n)∋(A,W)↦A⊤WA∈P(n)
\mathrm{GL}(n)\times P(n) \ni (A,W) \mapsto A^\top W A \in P(n)
Right action of GL(n) on P(n)
gA⊤WA(A⊤UA,A⊤UA)=gW(U,U)
g_{A^\top W A}(A^\top U A,A^\top U A) = g_W(U,U)
Compatible metric on GL(n)
gˉA(V,V)=21tr(ℓ(VA−1)⊤ℓ(VA−1)+σ(VA−1)σ(VA−1))
\displaystyle \bar g_A(V,V) = \frac{1}{2}\mathrm{tr}\big(\ell(VA^{-1})^\top\ell(VA^{-1})+\sigma(VA^{-1})\sigma(VA^{-1}) \big)
V
V
A
A
horizontal slice
I
I
R
R
fiber
π
\pi
fiber
I
I
W1
W_1
K
K
P(n)
P(n)
A
A
Q
Q
Horizontal distribution
HorA={V∈TAGL(n)∣ℓ(VA−1)=0}
\mathrm{Hor}_A = \{ V\in T_A\mathrm{GL}(n) \mid \ell(VA^{-1}) = 0 \}
K={R∈GL(n)∣ℓ(R)=0,Rii>0}⇒TRK=HorR
K = \{ R\in \mathrm{GL}(n)\mid \ell(R)=0, R_{ii}>0 \} \Rightarrow T_R K = \mathrm{Hor}_R
horizontal slice
I
I
R
R
fiber
π
\pi
fiber
I
I
W1
W_1
K
K
P(n)
P(n)
A
A
Q
Q
QR and Cholesky in one go
Voilà!
horizontal slice
I
I
R
R
I
I
W1
W_1
A
A
Lifted gradient flow
K
K
P(n)
\mathrm{P}(n)
- Convex functional on with
- Lifted functional on
- Consider gradient flow on horizontal slice
F
F
P(n)
\mathrm{P}(n)
∇gF(W1)=0
\nabla_{g}F(W_1) = 0
GL(n)
\mathrm{GL}(n)
Fˉ(A)=F(π(A))=F(A⊤A)
\bar F(A) = F(\pi(A)) = F(A^\top A)
R˙=−∇gˉFˉ(R)
\dot R = -\nabla_{\bar g} \bar F(R)
Use relative entropy
H(W)=2n−21tr(W1W−1)+21log(det(W1W−1))
H(W) = \frac{n}{2}- \frac{1}{2}\mathrm{tr}(W_1W^{-1}) + \frac{1}{2}\log(\det(W_1W^{-1}))
R˙=∇gˉHˉ(R),Hˉ(R)=H(R⊤R)
\displaystyle \dot R = \nabla_{\bar g}\bar H(R), \qquad \bar H(R) = H(R^\top R)
Final gradient flow
R˙=21R−⊤(W1−R⊤R)+ZR,Z∈o(n)
\displaystyle \dot R = \frac{1}{2} R^{-\top}(W_1-R^\top R) + ZR, \qquad Z\in\mathfrak{o}(n)
Use your favorite Runge-Kutta method!
That is, use RK4
Convergence
Convexity lemma:
−Hess(Hˉ)R=gˉR
-\mathrm{Hess}(\bar H)_R = \bar g_R
Corollary:
dˉ2(R(t),R∞)≤e−2tdˉ2(R(0),R∞)
\bar d^2(R(t),R_\infty) \leq \, \mathrm{e}^{-2 t} \bar d^2(R(0),R_\infty)
R=[30−12]
R = \begin{bmatrix}
3 & -1 \\
0 & 2
\end{bmatrix}
W1=π(R)=R⊤R
W_1 = \pi(R) = R^\top R
Example
Example
−Hˉ(R(t))
-\bar H(R(t))
dˉ(R(t),R∞)2
\bar d(R(t),R_\infty)^2
slope of exp(−2t)
\text{slope of }\exp(-2t)
THANKS!
A Silly Method for QR Factorization Klas Modin
Silly method for QR factorization
By Klas Modin
Silly method for QR factorization
Presentation given as CAM seminar 2017-02-22.
- 1,817
