Polar Alignment
and
Primal Retrieval
Zhenan Fan
Department of Computer Science
Collaborators:
Huang Fang, Yifan Sun, Halyun Jeong, Michael Friedlander
Atomic Decomposition
[Chen, Donoho & Sauders'01; Chandrasekaran et al.'12]
- sparse n-vectors
-
low-rank matrices
x=∑jcjejA={±e1,…,±en}
x = \sum_j c_j e_j \quad \mathcal{A} = \{\pm e_1, \dots, \pm e_n\}
X=∑jcjujvjTA={uvT∣∥u∥=∥v∥=1}
X = \sum_j c_ju_jv_j^T \quad \mathcal{A} = \{uv^T \mid \|u\| = \|v\| = 1\}
How do we identify the support of a vector x with respect to an arbitrary atomic set A?
cardinality
x = \sum\limits_{j=1}^\purple{\large r} \blue{c_j} a_j, \quad \green{a_j} \in \red{\mathcal{A}}
atomic set
weight
atom
Gauge and Support Functions
Gauge function
γA(x)=inf{a∈A∑ca x=a∈A∑caa,ca≥0}
\gamma_{\mathcal{A}}(x) =
\inf\left\{ \sum\limits_{a\in\mathcal{A}}c_a ~\big\vert~ x = \sum\limits_{a\in\mathcal{A}} c_a a, c_a \geq 0 \right\}
Support function
σA(z)=sup{⟨a,z⟩ a∈A}
\sigma_{\mathcal{A}}(z) =
\sup\left\{ \langle a, z \rangle ~\big\vert~ a \in \mathcal{A} \right\}
epiγA=cone(A×{1})
\mathop{epi} \gamma_\mathcal{A} = \mathop{cone}( \mathcal{A} \times \{1\})
epiσA=cone(A∘×{1})
\mathop{epi} \sigma_\mathcal{A} = \mathop{cone}( \mathcal{A}^\circ \times \{1\})
Polar Alignment
Polar inequality
⟨x,z⟩≤γA(x)⋅σA(z)∀(x,z)∈domγA×domσA
\langle x, z \rangle \leq \gamma_\mathcal{A}(x) \cdot \sigma_\mathcal{A}(z)
\quad
\forall (x, z) \in \mathop{dom}\gamma_\mathcal{A} \times \mathop{dom}\sigma_\mathcal{A}
Alignment
(x,z)isA−aligned⟺⟨x,z⟩=γA(x)⋅σA(z)
(x, z) \enspace\text{is}\enspace \mathcal{A}-\text{aligned}
\iff
\langle x, z \rangle = \gamma_\mathcal{A}(x) \cdot \sigma_\mathcal{A}(z)
(x,z)isA−aligned⇒{a∈A ∣ a exists in the decomposition of x}supp(A,x)⊆{a∈A ∣ ⟨a,z⟩=σA(z)}face(A,z)
(x, z) \enspace\text{is}\enspace \mathcal{A}-\text{aligned} \Rightarrow
\underbrace{ \mathop{supp}(\mathcal{A}, x) }_{\red{ \{a \in \mathcal{A} ~\mid~ a \text{ exists in the decomposition of } x\}}}
\subseteq
\underbrace{ \mathop{face}(\mathcal{A}, z)}_{\red{ \{a \in \mathcal{A} ~\mid~ \langle a, z \rangle = \sigma_\mathcal{A}(z)\}}}
Theorem
Examples
A={±e1,…,±en}
\mathcal{A} = \{\pm e_1, \dots, \pm e_n\}
face(A,z)={sgn(zi)⋅ei ∣ ∣zi∣=maxj∣zj∣}
\mathop{face}(\mathcal{A}, z) = \{ \mathop{sgn}(z_i)\cdot e_i ~\mid~ |z_i| = \max_j |z_j| \}
supp(A,x)={sgn(xi)⋅ei ∣ xi=0}
\mathop{supp}(\mathcal{A}, x) = \{ \mathop{sgn}(x_i)\cdot e_i ~\mid~ x_i \neq 0 \}
Sparse vector
Low-rank Matrix
A={uvT∣∥u∥=∥v∥=1}
\mathcal{A} = \{uv^T \mid \|u\| = \|v\| = 1\}
supp(A,X)={u1v1T,…,urvrT}
\mathop{supp}(\mathcal{A}, X) = \{ u_1v_1^T, \dots, u_rv_r^T \}
X has rank r
face(A,Z)={u1v1T,…,udvdT}
\mathop{face}(\mathcal{A}, Z) = \{ u_1v_1^T, \dots, u_dv_d^T \}
largest singular value of Z has multiplicity d
Alignment in Structured Optimization
xminf(Mx)+ργA(x)
\min\limits_x \enspace f(Mx) + \rho\gamma_\mathcal{A}(x)
xminf(Mx)subject toγA(x)≤τ
\min\limits_x \enspace f(Mx)
\enspace\text{subject to}\enspace
\gamma_\mathcal{A}(x) \leq \tau
xminγA(x)subject tof(Mx)≤α
\min\limits_x \enspace \gamma_\mathcal{A}(x)
\enspace\text{subject to}\enspace
f(Mx) \leq \alpha
(P1)
(P2)
(P3)
Theorem
(x∗,MTy∗)isA−alignedwherey∗=∇f(Mx∗)
(x^*, M^Ty^*) \enspace\text{is}\enspace \mathcal{A}-\text{aligned}
\quad\text{where}\quad
y^* = \nabla f(Mx^*)
(y* is same as optimal dual variable up to proper scaling.)
Extension to Sum of Sets
Theorem
A=i=1∑kAiand{xi∗}i=1k∈x1,…,xkargmin{i=1,…,kmaxγAi(xi)∣i=1∑kxi=x∗}⇒(xi∗,MTy∗)isAi−aligned
\mathcal{A} = \sum\limits_{i=1}^k \mathcal{A}_i
\enspace\text{and}\enspace
\{x_i^*\}_{i=1}^k \in \argmin\limits_{x_1,\dots,x_k}\left\{ \max\limits_{i=1,\dots,k} \gamma_{\mathcal{A}_i}(x_i) \mid \sum\limits_{i=1}^k x_i = x^*\right\}
\Rightarrow
(x_i^*, M^Ty^*) \enspace\text{is}\enspace \mathcal{A}_i-\text{aligned}
xminf(Mx)+ργA(x)
\min\limits_x \enspace f(Mx) + \rho\gamma_\mathcal{A}(x)
xminf(Mx)subject toγA(x)≤τ
\min\limits_x \enspace f(Mx)
\enspace\text{subject to}\enspace
\gamma_\mathcal{A}(x) \leq \tau
xminγA(x)subject tof(Mx)≤α
\min\limits_x \enspace \gamma_\mathcal{A}(x)
\enspace\text{subject to}\enspace
f(Mx) \leq \alpha
(P1)
(P2)
(P3)
Cardinality-Constrained Data-Fitting
Assumption
x∗∈xargminγA(x)subject to∥Mx−b∥≤α
x^* \in \argmin\limits_x \enspace \gamma_\mathcal{A}(x)
\enspace\text{subject to}\enspace
\|Mx - b\| \leq \alpha
Findx∈Xsuch thatcard(A,x)≤kand∥Mx−b∥≤α
\text{Find}\enspace x \in \mathcal{X}
\enspace\text{such that}\enspace
\red{ \mathop{card}(\mathcal{A}, x) } \leq k
\enspace\text{and}\enspace
\|Mx - b\| \leq \alpha
(P)
is feasible to (P)
inf{nnz(c)∣x=∑a∈Acaa,ca≥0}
\red{ \inf\{ \mathop{nnz}(c) \mid x = \sum_{a \in \mathcal{A}} c_a a, \enspace c_a \geq 0\} }
Dual problem
τ∈R+,y∈Yminτs.t.(y,τ)∈cone(MA×{1})andy∈B2(b,α)
\mathop{min}\limits_{\tau \in \mathbb{R}_+, y \in \mathcal{Y}}\enspace
\tau \enspace\text{s.t.}\enspace
(y, \tau) \in \mathop{cone}(M\mathcal{A} \times \{1\})
\enspace\text{and}\enspace y \in \mathbb{B}_2(b, \alpha)
Goal retrieve a primal variable near-feasible to (P) from a near-optimal dual variable
Primal Retrieval
Essential Cone of Atoms
EssConeA,k(M∗y)∈{x∣card(A,x)≤k}
\mathop{EssCone}_{\mathcal{A}, k}(M^*y) \in \{x \mid \mathop{card}(\mathcal{A}, x) \leq k\}
Primal retrieval
xy∈xargmin∥Mx−b∥subject tox∈EssConeA,k(M∗y)
x_y \in \argmin\limits_{x} \|Mx - b\|
\enspace\text{subject to}\enspace
x \in \mathop{EssCone}_{\mathcal{A}, k}(M^*y)
(PR)
Key Idea
card(A,xy)≤k
\mathop{card}(\mathcal{A}, x_y) \leq k
(PR) is easy to solve when k is small
xy∗
x_{y^*}
is feasible to (P)
Polyhedral Atomic Set
EssConeA,k(M∗y)=coneAk
\mathop{EssCone}_{\mathcal{A}, k}(M^*y) = \mathop{cone} \red{ \mathcal{A}_k }
Ak={ai}i=1k⊆Asuch that⟨M∗y,ai⟩≥⟨M∗y,a⟩∀a∈A∖{ai}i=1k
\red{
\mathcal{A}_k = \{a_i\}_{i=1}^k \subseteq \mathcal{A}
\enspace\text{such that}\enspace
\langle M^*y, a_i \rangle \geq \langle M^*y, a \rangle
\enspace \forall a \in \mathcal{A} \setminus \{a_i\}_{i=1}^k
}
xy=i=1∑kc^iaiwithc^∈c≥0argmin∥M∑i=1kciai−b∥
x_y = \sum\limits_{i=1}^k \hat c_i a_i
\enspace\text{with}\enspace
\hat c \in \argmin\limits_{\red{ c \geq 0 }}\enspace \|M\sum_{i=1}^k c_ia_i - b \|
(PR)
can be removed when A is symmetric
Theorem
Suppose the primal problem is non-degenerate
(a∈supp(A,x∗)or⟨a,M∗y∗⟩≤σA(M∗y∗)−δ)
\red{ ( a \in \mathop{supp}(\mathcal{A}, x^*)
\enspace\text{or}\enspace
\langle a, M^*y^* \rangle \leq \sigma_\mathcal{A}(M^*y^*) - \delta) }
ϵy∈O(δ)⇒card(A,x)≤kand∥Mx−b∥≤α
\red{\epsilon_y} \in \mathcal{O}(\sqrt{\delta})
\Rightarrow
\mathop{card}(\mathcal{A}, x) \leq k
\enspace\text{and}\enspace
\|Mx - b\| \leq \alpha
(duality gap)
Experiment: Basis Pursuit Denoise
Findx∈Rnsuch thatnnz(x)≤kand∥Mx−b∥≤α
\text{Find}\enspace x \in \mathbb{R}^n
\enspace\text{such that}\enspace
\mathop{nnz}(x) \leq k
\enspace\text{and}\enspace
\|Mx - b\| \leq \alpha
(P)
Test problems from Sparco [van den Berg et al.'09]
Spectral Atomic Set
A={uvT∣∥u∥=∥v∥=1}andM∗(Y)=[UkU−k][ΣkΣ−k][VkTV−kT]
\mathcal{A} = \{uv^T \mid \|u\| = \|v\| = 1\}
\enspace\text{and}\enspace
M^*(Y) =
\begin{bmatrix}
U_k & U_{-k}
\end{bmatrix}
\begin{bmatrix}
\Sigma_k & \\
& \Sigma_{-k}
\end{bmatrix}
\begin{bmatrix}
V_k^T \\
V_{-k}^T
\end{bmatrix}
EssConeA,k(M∗y)=coneAk={UkCVkT∣C∈Rk×k}
\mathop{EssCone}_{\mathcal{A}, k}(M^*y) = \mathop{cone} \red{ \mathcal{A}_k } = \{U_k C V_k^T \mid C \in \mathbb{R}^{k\times k}\}
Ak={uvT∣u∈range(Uk), v∈range(Vk), ∥u∥=∥v∥=1}⊂A
\red{
\mathcal{A}_k = \{ uv^T | u\in\mathop{range}(U_k),\ v\in\mathop{range}(V_k),\ \|u\| =\|v\| =1 \} \subset \mathcal{A}
}
XY=UkC^VkTwithC^∈C∈Rk×kargmin∥M(UkCVkT)−b∥
X_Y = U_k \hat C V_k^T
\enspace\text{with}\enspace
\hat C \in \argmin\limits_{C \in \mathbb{R}^{k\times k}}\enspace \|M( U_k C V_k^T) - b \|
(PR)
Theorem
rank(XY)≤kand∥M(XY)−b∥≤α+O(ϵY)
\mathop{rank}(X_Y) \leq k
\enspace\text{and}\enspace
\|M(X_Y) - b\| \leq \alpha + \mathcal{O}(\sqrt{\epsilon_Y})
Experimet: Low-Rank Matrix Completion
FindX∈Rm×nsuch thatrank(x)≤kand(i,j)∈Ω∑21(Xi,j−Bi,j)2≤α
\text{Find}\enspace X \in \mathbb{R}^{m\times n}
\enspace\text{such that}\enspace
\mathop{rank}(x) \leq k
\enspace\text{and}\enspace
\sum\limits_{(i,j)\in\Omega} \frac{1}{2}(X_{i,j} - B_{i,j})^2 \leq \alpha
(P)
Simiar experiment as in [Candès & Plan'10]
X♮∈R6798×366
X^\natural \in \mathbb{R}^{6798\times 366}
( from National Centers for Environmental Information)
X♮
X^\natural
is approximately low-rank
We subsample 50% of
X♮
X^\natural
Polar Alignment and Primal Retrieval Zhenan Fan Department of Computer Science Collaborators: Huang Fang, Yifan Sun, Halyun Jeong, Michael Friedlander