ISTA as a Recurrent neural network

Solving the lasso many times

Assume that we want to solve the Lasso for many observations $x_1, \cdots, x_N$ with a fixed dictionary $D$ 

Dictionary learning

$$\min_{D, z_1, \cdots, z_N} \sum_{i=1}^N \left(\frac12\|x_i - Dz_i \|^2 + \lambda \|z_i\|_1 \right)\enspace\text{s.t.} \enspace \|D_{:j}\|=1$$

Training / Testing

$\to$ $(x_1, \cdots, x_N)$ is the training set, drawn from a distribution $p$ and we want to accelerate ISTA on unseen data $x \sim p$

ISTA is a Neural Net

ISTA:

$z^{(t+1)} = \text{st}(z^{(t)} - \frac1LD^{\top}(Dz^{(t)} - x), \frac{\lambda}{L})$

Let $W_1 = I_m - \frac1LD^{\top}D$ and $W_2 = \frac1LD^{\top}$:

$z^{(t+1)} = \text{st}(W_1z^{(t)} +W_2x, \frac{\lambda}{L})$

3 iterations of ISTA = 3 layers NN

Learned-ISTA

Gregor, LeCun, Learning Fast Approximations of Sparse Coding, 2010

Learned-ISTA

A $T$-layer Lista network is a function $\Phi$ parametrized by $T$ parameters $\Theta = (W^t_1,W^t_2, \beta^t )_{t=0}^{T-1}$ 

• $z^{(0)} = 0$
• $z^{(t+1)} = st(W^t_1z^{(t)} + W^t_2x, \beta^t)$
• Return $z^{(T)}  = \Phi_{\Theta}(x)$

The parameters of the network are learned to get better results than ISTA

learning parameters

A $T$-layer Lista network is a function $\Phi$ parametrized by $T$ parameters $\Theta = (W^t_1,W^t_2, \beta^t )_{t=0}^{T-1}$

• $z^{(0)} = 0$
• $z^{(t+1)} = st(W^t_1z^{(t)} + W^t_2x, \beta^t)$
• Return $z^{(T)}  = \Phi_{\Theta}(x)$

what does the network learn?

Consider a "deep" LISTA network: $T\gg 1$, assume that we have solved the expected optimization problem:

$$\Theta \in \argmin \mathbb{E}_{x\sim p}\left[F_x(\Phi_{\Theta}(x))\right]$$

what does the network learn?

"The deep layers of LISTA only learn a better step size"

Sad result.

• The first layers of the network learn something more complicated
• Idea : learn step sizes only $\to$ SLISTA

Step-LISTA

Learn step sizes only

SLISTA seems better for high sparsity

(ALISTA works in the supervised framework, fails here)

Conclusion

- Much fewer parameters: easier to generalize

- Simpler to train ?

- This idea can be transposed to more complicated algorithms than ISTA