The LASSO:
least absolute shrinkage and selection operator
Pierre Ablin
12/03/2019
Parietal Tutorial
Overview
- Linear regression
- The Lasso model
- Non-smooth optimization: proximal operators
Supervised learning
y∈R,x∈RP,ε=N(0,σ2),f:RP→R
y \in \mathbb{R}, \enspace \mathbf{x} \in \mathbb{R}^P, \enspace \varepsilon = \mathcal{N}(0, \sigma^2), \enspace f: \mathbb{R}^P \rightarrow \mathbb{R}
y=f(x)+ε
y = f(\mathbf{x}) + \varepsilon
GOAL: Learn f from some realizations of (x,y)
Linear regression
y=f(x)+ε
y = f(\mathbf{x}) + \varepsilon
GOAL: Learn f from some realizations of (x,y)
Assumption:
f is linear
f(x)=β1x1+β2x2+⋯+βPxP=x⊤β
f(\mathbf{x}) = \beta_1x_1 + \beta_2 x_2 + \cdots + \beta_P x_P = \mathbf{x}^{\top} \mathbf{\beta}
GOAL: Learn β from some realizations of (x,y)
Maximum likelihood?
y=x⊤β+ε
y = \mathbf{x}^{\top} \mathbf{\beta} + \varepsilon
For one sample, if we observe xobs and yobs , we have:
p(y=yobs∣x=xobs)=p(xobs⊤β+ε=yobs∣x=xobs)
p(y = y_{\text{obs}} | \mathbf{x} = \mathbf{x}_{\text{obs}}) = p( \mathbf{x}_{\text{obs}}^{\top} \mathbf{\beta} + \varepsilon = y_{\text{obs}}| \mathbf{x} = \mathbf{x}_{\text{obs}})
i.e. :
p(y=yobs∣x=xobs)=p(ε=yobs−xobs⊤β)=2πσ21exp(−2σ2(yobs−xobs⊤β)2)
p(y = y_{\text{obs}} | \mathbf{x} = \mathbf{x}_{\text{obs}}) = p(\varepsilon = y_{\text{obs}} - \mathbf{x}_{\text{obs}}^{\top} \mathbf{\beta})
\\
= \frac{1}{\sqrt{2 \pi \sigma^2}} \exp{(- \frac{( y_{\text{obs}} - \mathbf{x}_{\text{obs}}^{\top} \mathbf{\beta})^2}{2\sigma^2})}
Maximum likelihood?
For one sample, if we observe xobs and yobs , we have:
p(y=yobs∣x=xobs)=2πσ21exp(−2σ2(yobs−xobs⊤β)2)
p(y = y_{\text{obs}} | \mathbf{x} = \mathbf{x}_{\text{obs}}) = \frac{1}{\sqrt{2 \pi \sigma^2}} \exp{(- \frac{( y_{\text{obs}} - \mathbf{x}_{\text{obs}}^{\top} \mathbf{\beta})^2}{2\sigma^2})}
If we observe a whole dataset of N i.i.d. samples(x1,y1),⋯,(xN,yN),
the likelihood of this observation is:
L(β)=p(y1=y1,⋯,yN=yN∣x1=x1,⋯,xN=xN)=p(y1=y1∣x1=x1)×⋯×p(yN=yN∣xN=xN)
\mathcal{L}(\beta) = p(y_1 = y_1, \cdots, y_N = y_N | \mathbf{x}_1 = \mathbf{x}_1, \cdots, \mathbf{x}_N=\mathbf{x}_N)
\\
= p(y_1 = y_1| \mathbf{x}_1 = \mathbf{x}_1) \times \cdots \times p(y_N = y_N| \mathbf{x}_N = \mathbf{x}_N)
\\
maximum likelihood estimator:
βMLE=argmaxL(β)=argmin∑n=1N21(yn−xn⊤β)2
\beta_{\text{MLE}} = \arg\max \mathcal{L}(\beta) = \arg\min \sum_{n=1}^N \frac12 (y_n - \mathbf{x}_n^{\top} \beta)^2
Matrix interlude
Define the ℓ2 norm of a vector:
∣∣v∣∣=n=1∑Nvn2
And note in condensed matrix form:
y=[y1,⋯,yN]⊤∈RN
X=[x1,⋯,xN]⊤∈RN×P
X = [\mathbf{x}_1,\cdots,\mathbf{x}_N]^{\top}\in\mathbb{R}^{N \times P}
Then,
∑n=1N(yn−xn⊤β)2=∣∣y−Xβ∣∣2
\sum_{n=1}^N (y_n - \mathbf{x}_n^{\top} \beta)^2 = ||\mathbf{y} - X \beta||^2
M.L.E.
βMLE=argmin21∣∣y−Xβ∣∣2
\beta_{\text{MLE}} = \arg\min \frac12 ||\mathbf{y} - X \beta||^2
Pros:
- Consistant: as N→∞, βMLE→β∗
- Comes from a clear statistical framework
- Can behave badly when the model is mispecified (e.g. f is not really linear)
- Lots of variance in βMLE when N is not big enough
- Ill posed problem when N<P !
Cons:
What do we do when n < p?
Bet on sparsity / Ockham's razor: only a few coefficients in β play a role
The ℓ0 Norm:
ℓ0(β)=Number of non-zero coefficients in β
\ell_0(\beta) = \text{Number of non-zero coefficients in }\beta
Example:
(1.1,2.5,−6.3,4.1)→4(1.1,0,0,4.1)→2(0,0,0,0)→0
(1.1, 2.5, -6.3, 4.1) \rightarrow 4
\\
(1.1, 0, 0, 4.1) \rightarrow 2
\\
(0, 0, 0, 0) \rightarrow 0
Subset selection/matching pursuit
Idea: select only a few active coefficients with the ℓ0 norm.
βsubset=argmin21∣∣y−Xβ∣∣2subject toℓ0(β)≤t
\beta_{\text{subset}} = \arg\min \frac12 || \mathbf{y} - X\beta ||^2 \enspace \text{subject to} \enspace \ell_0(\beta) \leq t
Where t is an integer controlling the sparsity of the solution.
- Non-convexity
- Instability: adding a sample may completely change β and its support.
- NP-Hard (loooong to solve)
Problems:
The Lasso
Idea: relax the ℓ0 norm to obtain a convex problem
βlasso=argmin21∣∣y−Xβ∣∣2subject toℓ1(β)≤t
\beta_{\text{lasso}} = \arg\min \frac12 || \mathbf{y} - X\beta ||^2 \enspace \text{subject to} \enspace \ell_1(\beta) \leq t
Where t is a level controlling the sparsity of the solution.
See notebook
ℓ1(β)=∣β∣1=∣β1∣+⋯+∣βP∣
\ell_1(\beta) = |\beta |_1 = |\beta_1| + \cdots + |\beta_P|
The Lasso: Lagrange formulation
βlasso=argmin21∣∣y−Xβ∣∣2+λ∣β∣1
\beta_{\text{lasso}} = \arg\min \frac12 || \mathbf{y} - X\beta ||^2 + \lambda |\beta|_1
Where λ is a level controlling the amount of sparsity.
- Relationship between λ and t depends on the data
- It promotes sparsity: there is a threshold λmax such that λ> λmax implies βlasso=0
Derivation
β=0 is a solution to the lasso if:
0∈∂β=0(21∣∣y−Xβ∣∣2+λ∣β∣1)
0 \in \partial_{\beta=0}(\frac12 ||y - X \beta||^2 + \lambda |\beta|_1)
Where ∂ is the subgradient.
∂(21∣∣y−Xβ∣∣2)={X⊤Xβ−X⊤y}
\partial(\frac12 || y - X\beta||^2) = \{X^{\top}X\beta - X^{\top} y\}
∂β=0(λ∣β∣1)=[−λ,λ]P
\partial_{\beta=0}(\lambda|\beta|_1) = [-\lambda, \lambda]^P
So the condition writes:
∀p,(X⊤y)p∈ [−λ,λ]
\forall p, \enspace (X^{\top}y)_p \in [-\lambda , \lambda]
λmax=∣X⊤y∣∞
\lambda_{\text{max}} = |X^{\top}y|_{\infty}
Lasso is useful because...
- It performs at the same time model selection and estimation : it tells you which coefficients in x are important.
Boston dataset:
Lasso is useful because...
- Leveraging sparsity enables fast solvers
If p=1000 and n=1000, computing Xβ takes ∼n×p=106 operations
Intuition:
But if we know that there are only 10 non-zero coefficients in β, it takes only ∼104 operations
The same reasoning applies to most quantities useful to estimate the model
Lasso estimation
how do we fit the model?
Gradient descent 101
Minimize f(x) where f is differentiable.
Iterate xt+1=xt−η∇f(xt)
Equivalent to minimizing a quadratic surrogate:
xt+1=argminxf(xt)+∇f(xt)⊤(x−xt)+2η1∣∣x−xt∣∣22
\mathbf{x}^{t+1} = \arg\min_{\mathbf{x}} f(\mathbf{x}^t) + \nabla f(\mathbf{x}^t)^{\top}(\mathbf{x} - \mathbf{x}^t) + \frac{1}{2\eta} ||\mathbf{x} - \mathbf{x}^t||_2^2
The simplest lasso:
P=1,N=1,X=1: can we minimize 21(y−β)2+λ∣β∣ ?
Yes: proximity operator
prox∣⋅∣(y,λ)=argminβ21(y−β)2+λ∣β∣
\text{prox}_{|\cdot |}(y, \lambda) = \arg\min_{\beta} \frac12(y-\beta)^2 + \lambda |\beta|
Soft-thresholding:
prox∣⋅∣(y,λ)=
\text{prox}_{|\cdot |}(y, \lambda) =
- 0 if ∣y∣≤λ
- y−λ if y>λ
- y+λ if y<−λ
Soft thresholding
st(y,λ)=
\text{st}(y, \lambda) =
- 0 if ∣y∣≤λ
- y−λ if y>λ
- y+λ if y<−λ
Separability:
prox∣⋅∣1(z,λ)=argminβ21∣∣z−β∣∣2+λ∣β∣1=(st(z1,λ),⋯,st(zP,λ))
\text{prox}_{|\cdot |_1}(\mathbf{z}, \lambda) = \arg\min_{\beta}\frac12 ||\mathbf{z} - \beta ||^2 + \lambda |\beta|_1 =(\text{st}(z_1, \lambda), \cdots, \text{st}(z_P, \lambda))
ISTA: Iterative soft threshdolding algorithm
βlasso=argmin21∣∣y−Xβ∣∣2+λ∣β∣1
\beta_{\text{lasso}} = \arg\min \frac12 || \mathbf{y} - X\beta ||^2 + \lambda |\beta|_1
Gradient of the smooth term:
∇β(21∣∣y−Xβ∣∣2)=X⊤(Xβ−y)
\nabla_{\beta}(\frac12||\mathbf{y} - X \beta ||^2) = X^{\top}( X \beta - \mathbf{y})
βt+1=prox∣⋅∣1(βt−ηX⊤(Xβt−y),λη)
\beta^{t+1} = \text{prox}_{|\cdot|_1}(\beta^{t} - \eta X^{\top}(X\beta^{t} - \mathbf{y}), \lambda\eta)
ISTA:
Comes from:
βt+1=argminβf(βt)+∇f(βt)⊤(β−βt)+2η1∣∣β−βt∣∣22+λ∣β∣1
\mathbf{\beta}^{t+1} = \arg\min_{\beta} f(\mathbf{\beta}^t) + \nabla f(\beta^t)^{\top}(\beta - \beta^t) + \frac{1}{2\eta} ||\beta - \beta^t||_2^2 + \lambda |\beta|_1
ISTA: Iterative soft threshdolding algorithm
βt+1=prox∣⋅∣1(βt−ηX⊤(Xβt−y),λη)
\beta^{t+1} = \text{prox}_{|\cdot|_1}(\beta^{t} - \eta X^{\top}(X\beta^{t} - \mathbf{y}), \lambda\eta)
- Some coefficients are 0 because of the prox
- One iteration takes O(min(N,P)×P)
Can be problematic for large P !
COORDInate descent
βjt+1=prox∣⋅∣1(βjt−ηX[j,:]⊤(Xβt−y),λη)
\beta^{t+1}_j = \text{prox}_{|\cdot|_1}(\beta^{t}_j - \eta X[j, :]^{\top}(X\beta^{t} - \mathbf{y}), \lambda\eta)
Idea : update only one coefficient of β at each iteration.
- If we know the residual r=Xβt−y, each update is O(n) :)
Residual update:
After updating the coordinate j:
r←r+(βjnew−βjold)X[j,:]
\mathbf{r} \leftarrow \mathbf{r} + (\beta^{\text{new}}_j - \beta^{\text{old}}_j)X[j, :]
βit+1=βit for i=j
Thanks!
The LASSO: least absolute shrinkage and selection operator Pierre Ablin 12/03/2019 Parietal Tutorial
The LASSO
By Pierre Ablin
The LASSO
- 1,576
