A very very short introduction
What (x,y) values will minimize my function ?
Convexity guarantees a single global minimum
Follow the slope !
Start from some point x
Here is another nice animation
xn+1=xn−[∇2f(xn)]−1∇f(xn) x_{n+1} = x_{n} - [ \nabla^2 f(x_n) ]^{-1} \nabla f (x_n) xn+1=xn−[∇2f(xn)]−1∇f(xn)
Computing and inverting the Hessian can be very costly, quasi-Newton work around it
Example loss function for regression: L=∥y−fw(x)∥2 L = \parallel y - f_{w}(x) \parallel^2 L=∥y−fw(x)∥2
L=∑i=1N(yi−fw(xi))2 L = \sum\limits_{i=1}^{N} (y_i - f_w(x_i))^2L=i=1∑N(yi−fw(xi))2
Deconvolution
Inpainting
Denoising
y=Ax+n y = \bm{A} x + n y=Ax+n A is non-invertible or ill-conditioned
L=∥y−Ax∥2+R(x) L = \parallel y - \bm{A} x \parallel^2 + R(x) L=∥y−Ax∥2+R(x)
R(x)=λ∥x∥2 R(x) = \lambda \parallel x \parallel_2 R(x)=λ∥x∥2
Checkout the numerical tours
R(x)=λ∥Φx∥1 R(x) = \lambda \parallel \Phi x \parallel_1 R(x)=λ∥Φx∥1
By eiffl
Practical statistics series
