A generalized semi-supervised elastic-net

Juan C. Laria
juancarlos.laria@uc3m.es

The semi-supervised framework

unsupervised

The semi-supervised framework

supervised

The semi-supervised framework

semi-supervised

The semi-supervised framework

semi-supervised

transfer-model

Elastic-net regularization

\|\bm{y} - \bm{X}\bm{\beta} \|_2^2 + \lambda_1 \|\bm{\beta}\|_1+ \lambda_2 \|\bm{\beta}\|_2^2

lasso

ridge

\bm{X}, \bm{y}

Joint trained elastic-net

\|\bm{y} - \bm{X}\bm{\beta} \|_2^2 + \gamma_1\|T(\gamma_2)\bm{\beta}\|_2^2 + \lambda_1 \|\bm{\beta}\|_1+ \lambda_2 \|\bm{\beta}\|_2^2
T(\gamma_2) = \sqrt{\gamma_2}\bm{U}(\bm{\Sigma}^2 + \gamma_2\bm{I})^{-1/2} \bm{U}^\top \bm{X}_U
\bm{X}_U = \bm{U \Sigma V}^\top
\bm{X}, \bm{y} , \bm{X}_U

Culp, M. (2013). On the semisupervised joint trained elastic net. Journal of Computational and Graphical Statistics 22 (2), 300–318.

Extended linear joint trained framework

\|\bm{y} - \bm{X}\bm{\beta} \|_2^2 + \gamma_1\|T(\gamma_2, \gamma_3)\bm{\beta}\|_2^2
T(\gamma_2, \gamma_3) = \sqrt{\gamma_2}\bm{U}(\bm{\Sigma}^2 + \gamma_2\bm{I})^{-1/2} \bm{\Sigma} \bm{V}^\top + \gamma_3 \bm{1} \bm{\mu}^\top.
\bm{X}_U - \bm{1}\bm{\mu}^\top = \bm{U \Sigma V}^\top
\bm{X}, \bm{y} , \bm{X}_U

Søgaard Larsen, J. et. al (2020). Semi-supervised covariate shift modelling of spectroscopic data. (in-press)

Semi-supervised elastic-net

L(\bm{\beta}~|~\bm{y},\bm{X}) + \gamma_1 L(\bm{\beta}~|~\bar{\bm{y}} , T(\gamma_2, \gamma_3))+ \lambda_1\|\bm{\beta}\|_1 + \lambda_2\|\bm{\beta}\|_2^2
T(\gamma_2, \gamma_3) = \sqrt{\gamma_2}\bm{U}(\bm{\Sigma}^2 + \gamma_2\bm{I})^{-1/2} \bm{\Sigma} \bm{V}^\top + \gamma_3 \bm{1} \bm{\mu}^\top.
\bm{X}_U - \bm{1}\bm{\mu}^\top = \bm{U \Sigma V}^\top
\bm{X}, \bm{y} , \bm{X}_U
\| \cdot \|_2^2
logit()
\bm{X}_U = \bm{U \Sigma V}^\top
T(\gamma_2) = \sqrt{\gamma_2}\bm{U}(\bm{\Sigma}^2 + \gamma_2\bm{I})^{-1/2} \bm{U}^\top \bm{X}_U
T(\gamma_2, \gamma_3)

Optimization

L(\bm{\beta}) + \lambda_1\|\bm{\beta}\|_{1} + \lambda_2\|\bm{\beta}\|_{2}^2\\ \textnormal{ where }\\ L(\bm{\beta}) = R(\bm{\beta}~|~\bm{y},\bm{X}) + \gamma_1 R(\bm{\beta}~|~\bar{\bm{y}} , T(\gamma_2, \gamma_3))

elastic-net with custom loss function

  • Fast iterative shrinkage-thresholding algorithm (FISTA)
  • Acelerated (block) gradient descent approach

Implementation

  • We develop a flexible and fast implementation for s2net in R, written in C++ using RcppArmadillo and integrated into R via Rcpp modules.

  • The software is available in the s2net package

install.packages("s2net")

library(s2net)
vignette(package = "s2net")

Hyper-parameter tuning

library(cloudml)
cloudml_train("main_script.R", config = "tuning.yml")
FLAGS <- flags(
    flag_numeric("lambda1", 0.01, "Lasso weight"),
    flag_numeric("lambda2", 0.01, "Ridge weight"),
    flag_numeric("gamma1", 0.1, "s2net global weight"),
    flag_numeric("gamma2", 100, "s2net covariance parameter"),
    flag_numeric("gamma3", 0.5, "s2net shift parameter")
 )
  • Cloud based solution
  • Easy to implement
  • Blackbox grid/random search, bayesian optimization available
  • Limited resources in the free tier
  • Can be pricey
  • Slow - not suitable for fast algorithms
  • All dependencies should be available from CRAN - otherwise it is very difficult to make it work

Hyper-parameter tuning

  • Cloud based solution that runs locally
  • Extremely fast, given the hardware
  • Open source
  • Very difficult to implement
  • Needs local hardware, or renting a cluster in the cloud
  • Native scala hyper-parameter tuning functions are only available for popular methods.
  • You have to implement your search function that works with your method.
library(sparklyr)
result = spark_apply(grid, my_function, context = datos)

THANK you!

Made with Slides.com