Frank Hutter and collaborators
from University of Freiburg

• Bayesian Optimization
• Auto-Weka
• Auto-SKlearn

Bayesian Optimization with RF

SMAC: Sequential Model-Based Algorithm Configuration

repeat

until time budget exhausted

construct RF model to predict performance
use that model to select promising configurations
compare each selected configuration against the best known

Bayesian Optimization with RF

Bayesian Hyperparameter Optimizers

Hyperparameter optimization library: automl.org/hpolib
 

Benchmarks

• From 2-dimensional continuous hyperparameter spaces
• To structured ones with 768 hyperparameters

Optimizers

• SMAC [Hutter et al, '11] , based on random forests
• Spearmint [Snoek et al, '12] , based on Gaussian processes
• TPE [Bergstra et al, '11] , based on 1-d distributions of good values

Results

• GP-based Spearmint is best for low-dimensional & continuous
• RF-based SMAC is best for high-dim, categorical & conditional

Auto WEKA

Feature selection

• Search method: which feature subsets to evaluate
• Evaluation method: how to evaluate feature subsets in search
• Both methods have subparameters

In total: 768 parameters, 10^47 configurations

Auto Sklearn

The AutoWEKA approach applied to scikit-lean

Improvements

• Meta-learning to warmstart Bayesian optimization
• Automated posthoc ensemble construction to combine the models Bayesian optimization evaluated

Auto Sklearn

Scikit-learn [Pedregosa et al, 2011-current]
instead of WEKA [Witten et al, 1999-current]

• 15 classifiers, with a total of 59 hyperparameters
• 13 feature preprocessors, 42 hyperparams
• 4 data preprocessors, 5 hyperparams

110 hyperpameters vs. 768 in Auto-WEKA

Auto Sklearn

• Separately Meta Learning & Ensembling helps
• Applied  together they prooved to be complementary

        Meta Learning provide better models earlier

        => Ensembling can start being helpful earlier

Meta Learning & Ensembling

Auto Sklearn

• Trivial to use, "scikit-learn like"

• Availabe online: https://github.com/automl/auto-sklearn
• Good overall results, even if not necessary the best on each data set
•  Perform better on small to medium-sized datasets

Intel's Team

with Eugene Tuv

• Scalable Ensemble Learning with stochastic feature boosting

{Code not released}

James Robert Lloyd

University of Cambridge (now at Qlearsite)

Sensible allocation of computation for
ensemble construction for multi class classification

An extension of Freeze-Thaw Bayesian Optimization to ensemble contruction

An extension of Freeze-Thaw Bayesian Optimization to ensemble contruction

Make use of the partial information gained during the training of a machine learning model in order to decide wether to:

• continue training of current model

• resume the training of a previously-considered model

An extension of Freeze-Thaw Bayesian Optimization to ensemble contruction

Components of the algorithm:

• Anytime interruptible base learning algorithms
• Evaluation of base learners
• A learning curve model
• A model of learning curve asymptotes
• A method for deciding which algorithm to explore further

An extension of Freeze-Thaw Bayesian Optimization to ensemble contruction

Components of the algorithm

• Base learning algorithms
• Evaluation of base learners
• A learning curve extrapolator
• An ensembling method 
• A model mapping individual algorithm performance to ensemble performance 

Lessons

• Time management 
• Memory management
• Run pilote algo on reduced size data

Why would you participate ?

Price money $30 000

Learning by doing

Fame

Test/Evaluate/Compare your skills and new tricks

Get a new job

Fun Game

Workshops to meet with others smart and cool humans

NIPS 2015

Build parts of the dream

Various Motivations

• Price Money
• Learning
• Test/ Evaluate / Compare your skills
• Get a new Job
• Play/Fun
• Workshops IRL
• Build parts of the dream