Stochastic Gradient Boosting Machines

the basics



Daina Bouquin


I'm a librarian


MS Data Analytics

MS Library and Information Science

CAS Data Science


Can a set of weak learners create a single strong learner?


Boosting algorithms  iteratively learn weak classifiers with respect to a distribution and add them to a final strong classifier

ML ensemble method/metaheuristic

Helps with bias-variance tradeoff (reduces both)


metaheuristic is a higher-level procedure or heuristic designed to find, generate, or select a heuristic (partial search algorithm) that may provide a sufficiently good solution to an optimization problem, especially with incomplete or imperfect information or limited computational capacity

Bias = error from erroneous assumptions in the learning algorithm




Variance = sensitivity to small fluctuations in the training set 

You don't want to model noise

high bias means you could miss relevant relations between features 

∴ underfitting

∴ overfitting


Boosting algorithms:

Weighted in relation to the weak predictors' accuracy

Weighting decorrelates the predictors by focusing on regions missed by past predictors

New predictors learn from previous predictor mistakes

∴ take fewer iterations to converge

Boosting means observations have an unequal probability of appearing in subsequent models


Observations with highest error

appear most




Handles overfitting

Reduces variance

Independent classifiers

Can overfit

Reduces bias & variance

Sequential classifiers

e.g. Random Forest

e.g. Gradient Boosting

Helps address main causes of differences between actual and predicted values:  variance and bias

(noise is somewhat irreducible)

Boosting with

Gradient Discent

gradient descent assuming a convex cost function

Local minimum must be a global minimum




Most common cost function is mean squared error

Too much random noise can be an issue with convex optimization.

Non-convex optimization options for boosting exist though e.g. BrownBoost

If you're worried about local minima check out restarts  (SGDR)

*The point of GD is to minimize the cost function*

(find the lowest error value/the deepest valley in that function)

 Slope points to the nearest valley

Choice of cost function will affect calculation of the gradient of each weight.


Cost function is for monitoring the error with each training example


The derivative of the cost function with respect to the weight (slope!) is where we shift the weight to minimize the error for that training example 


This gives us direction

GD optimizers use a technique called “annealing” to determine the learning rate (how small/large of a step to take) = α


Theta (weight) should decrease at each iteration

if alpha is too large we overshoot the min

if alpha is too small we take too many iterations to find the min


Black line represents a non linear loss function

If our parameters are initialized to the blue dot, we need a way to move around parameter space to the lowest point.

  1. Then just do it stochastically

  3. With every GD iteration shuffle the training set and pick a random training example
  5. Since you’re only using one training example, the path to the minima will be all zig-zag crazy


(Imagine trying to find the fastest way down a hill

only you can't see all of the curves in the hill)

May want to consider mini-batching rather than stochastic approach with very large datasets

Gradient boosting machine - Linear Regression Example

GBM can be configured to different base learners (e.g. tree, stump, linear model)

basic assumption: sum of residuals = 0

leverage pattern in residuals  to strengthen weak prediction model until residuals become randomly distributed

if you keep going you risk overfitting

Algorithmically we are minimizing our loss function such that the test loss reaches its minima

Adjusted our predictions using the fit on the residuals and accordingly adjusting value of alpha

We are doing supervised learning here


you can check for overfitting using a

k-fold cross validation

resampling procedure used to evaluate machine learning models on a limited data sample

Pseudocode for a generic gradient boosting method


StackOverflow fixed my problems

(there are a lot of people who can help you if you're lost)

Please make your code citable



Stochastic Gradient Boosting Machines: the basics

By Daina Bouquin

Stochastic Gradient Boosting Machines: the basics

Presentation given at the Center for Astrophysics Machine Learning Journal Club. December 7, 2018

  • 496