Intro to Machine Learning

https://introml.mit.edu/

Lecture 3:  Gradient Descent Methods

Shen Shen

Feb 16, 2024

(many slides adapted from Tamara Broderick)

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one out of the finite sum

Recall

  • A general ML approach
    • Collect data
    • Choose hypothesis class, hyperparameter, loss function
    • Train (optimize for) "good" hypothesis by minimizing loss. e.g. ridge regression
  • Great when have analytical solutions
    • But don't always have them (recall, half-pipe)
    • Even when do have analytical solutions, can be expensive to compute (recall, lab2, Q2.8,)
  • Want a more general, efficient way! => GD methods

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one out of the finite sum

Gradient

  • Def: For \(f: \mathbb{R}^m \rightarrow \mathbb{R}\), its gradient \(\nabla f: \mathbb{R}^m \rightarrow \mathbb{R}^m\) is defined at the point \(p=\left(x_1, \ldots, x_m\right)\) in \(m\)-dimensional space as the vector

 

\nabla f(p)=\left[\begin{array}{c} \frac{\partial f}{\partial x_1}(p) \\ \vdots \\ \frac{\partial f}{\partial x_m}(p) \end{array}\right]
f(x, y, z) = x^2 + y^3 + z

e.g.

another example

\nabla f(x, y, z) = \begin{bmatrix} 2x \\ 3y^2 \\ 1 \end{bmatrix}

When gradient is zero:

5 cases:

When minimizing a function, we'd hope to get a global min

Convex Functions

  • A function \(f\) on \(\mathbb{R}^m\) is convex if any line segment connecting two points of the graph of \(f\) lies above or on the graph.
  • (\(f\) is concave if \(-f\) is convex.)
  • For convex functions, local minima are all global minima.

Simple examples

Convex functions

Non-convex functions

Convex Functions (cont'd)

What do we need to know:

  • Intuitive understanding of the definition
  • If given a function, can determine if it's convex or not. (We'll only ever give at most 2D, so visually is enough)
  • Understand how (stochastic) gradient descent algorithms would behave differently depending on if convexity is satisfied.
  • For this class, OLS loss function is convex, ridge regression loss is (strictly) convex, and later cross-entropy loss function is convex too.

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum (over data)

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one data out of the finite sum

hyperparameters

Gradient descent properties

if violated:

can't run gradient descent

Gradient descent properties

if violated:

e.g. get stuck at a saddle point

Gradient descent properties

if violated:

e.g. may not terminate

Gradient descent properties

if violated:

see demo, and lab

Recall: need step-size sufficiently small 

run long enough

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one out of the finite sum

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one out of the finite sum

Gradient descent on ML objective

  • ML objective functions has typical form: finite sum
  • For instance, MSE we've seen so far:
  • Because (gradient of sum) = (sum of gradient), gradient of an ML objective :
\nabla f(\Theta)= \frac{1}{n} \sum_{i=1}^n \nabla f_i(\Theta)
  • gradient of that MSE w.r.t. \(\theta\):
\frac{2}{n} \sum_{i=1}^n\left(\theta^{\top} x^{(i)}+\theta_0-y^{(i)}\right) x^{(i)}

Outline

  • Recall (Ridge regression) => Why care about GD

  • Optimization primer

    • Gradient, optimality, convexity

  • GD as an optimization algorithm for generic function 

  • GD as an optimization algorithm for ML applications

    • ​Loss function typically a finite sum

  • Stochastic gradient descent (SGD) for ML applications

    • Pick one out of the finite sum

Stochastic gradient descent

\nabla f(\Theta)= \frac{1}{n} \sum_{i=1}^n \nabla f_i(\Theta)
\approx \nabla f_i(\Theta)

for a randomly picked \(i\)

More "random"

More "demanding"

Thanks!

We'd love it for you to share some lecture feedback.