Intro to Machine Learning

https://introml.mit.edu/

Lecture 2:  Linear regression and regularization

Shen Shen

Feb 9, 2024

(many slides adapted from Tamara Broderick)

Logistical issues? Personal concerns?
We’d love to help out at
6.390-personal@mit.edu

Logistics

• 11am Section 3 and 4 are completely full and we have many requests to switch. Physical space packed.
• If at all possible, please help us by signup/switch to other slots.

• OHs start this Sunday, please also join our Piazza 
• Thanks for all the assignments feedback. We are adapting on-the-go but these certainly benefit future semesters.
• Start to get assignments due now. (first up, exercises 2, keep an eye on the "due")

How do Lectures fit in the course components/line up?

Optimization + first-principle physics 

5min primer to an optimization problem

using OLS

• When \(\theta^*\) exists, guaranteed to be unique minimizer of 

Now, the catch:

may not be well-defined

• \(\theta^*=\left(\tilde{X}^{\top} \tilde{X}\right)^{-1} \tilde{X}^{\top} \tilde{Y}\) is not well-defined if \(\left(\tilde{X}^{\top} \tilde{X}\right)\) is not invertible

• In particular,\(\left(\tilde{X}^{\top} \tilde{X}\right)\) is not invertible if and only if \(\tilde{X}\) is not full column rank

Now, the catch:

is not well-defined

 if \(\tilde{X}\) is not full column rank

1. if \(n\)<\(d\) 
2. if columns (features) in \( \tilde{X} \) have linear dependency

Recap:

1. if \(n\)<\(d\) (i.e. not enough data)
2. if columns (features) in \( \tilde{X} \) have linear dependency (i.e., so-called co-linearity)

• Both cases do happen in practice
• In both cases, loss function is a "half-pipe"
• In both cases, infinitily-many optimal hypotheses
• Side-note: sometimes noise can resolve invertabiliy issue, but undesirable

is not defined

Regularization

Cross-validation

Cross-validation

Cross-validation

Cross-validation

Cross-validation

Cross-validation

Cross-validation

Summary

• One strategy for finding ML algorithms is to reduce the ML problem to an optimization problem.  
• For the ordinary least squares (OLS), we can find the optimizer analytically, using basic calculus!  Take the gradient and set it to zero. (Generally need more than gradient info; suffices in OLS)
• Two ways to approach the calculus problem:  write out in terms of explicit sums or keep in vector-matrix form.   Vector-matrix form is easier to manage as things get complicated (and they will!)   There are some good discussions in the lecture notes.

Summary

• What does it mean to well posed.  
• When there are many possible solutions, we need to indicate our preference somehow.
• Regularization is a way to construct a new optimization problem
• Least-squares regularization leads to the ridge-regression formulation.  Good news:  we can still solve it analytically!
• Hyper-parameters and how to pick them.  Cross-validation