introml-fa24-midterm-review

https://introml.mit.edu/

Fall 24 Midterm Review

Shen Shen

October 18, 2024

Intro to Machine Learning

Outline

Rundown
Past Exams
Q&A

Week 1 - IntroML

Terminologies
- Training, validation, testing
- Identifying overfitting and underfitting
Concrete processes
- Learning algorithm
- Validation and Cross-validation
- Concept of hyperparameter

Week 2 - Regression

Problem Setup
Analytical solution formula \(\theta^*=\left(\tilde{X}^{\top} \tilde{X}\right)^{-1} \tilde{X}^{\top} \tilde{Y}\) (and what's \(\tilde{X}\))
When \(\tilde{X}^{\top} \tilde{X}\) not invertible (optimal solutions still exist; just not via the "formula")
- Practically (two scenarios)
- Visually (obj fun no longer of "bowl" shape, instead has "half-pipe" shape)
- Mathematically (loss of solution uniqueness)
Regularization
- Motivation, how to, when to

Week 3 - Gradient Descent

The gradient vector (both analytically and conceptually)
The gradient-descent algorithm and the key update formula
(Convex + small-enough step-size + gradient descent + global min exists + run long enough) guarantee convergence to a global min
- What happens when any of these conditions is violated
How does the stochastic variant differ (Set up, run-time behavior, and conclusion)

Week 4 - Classification

(Binary) linear classifier (sign based)
(Binary) Logistic classifiers (sigmoid, NLL loss)
Linear separator (the equation form, visual form with normal vector)
Linear separability (interplay with features)
How to handle multiple classes
- Softmax generalization (Softmax, cross-entropy)
- Multiple sigmoids
- One-vs-one, one-vs-all

Week 5 - Features

Feature transformations
- Apply a fixed feature transformation
- Hand-design feature transformation (e.g. towards getting linear separability)
- Interplay between the number of features, the quality of features, and the quality of learning algorithms
Feature encoding
- One-hot, thermometer, factored, numerical, standardization
- When and why to use any of those

Week 6 - Neural Networks

Forward-pass (for evaluation)
Backward-pass (via backpropogation, for optimization)
Source of expressiveness
Output layer design
- dimension, activation, loss
Hand-designing weights
- to match some given function form
- achieve some goal (e.g. separate a given data set)

import random
terms= ["spring2024",
        "fall2023",
        "spring2023",
        "fall2022",
        "spring2022",
        "fall2021",
        "fall2019",
        "spring2019",
        "fall2018"]
qunums = range(1,10)
base_URL = "https://introml.mit.edu/_static/fall24/midterm/review/midterm-"

term = random.choice(terms)
num = random.choice(qunums)
print("term:", term)
print("question number:", num)
print(f"Link: {base_URL+term}.pdf")

Review Question Sampler