federica bianco
astro | data science | data for good
foundations of data science for everyone
dr.federica bianco | fbb.space | fedhere | fedhere
IV: Machine Learning & Linear Regression
this slide deck:
what is machine learning?
1
a model is a low dimensional representation of a higher dimensionality datase
the best way to think about it in the ML context:
what is a model?
what is machine learning?
[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed.
Arthur Samuel, 1959
what is machine learning?
model
parameters: slope, intercept
[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed.
Arthur Samuel, 1959
what is machine learning?
[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed.
Arthur Samuel, 1959
model
parameters: slope, intercept
data
what is machine learning?
ML: any model with parameters learnt from the data
Machine Learning models are parametrized representation of "reality" where the parameters are learned from finite sets of realizations of that reality
(note: learning by instance, e.g. nearest neighbours, may not comply to this definition)
Machine Learning is the disciplines that conceptualizes, studies, and applies those models.
Key Concept
what is machine learning?
used to:
- understand the structure of a feature space
- regression : predict values based on conditions based on examples
- classify based on examples
- understand which features are important for the success of the model (to get close to causality)
General ML points
unsupervised vs supervised learning
Clustering
partitioning the feature space so that the existing data is grouped (according to some target function!)
understand the structure of a feature space
Clustering
partitioning the feature space so that the existing data is grouped (according to some target function!)
Unsupervised learning
- understanding structure
- anomaly detection
- dimensionality reduction
All features are observed for all datapoints
unsupervised vs supervised learning
understand the structure of a feature space
Clustering
partitioning the feature space so that the existing data is grouped (according to some target function!)
Classifying & regression
finding functions of the variables that allow to predict unobserved properties of new observations
Unsupervised learning
- understanding structure
- anomaly detection
- dimensionality reduction
All features are observed for all datapoints
unsupervised vs supervised learning
prediction and classification based on examples
Clustering
partitioning the feature space so that the existing data is grouped (according to some target function!)
Classifying & regression
finding functions of the variables that allow to predict unobserved properties of new observations
Unsupervised learning
- understanding structure
- anomaly detection
- dimensionality reduction
All features are observed for all datapoints
unsupervised vs supervised learning
prediction and classification based on examples
Clustering
partitioning the feature space so that the existing data is grouped (according to some target function!)
Classifying & regression
finding functions of the variables that allow to predict unobserved properties of new observations
Unsupervised learning
- understanding structure
- anomaly detection
- dimensionality reduction
Supervised learning
- classification
- prediction
- feature selection
All features are observed for all datapoints
Some features are not observed for some data points we want to predict them.
unsupervised vs supervised learning
prediction and classification based on examples
Unsupervised learning
Supervised learning
All features are observed for all datapoints
and we are looking for structure in the feature space
Some features are not observed for some data points we want to predict them.
The datapoints for which the target feature is observed are said to be "labeled"
Semi-supervised learning
Active learning
A small amount of labeled data is available. Data is cluster and clusters inherit labels
The code can interact with the user to update labels and update model.
also...
unsupervised vs supervised learning
what is machine learning?
extract features and create models that allow prediction where the correct answer is known for a subset of the data
supervised learning
identify features and create models that allow to understand structure in the data
unsupervised learning
-
k-Nearest Neighbors
-
Regression
-
Support Vector Machines
-
Classification/Regression Trees
-
Neural networks
-
clustering
-
Principle Component Analsysis
-
Apriori (association rule)
what is a model?
2
1
choose your model :
choose a mathematical formula to represent the behavior you see/expect in the data
line model: ax+b
1
choose your model :
choose a mathematical formula to represent the behavior you see/expect in the data
line model: ax+b
a mathematical formula describes a family of shapes. The parameters define the exact shape: in a line fit the parameters are...
- a: slope
- b: intercept
1
choose your model :
choose a mathematical formula to represent the behavior you see/expect in the data
polynomial model:
generalizatoin to a polynomial fit:
- the degree N of the polynomial is a hyperparameter of the model
we choose hyperparameters,
we fit parameters
polyn = \sum_{i=0}^N c_i x^i
N=12: goes exactly through each data point but it has too many parameters- N = number of data points
N=2: a conservative hyperparameter choice
1
choose your model :
choose a mathematical formula to represent the behavior you see/expect in the data
polynomial model:
generalizatoin to a polynomial fit:
- the degree N of the polynomial is a hyperparameter of the model
we choose hyperparameters,
we fit parameters
polyn = \sum_{i=0}^N c_i x^i
N=2: a conservative hyperparameter choice
2
choose an objective function :
you need a plan to choose the parameters of the model: to "optimize" the model.
You need to choose something to be MINIMIZED or MAXIMIZED
line model: ax+b
In principle, there are many choices for objective function. But the only procedure that is truly justified—in the sense that it leads to interpretable probabilistic inference, is to make a generative model for the data.
objective function:
what you want to optimize for
objective function:
what you want to optimize for
objective function:
what you want to optimize for
objective function:
what you want to optimize for
Fit model parameters <==> minimize the Sum of residuals squared
yi: i-th observation
xi: i-th measurement "location"
e.g. Sum of residual squared (least square fit method)
SSE = \sum{(y_i - (mx_i + b) )^2}
SSE = \sum{(y_{i, observed} - y_{i, predicted} )^2}
2
choose an objective function :
you need a plan to choose the parameters of the model: to "optimize" the model.
You need to choose something to be MINIMIZED or MAXIMIZED
line model: ax+b
a line is a family of models
a line with set parameters is a models
why do we model?
2.1
why do we model?
to explain
to predict
fitting a line to data
(the longish story)
3
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
Choosing the objective function
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
\mathrm{Sum~of~errors} : \sum_{i=0}^N y_i - y_{i,predicted}
minimize something:
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
\mathrm{Sum~of~errors} : \sum_{i=0}^N y_i - y_{i,predicted}
minimize something:
minimize something:
why square?
- So that the errors do not cancel themselves out
- To add more weight to predictions that are worse
\mathrm{Sum~of~quared~errors:} ~SSE = \sum_{i=0}^N (y_i - y_{i,predicted})^2
Ordinary least square
minimize something:
def sumsqerror(y, yp):
''' objective function squared error
y: vector of observations
yp: vector of predictions
return: sum squared difference
'''
return ((y - yp) ** 2).sum()
minnow = 1e7
for s in np.arange(0, 3, 0.01):
for i in np.arange(0, 2.5, 0.01):
prediction = df['population'] * s + i
sse = sumsqerror(df.wspeed, prediction)
if sse < minnow:
minnow = sse
slope_manual, inrercept_manual = s, i
slope_manual, inrercept_manualwe can minimize manually...
\mathrm{Sum~of~quared~errors:} ~SSE = \sum_{i=0}^N (y_i - y_{i,predicted})^2
Ordinary least square
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
minimize something:
but its a lot easier to use
built in functions
# seaborn (just plots)
sns.regplot(df['population'], df['wspeed'])
# numpy
slope, intercept = np.polyfit(df['population'], df['wspeed'], 1)
# statsmodels formula
smf.ols(formula='wspeed ~ population', data=df)
# statsmodels OLS works for any degree polynomial
polynomial_features = PolynomialFeatures(degree=1)
xp1 = polynomial_features.fit_transform(x)
model = sm.OLS(y[:10], xp1[:10]).fit()
\mathrm{Sum~of~quared~errors:} ~SSE = \sum_{i=0}^N (y_i - y_{i,predicted})^2
We are trying to find the "best" line that goes through the data... but "best" is a judgment call
minimize something:
\mathrm{Sum~of~quared~errors:} ~SSE = \sum_{i=0}^N (y_i - y_{i,predicted})^2
but its a lot easier to use
built in functions
# seaborn (just plots)
import seaborn as sns
sns.regplot(df['population'], df['wspeed'])
# numpy
import numpy as np
slope, intercept = np.polyfit(df['population'], df['wspeed'], 1)
# statsmodels formula
import statsmodels.formula.api as smf
smf.ols(formula='wspeed ~ population', data=df)
# statsmodels OLS works for any degree polynomial
from sklearn.preprocessing import PolynomialFeatures
polynomial_features = PolynomialFeatures(degree=1)
xp1 = polynomial_features.fit_transform(x)
model = sm.OLS(y[:10], xp1[:10]).fit()
# sklearn
from sklearn.linear_model import LinearRegression
lm = LinearRegression().fit(X_train, y_train, sample_weight=None)how good is the model?
but its a lot easier to use
built in functions
SSE ( prediction)
variance ( mean)
but its a lot easier to use
built in functions
Adjusted R2 takes into account how many datapoints you have and how many paramters you have
how good is the model?
but its a lot easier to use
built in functions
- increasing the model complexity increases the R2 but does not guarantee a better model
- the likelihood ratio is a way to assess if the more complex model is better in a NHRT sense
but its a lot easier to use
built in functions
- increasing the model complexity increases the R2 but does not guarantee a better model
- the likelihood ratio is a way to assess if the more complex model is better in a NHRT sense
NH: the more complex model is better
under the NH the LR statistics is distributed like a chi^2 distribution with number of dof = number of extra parameters
p-value of LR in a chi^2 distribution with dof
but its a lot easier to use
built in functions
- increasing the model complexity increases the R2 but does not guarantee a better model
- the likelihood ratio is a way to assess if the more complex model is better in a NHRT sense
not all points are equal!
This function creates a “bubble” plot of Studentized residuals versus hat leverage values.
hi residual: outlier on the Y axis
high leverage : at the edge of the X axis
influence analysis
identify data points that have strong influence on the model fit: unusual x values or their y value is an "outlier". Points that are both are on the top right of the plot and are high influence points. Cook's distance is represented by the size of the bubble.
It can be shown that OLS minimization of the sum of the square errors (SSE) is equivalent to calculating the slope and intercept as:
s=\frac{\sum_{i=1}^N(x_i−\bar{X})(y_i- \bar{Y})}{\sum_{i=1}{N}(x_i−\bar{X})^2}\\
\\
b=\bar{Y}−\bar{X}
Normal equation
what is a machine learning?
ML: Any model with parameters learned from the data
ML models are a parameterized representation of "reality" where the parameters are learned from finite sets (samples) of realizations of that reality (population)
how do we model?
Choose the model:
a mathematical formula to represent the behavior in the data
1
example: line model y = a x + b
parameters
how do we model?
Choose the model:
a mathematical formula to represent the behavior in the data
1
example: line model y = a x + b
parameters
Choose the hyperparameters:
parameters chosen before the learning process, which govern the model and training process
example: the degree N of the polynomial
y = \sum^N_{i=0}c_i x^i
how do we model?
Choose an objective function:
in order to find the "best" parameters of the model: we need to "optimize" a function.
We need something to be either MINIMIZED or MAXIMIZED
2
example:
line model: y = a x + b
parameters
objective function: sum of residual squared (least square fit method)
SSE = \sum(y_{i,observed}-y_{i,predicted})^2
SSE = \sum(y_{i,observed}-(ax_i+b))^2
we want to minimize SSE as much as possible
Optimizing the Objective Function
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
Minimum (optimal) SSE
a = 4
Optimizing the Objective Function
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
How do we find the minimum if we do not know beforehand how the SSE curve looks like?
Optimizing the Objective Function
Minimum (optimal) SSE
a = 4
3.1
stochastic gradient descent (SGD)
the algorithm: Stochastic Gradient Descent (SGD)
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
-1
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
assume a simpler line model y = ax
(b = 0) so we only need to find the "best" parameter a
1. choose initial value for a
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
for a line model y = ax + b
we need to find the "best" parameters a and b
1. choose initial value for a & b
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
for a line model y = ax + b
we need to find the "best" parameters a and b
1. choose initial value for a & b
2. calculate the SSE
3. calculate best direction to go to decrease the SSE
4. step in that direction
5. go back to step 2 and repeat
the algorithm: Stochastic Gradient Descent
Things to consider:
the algorithm: Stochastic Gradient Descent
Things to consider:
- local vs. global minima
local minima
global minimum
the algorithm: Stochastic Gradient Descent
Things to consider:
- local vs. global minima
- initialization: choosing starting spot?
global minimum
local minima
the algorithm: Stochastic Gradient Descent
Things to consider:
- local vs. global minima
- initialization: choosing starting spot?
- learning rate: how far to step?
the algorithm: Stochastic Gradient Descent
Things to consider:
- local vs. global minima
- initialization: choosing starting spot?
- learning rate: how far to step?
- stopping criterion: when to stop?
the algorithm: Stochastic Gradient Descent
Things to consider:
- local vs. global minima
- initialization: choosing starting spot?
- learning rate: how far to step?
- stopping criterion: when to stop?
Stochastic Gradient Descent (SGD): use a different (random) sub-sample of the data at each iteration
epistemological rooots of overfitting
4
Okham's razor
Ockham’s razor: Pluralitas non est ponenda sine neccesitate
or “the law of parsimony”
William of Ockham (logician and Franciscan friar) 1300ca
but probably to be attributed to John Duns Scotus (1265–1308)
”Complexity needs not to be postulated without a need for it”
“Between 2 theories choose the simpler one”
Okham's razor
Ockham’s razor: Pluralitas non est ponenda sine neccesitate
or “the law of parsimony”
William of Ockham (logician and Franciscan friar) 1300ca
but probably to be attributed to John Duns Scotus (1265–1308)
”Complexity needs not to be postulated without a need for it”
“Between 2 theories choose the simpler one”
“Between 2 theories choose the one with fewer parameters"
Heliocentric model from Nicolaus Copernicus'
"De revolutionibus orbium coelestium".
Author Dr Long's copy of Cassini, 1777
Peter Apian, Cosmographia, Antwerp, 1524
Okham's razor
Heliocentric model from Nicolaus Copernicus'
"De revolutionibus orbium coelestium".
Author Dr Long's copy of Cassini, 1777
Okham's razor
Two theories may explain a phenomenon just as well as each other. In that case you should prefer the simpler one
Okham's razor
data
model fit to data
Okham's razor
model fit to data
y = ax^2 + bx + c
y = ax + b
Okham's razor
model fit to data
1 variable: x
y = ax^2 + bx + c
y = ax + b
Okham's razor
model fit to data
parameters
the complexity of a model can be measured by the number of variables and the numbers of parameters
y = ax^2 + bx + c
y = ax + b
Okham's razor
the complexity of a model can be measured by the number of variables and the numbers of parameters
mathematically: given N data points there exist an N-features model that goes exactly through each data point. but is it useul??
Overfitting: fitting data with a model that is too complex and that does not extend to new data (low predictive power on test data)
Okham's razor
data
model fit to data
Okham's razor
how to avoid overfitting in machine learning
5
Cross validation
Cross validation
test train validation
train parameters on training set
run only once on the test set to assess the model performance
Cross validation
test + train + validation
train parameters on training set
adjust parameters on validation set
run only once on the test set to assess the model performance
Cross validation
k-fold cross validation
Cross validation
NEXT WEEK:
10/5 : MIDTERM in class
what is covered in the midterm? everything we do until 9/31!
how is the midterm graded?
- data ingestion (30%)
- data exploration (descriptive statistics) (50%)
- NHRT (P-value tests) (20%)
RULES: work on your on, ALL CAMERAS MUST BE ON
The grading scheme reflects the complexity of the tasks and the expectation of how well they have been absorbed based on how long we have been working on it
key concepts
line models and polynomial models
parameters and hyperparameters
objective function: what do we minimize to choose parameters?
model diagnostics and choosing models
influence points
cross validation
references
this is a pretty comprehensive and clear medium post with coding examples
this is a whole paper about modeling in practice and in theory which covers extensively how to deal with uncertainties. it is not for the faint of heart. The notes are entertaining
reading
viewing
Foundations of Data Science for Everyone
By federica bianco
Foundations of Data Science for Everyone
Machine Learning and linear regression
