federica bianco PRO
astro | data science | data for good
Machine Learning for Time Series Analysis II
Spring 2025 - UDel PHYS 664
dr. federica bianco
@fedhere
this slide deck:
MLTSA:
class tools
python github google-colab stackoverflow
1
github
Reproducible research means:
all numbers in a data analysis can be recalculated exactly (down to stochastic variables!) using the code and raw data provided by the analyst.
Claerbout, J. 1990,
Active Documents and Reproducible Results, Stanford Exploration Project Report, 67, 139
reproducibility
allows reproducibility through code distribution
github
the Git software
is a distributed version control system:
a version of the files on your local computer is made also available at a central server.
The history of the files is saved remotely so that any version (that was checked in) is retrievable.
version control
allows version control
github
collaboration tool
by fork, fork and pull request, or by working directly as a collaborator
collaborative platform
allows effective collaboration
python
- intuitive and readable
- open source
- support C integration for performance
- packages designed for science:
- scipy
- statsmodels
- numpy (computation)
- sklearn (machine learning)
python
series of notebooks designed for Urban Science students by Dr. Mohit Sharma (in consultation with me)
recommanded if you are brand new to python and coding or are serious about cleaning up your foundamentals
python
quick bootcamp
recommanded if you know some python or if you know some other conding language reasonably proficiently
python
online book
python
PEP8: Python Enhancement Proposals 8
“This document gives coding conventions for the Python code comprising the standard library in the main Python distribution.”
Jupyter Notebook Google Colaboratory
stackoverflow
for when you need help
you can ask coding questions, installation questions, colab questions...
stackoverflow
for when you need help
you can ask coding questions, installation questions, colab questions...
stackoverflow
for when you need help
you can ask ciding questions, installation questions, colab questions...
it can be a toxic environment...
MLTSA:
Scientific method
Science Guiding Principles
2
epistemology:
the philosophy of science and of the scientific method
My proposal is based upon an asymmetry between verifiability and falsifiability; an asymmetry which results from the logical form of universal statements. For these are never derivable from singular statements, but can be contradicted by singular statements.
—Karl Popper, The Logic of Scientific Discovery
the demarcation problem:
what is science? what is not?
a scientific theory must be falsifiable
My proposal is based upon an asymmetry between verifiability and falsifiability; an asymmetry which results from the logical form of universal statements. For these are never derivable from singular statements, but can be contradicted by singular statements.
—Karl Popper, The Logic of Scientific Discovery
the demarcation problem:
what is science? what is not?
model
prediction
the demarcation problem
Einstein GR
the demarcation problem
model
prediction
Light rays are deflected by mass
model
prediction
data
does not falsify
falsifies
GR
still holds
GR
rejected
the demarcation problem
position of star changes during eclipse
position of star does not change during eclipse
is astrology a science?
the demarcation problem
DISCUSS!
the demarcation problem
things can get more complicated though:
most scientific theories are actually based largely on probabilistic induction and
modern inductive inference (Solomonoff, frequentist vs Bayesian methods...)
the demarcation problem
A theory can be said to be scientific if it makes falsifiable predictions
Experiments should be designed to falsify the predictions
Key Concept
Reproducibility
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
Reproducibility
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
why?
assures a result is grounded in evidence
1
#openscience
#opendata
Reproducibility
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
why?
facilitates scientific progress by avoiding the need to duplicate unoriginal research
2
Reproducibility
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
why?
facilitate collaboration and teamwork
3
Reproducible research in practice:
using the code and raw data provided by the analyst.
Claerbout, J. 1990,
Active Documents and Reproducible Results, Stanford Exploration Project Report, 67, 139
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
Reproducibility
all numbers in a data analysis can be recalculated exactly (down to stochastic variables!)
Reproducible research means:
the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator. That is, a second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results.
- provide raw data and code to reduce it to all stages needed to get outputs
- provide code to reproduce all figures
- provide code to reproduce all number outcomes
Reproducible research in practice:
using the code and raw data provided by the analyst.
all numbers in a data analysis can be recalculated exactly (down to stochastic variables!)
Reproducibility
Reproducibility
A research product is reproducible if all numbers can be reproduced exactly be applying the same code to the same raw data.
It is the responsibility of the researcher to provide the data and code that make a research product reproducible
Key Concept
MLTSA:
Linear Regression
3
Linear Regression
WHY?
Fitting a line
ax+b
to data y
WHY?
Fitting a line
ax+b
to data y
To predict and forecast
time (year)
See level contribution (mm)
Linear Regression
To explain
distance / age of the Universe
Universe's expansion rate
supernova (stellar explosion)
measure the expansion rate at the Universe as a function of time.
Deviation from linear falsify an adiabatically expanding Universe
time (year)
WHY?
Fitting a line
ax+b
to data y
To predict and forecast
See level contribution (mm)
Linear Regression
Key Concept
Model Fitting
We fit models to data in order to:
Predict and forecast: predict the value of the endogenous (dependent) variable at locations of the exogenous (independent, time) variable where we have no observations. This can be within the observed range, or outside of the range, which in time-series means predict the future (forecast)
Explain: relate observed behavior to first principles or behavior of possibly variables to explain the evolution and assess causality.
E.g. fitting a parabola to a bouncing ball demonstrates that gravity (and initial velocity) explains the behavior
MLTSA:
Linear Regression
analytical solution
3.0
Linear Regression
Normal Equation
It can be shown that the optimal parameters for a line fit to data without uncertainties is:
(X^T \cdot X)^{-1} \cdot X^T \cdot \vec{y} ~=~\left(\substack{a\\b}\right)
X = np.c_[np.ones((len(grbAG) - grbAG.upperlimit.sum(), 1)),
grbAG[grbAG.upperlimit == 0].logtime]
y = grbAG.loc[grbAG.upperlimit == 0].mag
theta_best = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)Linear Regression
Normal Equation
It can be shown that the optimal parameters for a line fit to data without uncertainties is:
(X^T \cdot X)^{-1} \cdot X^T \cdot \vec{y} ~=~\left(\substack{a\\b}\right)
x = \begin{pmatrix}
1 & x_{1} \\
1 & x_{2} \\
\vdots & \vdots \\
1 & x_{m}
\end{pmatrix}
2xN Nx2 2xN Nx1
X = np.c_[np.ones((len(grbAG) - grbAG.upperlimit.sum(), 1)),
grbAG[grbAG.upperlimit == 0].logtime]
y = grbAG.loc[grbAG.upperlimit == 0].mag
theta_best = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)It can be shown that the optimal parameters for a line fit to data without uncertainties is:
from sklearn.linear_model import LinearRegression
lr = LinearRegression()
X = np.c_[np.ones((len(grbAG) -
grbAG.upperlimit.sum(), 1)),
grbAG[grbAG.upperlimit == 0].logtime]
y = grbAG.loc[grbAG.upperlimit == 0].mag
lr.fit(X, y)
lr.coef_, lr.intercept_We can let sklearn solve the equation for us:
(X^T \cdot X)^{-1} \cdot X^T \cdot \vec{y} ~=~\left(\substack{a\\b}\right)
2x1
2xN Nx2 2xN Nx1
Linear Regression
Normal Equation
X = np.c_[np.ones((len(grbAG) - grbAG.upperlimit.sum(), 1)),
grbAG[grbAG.upperlimit == 0].logtime]
y = grbAG.loc[grbAG.upperlimit == 0].mag
theta_best = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)MLTSA:
Linear Regression
linear correlation
3.1
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
correlated
"positively" correlated
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
correlated
"positively" correlated
r_{xy} = 1~\mathrm{iff}~y=ax\\
~\mathrm{maximally~correlated}
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
anticorrelated
"negatively" correlated
correlation
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
anticorrelated
"negatively" correlated
r_{xy} = 1~\mathrm{iff}~y=-ax\\
~\mathrm{maximally~anticorrelated}
correlation
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Pearson's correlation measures linear correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
not linearly correlated
Pearson's coefficient = 0
does not mean that x and y are independent!
\rho_{xy} = 1-\frac{6\sum_{i=1}^N(x_i - y_i)^2}{n(n^2-1)}
Pearson's correlation
Spearman's test
(Pearson's for ranked values)
correlation
\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
Correlation does not imply causality!!
2 things may be related because they share a cause but not cause each other:
icecream sales with temperature |death by drowning
with temperature
In the era of big data you may encounter truly spurious correlations
divorce rate in Maine | consumption of Margarine
correlation
correlation
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
import pandas as pd
df = pd.read_csv(file_name)
df.corr()\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
correlation
import pandas as pd
df = pd.read_csv(file_name)
df.corr()
pl.imshow(vdf.corr(), clim=(-1,1), cmap='RdBu')
pl.xticks(list(range(len(df.corr()))),
df.columns, rotation=45)
pl.yticks(list(range(len(df.corr()))),
df.columns, rotation=45)
pl.colorbar();
<- anticorrelated | correlated ->
correlation
Pearson's correlation
r_{xy} = \frac{1}{n-1}\sum_{i=1}^N\left(\frac{x_i - \bar{x}}{s_x}\right)\left(\frac{y_i - \bar{y}}{s_y}\right)
import pandas as pd
df = pd.read_csv(file_name)
df.corr()\bar{x} : \mathrm{mean~value~of~}x\\
\bar{y} : \mathrm{mean~value~of~}y\\
n: \mathrm{number~of~datapoints}\\
s_x ~=~\sqrt{\frac{1}{n-1}\sum_{i=1}^N(x_i - \bar{x})^2}
correlation
import pandas as pd
df = pd.read_csv(file_name)
df.corr()pl.imshow(vdf.corr(), clim=(-1,1), cmap='RdBu')
pl.xticks(list(range(len(df.corr()))),
df.columns, rotation=45)
pl.yticks(list(range(len(df.corr()))),
df.columns, rotation=45)
pl.colorbar();
MLTSA:
Regression
objective function
3.2
If there is no analytical solution
to select the "best" set of parameters we need a plan: we need to choose a function of the parameters to minimize or maximize
Objective Function
If there is no analytical solution
Objective Function
time
time
time
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
which is the "best fit" line? A , B, C, D?
A
B
C
D
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
which is the "best fit" line? A , B, C, D?
A
B
C
D
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
which is the "best fit" line? A , B, C, D?
A
B
C
D
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
which is the "best fit" line? A , B, C, D?
A
B
C
D
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
which is the "best fit" line? A , B, C, D?
A
B
C
D
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
time
time
time
L_1 = \sum_{i=1}^N|f(x) - y|
L_2 = \sum_{i=1}^N(f(x) - y)^2
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
Objective Function
L_1 = \sum_{i=1}^N|f(x) - y|
L_2 = \sum_{i=1}^N(f(x) - y)^2
\chi^2 = \sum_{i=1}^N\frac{(f(x) - y)^2}{\sigma^2}
chi square: relates to the likelihood if the distribution is Gaussian
to select the best fit parameters we define a function of the parameters to minimize or maximize
If there is no analytical solution
to select the "best" set of parameters we need a plan: we need to choose a function of the parameters to minimize or maximize
Objective Function
L_1 = \sum_{i=1}^N|f(x) - y|
L_2 = \sum_{i=1}^N(f(x) - y)^2
\chi^2 = \sum_{i=1}^N\frac{(f(x) - y)^2}{\sigma^2}
from scipy.optimize import minimize
def line(x, b, a):
return a * x + b
def fitfunc(args, x, y):
a, b = args
return sum((y - line(a, b, x))**2)
x = grbAG.logtime.values
y = grbAG.mag.values
initialGuess = (10, 1)
fitfunc(initialGuess, x, y)
solution = minimize(fitfunc, initialGuess, args=(x, y))If there is no analytical solution
to select the "best" set of parameters we need a plan: we need to choose a function of the parameters to minimize or maximize
Objective Function
L_1 = \sum_{i=1}^N|f(x) - y|
L_2 = \sum_{i=1}^N(f(x) - y)^2
\chi^2 = \sum_{i=1}^N\frac{(f(x) - y)^2}{\sigma^2}
from scipy.optimize import minimize
def line(x, b, a):
return a * x + b
def chi2(args, x, y, s):
a, b = args
return sum((y - line(x, b, a))**2 / s)
x = grbAG.logtime.values
y = grbAG.mag.values
s = grbAG.magerr.values
initialGuess = (10, 1)
fitfunc(initialGuess, x, y)
solution = minimize(chi2, initialGuess, args=(x, y, s))
solution
viz of the week
W.E.B. DuBois
a few historical plots and why they made history
Florence Nightingale Coxcombs
Diagram of the causes of mortality in the army in the East,
a few historical plots and why they made history
Florence Nightingale Coxcombs
Key Concepts
Reproduciblity: A research product is reproducible if all numbers can be reproduced exactly be applying the same code to the same raw data. It is the responsibility of the researcher to provide the data and code that make a research product reproducible
What is Machine Learning? Machine Learning models are parametrized representations of "reality" where the parameters are learned from finite sets of realizations of that reality. Machine Learning is the discipline that conceptualizes, studies, and applies those models.
Model selection: Choosing a model i.e. a mathematical formula which we expect to be a simplified representation of our observations.
Objective Functions and optimization: To find the best model parameters we define a function of the data and parameters f(data, parameters) to be minimized or maximized.
Model fitting: Determining the best set of parameters to fit the observations within a chosen model.
Linear Regression as your ML first step: fitting a linear model (line or polynomial) to data is an approachable data analysis method that reveals _trends_ .
References
Data analysis recipes: Fitting a model to data
D. Hogg et al. https://arxiv.org/abs/1008.4686 - lots of details about how to properly treat outliers, uncertainties, assumptions in fitting a line to data. Witty comments make it entertaining. Exercise it make it very helpful
AstroML Chapter 10 - Intro
HOMLwSKLKerasTF Chapter 4 pages 111-117
Elements of Statistical Learning Chapter 3 Section 1 and 2
Additional
Reading
Data analysis recipes: Fitting a model to data
Intro and Chapter 1; pages 1-8
D. Hogg et al. https://arxiv.org/abs/1008.4686
Lots of details about how to properly treat outliers, uncertainties, assumptions in fitting a line to data. Witty comments make it entertaining. Exercise it make it very helpful
Key Concepts
Falisifiability: A theory can be said to be scientific if it makes falsifiable predictions. Experiments should be designed to falsify the predictions
Reproduciblity: A research product is reproducible if all numbers can be reproduced exactly be applying the same code to the same raw data. It is the responsibility of the researcher to provide the data and code that make a research product reproducible
What is special about time series? Time series are series of exogenous-endogenous variable pairs where the exogenous variable is time, and therefore it is a sequential quantity with a specific direction of evolution.
What is Machine Learning? Machine Learning models are parametrized representations of "reality" where the parameters are learned from finite sets of realizations of that reality. Machine Learning is the discipline that conceptualizes, studies, and applies those models.
Objective Functions and optimization: To find the best model parameters we define a function of the data and parameters f(data, parameters) to be minimized or maximized.
Model fitting: Determining the best set of parameters to fit the observations within a chosen model.
References
Data analysis recipes: Fitting a model to data
D. Hogg et al. https://arxiv.org/abs/1008.4686 - lots of details about how to properly treat outliers, uncertainties, assumptions in fitting a line to data. Witty comments make it entertaining. Exercise it make it very helpful
AstroML Chapter 10 - Intro
HOMLwSKLKerasTF Chapter 4 pages 111-117
Elements of Statistical Learning Chapter 3 Section 1 and 2
MLTSA_02 2025
By federica bianco
MLTSA_02 2025
intro to time series and regression
