https://introml.mit.edu/

Lecture 6: Neural Networks

 

Shen Shen

Oct 4, 2024

Intro to Machine Learning

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • Design choices
  • Forward pass
  • Backward pass
    • Back-propagation
Recap:

leveraging  nonlinear transformations

👆

importantly, linear in \(\phi\), non-linear in \(x\)

\phi\left(\left[x_1; x_2\right]\right)=\left[1;|x_1-x_2|\right]

transform via

​Pointed out key ideas (enabling neural networks):

  • ​Nonlinear feature transformation
  • "Composing" simple transformations
  • Backpropagation 
\}

expressiveness

efficient training

\sigma_1 = \sigma(5 x_1 -5 x_2 + 1)
\sigma_2 = \sigma(-5 x_1 + 5 x_2 + 1)

Two epiphanies:  

  • nonlinear transformation empowers linear tools
  • "composing" simple nonlinearities amplifies such effect

some appropriate weighted sum

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • Design choices
  • Forward pass
  • Backward pass
    • Back-propagation

 

👋 heads-up, in this section, for simplicity:

all neural network diagrams focus on a single data point

A neuron:

\(w\): what the algorithm learns

  • \(x\): \(d\)-dimensional input
a = f(z)
\Sigma

A neuron:

  • \(a\): post-activation output
  • \(f\): activation function
  • \(w\): weights (i.e. parameters)
  • \(z\): pre-activation output

\(f\): what we engineers choose

\dots
x_1
x_2
x_d
x = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
w_1
w_d
\dots
w_2
= w^Tx
z
f(\cdot)
= f(w^Tx)

\(z\): scalar

\(a\): scalar

Choose activation \(f(z)=z\)

learnable parameters (weights)

e.g. linear regressor represented as a computation graph

= z
= w^Tx
w_1
w_d
\dots
x_1
x_2
x_d
\dots
x = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
w_2
\Sigma
f(\cdot)
g
z

Choose activation \(f(z)=\sigma(z)\)

learnable parameters (weights)

e.g. linear logistic classifier represented as a computation graph

= \sigma(z)
= w^Tx
w_1
w_d
\dots
x_1
x_2
x_d
\dots
w_2
\Sigma
f(\cdot)
g
z
x = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
\dots

A layer:

learnable weights

A layer:

a^1
z^1
\Sigma
f(\cdot)
z^2
\Sigma
f(\cdot)
a^2
z^m
a^m
\Sigma
f(\cdot)
\dots
x_1
x_2
x_d
  • (# of neurons) = (layer's output dimension).
  • typically, all neurons in one layer use the same activation \(f\) (if not, uglier algebra).
  • typically fully connected, where all \(x_i\) are connected to all \(z_j,\) meaning each \(x_i\) influences every \(a_j\) eventually.
  • typically, no "cross-wiring", meaning e.g. \(z_1\) won't affect \(a^2.\) (the final layer may be an exception if softmax is used.)
\dots

layer

linear combo

activations

A (fully-connected, feed-forward) neural network: 

\dots
\dots

layer

\dots
x_1
x_2
x_d

input

\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)

neuron

learnable weights

We choose:

  • activation \(f\) in each layer
  • # of layers
  • # of neurons in each layer

hidden

output

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • ​Design choices
  • ​Forward pass
  • Backward pass
    • Back-propagation
\sigma_1 = \sigma(5 x_1 + -5 x_2 + 1)
\sigma_2 = \sigma(-5 x_1 + 5 x_2 + 1)

some appropriate weighted sum

recall this example

x_1
x_2
1
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)

\(f(\cdot) = \sigma(\cdot)\) 

\(f(\cdot) \) identity function

it can be represented as

Activation function \(f\) choices

\(\sigma\) used to be the most popular

  • firing rate of a neuron
  • elegant gradient \(\sigma^{\prime}(z)=\sigma(z) \cdot(1-\sigma(z))\)
a = f(z)
\Sigma
\dots
x_1
x_2
x_d
x = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
w_1
w_d
\dots
w_2
= w^Tx
z
f(\cdot)
= f(w^Tx)
  • default choice in hidden layers
  • very simple function form, so is the gradient.

 

 

 

 

\operatorname{ReLU}(z)=\left\{\begin{array}{ll} 0 & \text { if } z<0 \\ z & \text { otherwise } \end{array}\\ \\ \right.
=\max (0, z)

nowadays

  • drawback: if strongly in negative region, a single ReLU can be "dead" (no gradient).
  • Luckily, typically we have lots of units, so not everyone is dead. 
\frac{\partial \text{ReLU}(z)}{\partial z}:=\left\{\begin{array}{lll} 0, & \text { if } \quad z<0 \\ 1, & \text { if } \quad \text{otherwise} \end{array}\right.

compositions of ReLU(s) can be quite expressive

in fact, asymptotically, can approximate any function!

(image credit: Phillip Isola)

x_1
x_2

(image credit: Tamara Broderick)

or give arbitrary decision boundaries!

+
=

(image credit: Tamara Broderick)

+
=

output layer design choices

  • # neurons, activation, and loss depend on the high-level goal. 
  • typically straightforward.
  • Multi-class setup: if predict one and only one class out of \(K\) possibilities, then last layer: \(K\) neurons, softmax activation, cross-entropy loss

 

 

 

 

  • other multi-class settings, see discussion in lab.

e.g., say \(K=5\) classes

input \(x\)

hidden layer(s)

\dots

output layer

  • Width: # of neurons in layers 
  • Depth: # of layers
  • More expressive if increasing either the width or depth.

 

  • The usual pitfall of overfitting (though in NN-land, it's also an active research topic.)

(The demo won't embed in PDF. But the direct link below works.)

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • ​Design choices
  • ​Forward pass
  • Backward pass
    • Back-propagation
  • Evaluate the loss \(\mathcal{L} = (g-y)^2\)
  • Repeat for each data point, average the sum of \(n\) individual losses

e.g. forward-pass of a linear regressor

y^{(i)}
f(\cdot)
\mathcal{L}(g, y)
\mathcal{L}(g, y)
\underbrace{\quad \quad \quad } \\ n
\mathcal{L}(g, y)
\dots
= w^Tx
w_1
w_d
\dots
w_2
\Sigma
z
= z
g
\dots
x_1
x_2
x_d
x^{(i)} \quad = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
\dots
\dots
  • Evaluate the loss \(\mathcal{L} = - [y \log g+\left(1-y\right) \log \left(1-g\right)]\)
  • Repeat for each data point, average the sum of \(n\) individual losses
f(\cdot)
\mathcal{L}(g, y)
\mathcal{L}(g, y)
\underbrace{\quad \quad \quad } \\ n
\mathcal{L}(g, y)
\dots
x_1
x_2
x_d
x^{(i)} \quad = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
= w^Tx
w_1
w_d
\dots
w_2
\Sigma
z
= \sigma(z)
g

e.g. forward-pass of a linear logistic classifier

\dots
y^{(i)}
\dots
\dots
x^{(1)}
y^{(1)}
f^1

linear combination

nonlinear activation

\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
g^{(1)}

\(\dots\)

f^2\left(\hspace{2cm}; \mathbf{W}^2\right)
f^1(\mathbf{x}^{(i)}; \mathbf{W}^1)
f^L\left(\dots \hspace{3.5cm}; \dots \mathbf{W}^L\right)

Forward pass:

evaluate, given the current parameters,

  • the model output \(g^{(i)}\) =  
  • the loss incurred on the current data \(\mathcal{L}(g^{(i)}, y^{(i)})\)
  • the training error \(J = \frac{1}{n} \sum_{i=1}^{n}\mathcal{L}(g^{(i)}, y^{(i)})\)
\mathcal{L}(g^{(1)}, y^{(1)})
\mathcal{L}(g, y)
\mathcal{L}(g^{(n)}, y^{(n)})
\underbrace{\quad \quad \quad \quad \quad } \\ n
\dots
\dots
\dots

loss function

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • Design choices
  • Forward pass
  • Backward pass
    • Back-propagation
  • Randomly pick a data point \((x^{(i)}, y^{(i)})\)
  • Evaluate the gradient \(\nabla_{W^2} \mathcal{L(g^{(i)},y^{(i)})}\) 
  • Update the weights \(W^2 \leftarrow W^2 - \eta \nabla_{W^2} \mathcal{L(g^{(i)},y^{(i)}})\) 
x^{(i)}
y^{(i)}
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
g^{(i)}

\(\dots\)

\mathcal{L}(g^{(i)}, y^{(i)})
\mathcal{L}(g, y)
\mathcal{L}(g^{(n)}, y^{(n)})
\underbrace{\quad \quad \quad \quad \quad } \\ n
\dots
\dots
\dots

Backward pass:

Run SGD to update the parameters, e.g. to update \(W^2\)

\(\nabla_{W^2} \mathcal{L(g^{(i)},y^{(i)})}\)

x

\(\dots\)

y
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g

\(\nabla_{W^2} \mathcal{L(g,y)}\)

Backward pass:

Run SGD to update the parameters, e.g. to update \(W^2\)

Evaluate the gradient \(\nabla_{W^2} \mathcal{L(g^{(i)},y^{(i)})}\) 

Update the weights \(W^2 \leftarrow W^2 - \eta \nabla_{W^2} \mathcal{L(g^{(i)},y^{(i)}})\) 

x
y
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\mathcal{L}(g,y)
g

How do we get these gradient though?

\(\nabla_{W^1} \mathcal{L(g,y)}\)

Backward pass:

Run SGD to update the parameters, e.g. to update \(W^1\)

Evaluate the gradient \(\nabla_{W^1} \mathcal{L(g^{(i)},y^{(i)})}\) 

Update the weights \(W^1 \leftarrow W^1 - \eta \nabla_{W^1} \mathcal{L(g^{(i)},y^{(i)}})\) 

\(\dots\)

\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L

Outline

  • Recap, the leap from simple linear models
  • (Feedforward) Neural Networks Structure
    • Design choices
  • Forward pass
  • Backward pass
    • Back-propagation
\mathcal{L}(g, y)
\mathcal{L}(g, y)
\underbrace{\quad \quad \quad } \\ n
\mathcal{L}(g, y)
\dots
x_1
x_2
x_d
x^{(i)} \quad = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]

e.g. backward-pass of a linear regressor

\dots
y^{(i)}
\dots
\dots
  • Randomly pick a data point \((x^{(i)}, y^{(i)})\)
  • Evaluate the gradient \(\nabla_{w} \mathcal{L(g^{(i)},y^{(i)})}\) 
  • Update the weights \(w \leftarrow w - \eta \nabla_w \mathcal{L(g^{(i)},y^{(i)}})\) 
w
\left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
= w^Tx
\Sigma
w_1
w_2
\dots
w_d
w_1
w_2
\dots
w_d
=
\nabla_{w} \mathcal{L(g^{(i)},y^{(i)})}
g
\mathcal{L}(g, y)
\dots
x_1
x_2
x_d
x \quad = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]

e.g. backward-pass of a linear regressor

y
\left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
\Sigma
w_1
w_2
\dots
w_d
y \in \mathbb{R}
x \in \mathbb{R^d}
w \in \mathbb{R^d}
= \frac{\partial \mathcal{L}(g,y)}{\partial w}
\nabla_{w} \mathcal{L(g,y)}
= x \cdot 2(g - y)
\frac{\partial g}{\partial w}
= \frac{\partial[(g - y)^2] }{\partial w}
= \frac{\partial[(w^T x - y)^2] }{\partial w}
\nabla_{w} \mathcal{L(g,y)}
\frac{\partial \mathcal{L}}{\partial g}
w
=
\nabla_{w} \mathcal{L(g,y)}
= w^Tx
g
\text{ReLU}
\mathcal{L}(g, y)
\dots
x_1
x_2
x_d
x \quad = \left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]

e.g. backward-pass of a non-linear regressor

y
\left[ \begin{array}{c} \\ \\ \\ \\ \\ \\ \\ \end{array} \right]
= w^Tx
\Sigma
z
w_1
w_2
\dots
w_d
y \in \mathbb{R}
x \in \mathbb{R^d}
w \in \mathbb{R^d}
= \frac{\partial \mathcal{L}(g,y)}{\partial w}
\nabla_{w} \mathcal{L(g,y)}
= x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)
\frac{\partial g}{\partial z}
\frac{\partial z}{\partial w}
= \frac{\partial[(g - y)^2] }{\partial w}
\nabla_{w} \mathcal{L(g,y)}
\frac{\partial \mathcal{L}}{\partial g}
w
=
\nabla_{w} \mathcal{L(g,y)}
g
= \text{ReLU}(z)
x

\(\dots\)

y
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g
\underbrace{\hspace{4.7cm}}

Now, back propagation: reuse of computation

Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^4}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial Z^2}{\partial W^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}

how to find

?

\frac{\partial \mathcal{L}(g,y)}{\partial W^1}
x

\(\dots\)

y
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^4}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}

back propagation: reuse of computation

\frac{\partial Z^2}{\partial W^{2}}
\underbrace{\hspace{4.7cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}

how to find

?

\underbrace{\hspace{6.5cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^1}
\frac{\partial Z^2}{\partial A^{1}}
\frac{\partial A^1}{\partial Z^{1}}
\frac{\partial Z^1}{\partial W^{1}}
x

\(\dots\)

y
f^1
\begin{aligned} & W^1 \\ \end{aligned}
\begin{aligned} & W^2 \\ \end{aligned}
\begin{aligned} & W^L \\ \end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^4}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}

back propagation: reuse of computation

how to find

\frac{\partial \mathcal{L}(g,y)}{\partial W^1}

?

Summary

  • We saw that introducing non-linear transformations of the inputs can substantially increase the power of linear tools. But it’s kind of difficult/tedious to select a good transformation by hand.
  • Multi-layer neural networks are a way to automatically find good transformations for us!
  • Standard NNs have layers that alternate between parametrized linear transformations and fixed non-linear transforms (but many other designs are possible.)
  • Typical non-linearities include sigmoid, tanh, relu, but mostly people use relu.
  • Typical output transformations for classification are as we've seen: sigmoid, or softmax.
  • There’s a systematic way to compute gradients via back-propagation, in order to update parameters.

Thanks!

We'd love to hear your thoughts.