Lecture 6: Neural Networks II
Intro to Machine Learning
\dots
layer
linear combo
activations
\dots
\dots
layer
\dots
x_1
x_2
x_d
input
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
neuron
learnable weights
Engineers choose:
- activation \(f\) in each layer
- # of layers
- # of neurons in each layer
hidden
output
Recall:
(aka, multi-layer perceptrons)
a (fully-connected, feed-forward) neural network
x^{(1)}
y^{(1)}
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^2
f^L
g^{(1)}
\(\dots\)
\phantom{g^{(i)} = f^L\big(\dots\, }f^2\big(\phantom{f^1(\mathbf{x}^{(i)}; \mathbf{W}^1)}; \mathbf{W}^2\big)\phantom{; \dots\, \mathbf{W}^L\big)}
\phantom{g^{(i)} = f^L\big(\dots\, f^2\big(}f^1(\mathbf{x}^{(i)}; \mathbf{W}^1)\phantom{; \mathbf{W}^2\big); \dots\, \mathbf{W}^L\big)}
\phantom{g^{(i)} = }f^L\big(\dots\, \phantom{f^2\big(f^1(\mathbf{x}^{(i)}; \mathbf{W}^1); \mathbf{W}^2\big)}; \dots\, \mathbf{W}^L\big)
g^{(i)} = \phantom{f^L\big(\dots\, f^2\big(f^1(\mathbf{x}^{(i)}; \mathbf{W}^1); \mathbf{W}^2\big); \dots\, \mathbf{W}^L\big)}
Forward pass: evaluate, given the current parameters
- the model outputs
- the loss incurred on the current data \(\mathcal{L}(g^{(i)}, y^{(i)})\)
- the training error \(J = \frac{1}{n} \sum_{i=1}^{n}\mathcal{L}(g^{(i)}, y^{(i)})\)
\mathcal{L}(g^{(1)}, y^{(1)})
\mathcal{L}(g^{(2)}, y^{(2)})
\mathcal{L}(g^{(n)}, y^{(n)})
\underbrace{\quad \quad \quad \quad \quad }
\dots
\dots
\dots
n
linear combination
loss function
(nonlinear) activation
Recall:
x_1
x_2
1
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\Sigma
f(\cdot)
\(f =\sigma(\cdot)\)
\(f(\cdot) \) identity function
Recall:
W^1
W^2
\sigma_1 = \sigma(5 x_1 -5 x_2 + 1)
\sigma_2 = \sigma(-5 x_1 + 5 x_2 + 1)
5
-5
1
-5
5
1
-3
-3
\(-3(\sigma_1 +\sigma_2)\)
(The demo won't embed in PDF. The direct link below gets to the demo.)
Outline
-
Backward pass (to learn parameters/weights)
- Backpropagation (gradient descent & the chain rule)
- Recursive reuse of computation
- Practical gradient issues and remedies
\mathcal{L}(g, y)
\mathcal{L}(g, y)
\underbrace{\quad \quad \quad } \\ n
\mathcal{L}(g, y)
\dots
x_1
x_2
x_d
x^{(i)} \quad = \left[ \begin{array}{c}
\\
\\
\\
\\
\\
\\
\\
\end{array} \right]
stochastic gradient descent to learn a linear regressor
\dots
y^{(i)}
\dots
\dots
- Randomly pick a data point \((x^{(i)}, y^{(i)})\)
- Evaluate the gradient \(\nabla_{w} \mathcal{L}(g^{(i)},y^{(i)})\)
- Update the weights \(w \leftarrow w - \eta \nabla_w \mathcal{L}(g^{(i)},y^{(i)})\)
= w^Tx
\Sigma
\nabla_{w} \mathcal{L}(g^{(i)},y^{(i)})
g
w
=
\left[ \begin{array}{c}
\\
\\
\\
\\
\\
\\
\\
\end{array} \right]
w_1
w_2
\dots
w_d
w_1
w_2
\dots
w_d
for simplicity, say training data set is just\((x,y)\) and squared loss
y
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)}} = \frac{\partial \mathcal{L}(g,y)}{\partial w} \phantom{= \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g} = \frac{\partial (wx)}{\partial w} \cdot 2(g - y) = x \cdot 2(g - y)}
\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} \phantom{= \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g} = \frac{\partial (wx)}{\partial w} \cdot 2(g - y) = x \cdot 2(g - y)}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g} = \frac{\partial (wx)}{\partial w} \cdot 2(g - y)} = x \cdot 2(g - y)
\frac{\partial g}{\partial w}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w}} = \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g} \phantom{= \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g} = \frac{\partial (wx)}{\partial w} \cdot 2(g - y) = x \cdot 2(g - y)}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g}} = \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g} \phantom{= \frac{\partial (wx)}{\partial w} \cdot 2(g - y) = x \cdot 2(g - y)}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial g}{\partial w} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial g}{\partial w} \cdot \frac{\partial (g - y)^2}{\partial g}} = \frac{\partial (wx)}{\partial w} \cdot 2(g - y) \phantom{= x \cdot 2(g - y)}
\nabla_{w} \mathcal{L}(g,y)
\frac{\partial \mathcal{L}}{\partial g}
\nabla_{w} \mathcal{L}(g,y)
g
walk-through on board
\mathcal{L}(g, y)
= wx
\Sigma
x
w
\begin{array}{r}\text{1-d example:}\\ x = 3,\ y = 6,\ w = 4\end{array}
\text{ReLU}
Slightly more interesting:
y
= wx
\Sigma
z
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)}} = \frac{\partial \mathcal{L}(g,y)}{\partial w} \phantom{= \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial (g - y)^2}{\partial g} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)}
\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} \phantom{= \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial (g - y)^2}{\partial g} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial \mathcal{L}}{\partial g} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial (g - y)^2}{\partial g}} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)
\frac{\partial g}{\partial z}
\frac{\partial z}{\partial w}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w}} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial \mathcal{L}}{\partial g} \phantom{= \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial (g - y)^2}{\partial g} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = \frac{\partial \mathcal{L}(g,y)}{\partial w} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial \mathcal{L}}{\partial g}} = \frac{\partial z}{\partial w} \cdot \frac{\partial g}{\partial z} \cdot \frac{\partial (g - y)^2}{\partial g} \phantom{= x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)}
\nabla_{w} \mathcal{L}(g,y)
\frac{\partial \mathcal{L}}{\partial g}
\nabla_{w} \mathcal{L}(g,y)
g
= \text{ReLU}(z)
\mathcal{L}(g, y)
walk-through on board
x
w
\begin{array}{r}\text{1-d example:}\\ x = 3,\ y = 6,\ w = 4\end{array}
\text{ReLU}
Slightly more interesting:
y
= wx
\Sigma
z
\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} \phantom{= x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y) =\left\{\begin{array}{ll}
0, & \text{if } z < 0 \\
2\,x\,(g - y), & \text{otherwise}
\end{array}\right.}
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)}} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y) \phantom{=\left\{\begin{array}{ll}
0, & \text{if } z < 0 \\
2\,x\,(g - y), & \text{otherwise}
\end{array}\right.}
\frac{\partial g}{\partial z}
\frac{\partial z}{\partial w}
\nabla_{w} \mathcal{L}(g,y)
\frac{\partial \mathcal{L}}{\partial g}
\nabla_{w} \mathcal{L}(g,y)
g
\mathcal{L}(g, y)
walk-through on board
=\left\{\begin{array}{lll}
0, & \text {if } z<0 \\
1, & \text{otherwise}
\end{array}\right.
\phantom{\textcolor{#93c47d}{\nabla_{w} \mathcal{L}(g,y)} = x \cdot \frac{\partial[(\text{ReLU}(z))] }{\partial z} \cdot 2(g - y)} =\left\{\begin{array}{ll}
0, & \text{if } z < 0 \\
2\,x\,(g - y), & \text{otherwise}
\end{array}\right.
= \text{ReLU}(z)
x
w
\begin{array}{r}\text{1-d example:}\\ x = 3,\ y = 6,\ w = 4\end{array}
The middlemen
one partial derivative per hop,
traced backwards along the path, from the loss to the variable of interest
\text{ReLU}
y
= wx
\Sigma
z
\frac{\partial g}{\partial z}
\frac{\partial z}{\partial w}
\nabla_{w} \mathcal{L}(g,y)
\frac{\partial \mathcal{L}}{\partial g}
g
\mathcal{L}(g, y)
= \text{ReLU}(z)
“\(z\) tells \(w\) how to change”
\(\dfrac{\partial z}{\partial w} = x = 3\)
“the guess tells \(z\) how to change”
\(\dfrac{\partial g}{\partial z} = \text{ReLU}'(z) = 1\)
“the loss tells the guess how to change”
\(\dfrac{\partial \mathcal{L}}{\partial g} = 2(g - y) = 12\)
x
w
Outline
-
Backward pass (to learn parameters/weights)
- Backpropagation (gradient descent & the chain rule)
- Recursive reuse of computation
- Practical gradient issues and remedies
- Randomly pick a data point \((x^{(i)}, y^{(i)})\)
- Evaluate the gradient \(\nabla_{W^2} \mathcal{L}(g^{(i)},y^{(i)})\)
- Update the weights \(W^2 \leftarrow W^2 - \eta \nabla_{W^2} \mathcal{L}(g^{(i)},y^{(i)})\)
x^{(i)}
y^{(i)}
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^2
f^L
g^{(i)}
\(\dots\)
\mathcal{L}(g^{(i)}, y^{(i)})
\mathcal{L}(g, y)
\mathcal{L}(g^{(n)}, y^{(n)})
\underbrace{\quad \quad \quad \quad \quad }
\dots
\dots
\dots
\(\nabla_{W^2} \mathcal{L}(g^{(i)},y^{(i)})\)
n
Backward pass: run SGD to update all parameters
e.g. to update \(W^2\)
x
\(\dots\)
y
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g
\underbrace{\hspace{4.7cm}}
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^3}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial Z^2}{\partial W^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}
how to find
?
\frac{\partial \mathcal{L}(g,y)}{\partial W^1}
x
\(\dots\)
y
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^L
\mathcal{L}(g,y)
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^3}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\frac{\partial Z^2}{\partial W^{2}}
\underbrace{\hspace{4.7cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}
how to find
?
Previously, we found
f^2
\underbrace{\hspace{6.5cm}}
\frac{\partial Z^2}{\partial A^{1}}
\frac{\partial A^1}{\partial Z^{1}}
\frac{\partial Z^1}{\partial W^{1}}
x
\(\dots\)
y
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^L
\mathcal{L}(g,y)
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^3}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^1}
how to find
Now
\frac{\partial \mathcal{L}(g,y)}{\partial W^1}
f^2
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^3}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial A^2}{\partial Z^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\underbrace{\hspace{4.7cm}}
\frac{\partial Z^2}{\partial W^{2}}
\frac{\partial \mathcal{L}(g,y)}{\partial W^2}
backpropagation: reuse of computation
x
\(\dots\)
y
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^2
f^L
\mathcal{L}(g,y)
\underbrace{\hspace{6.5cm}}
\frac{\partial Z^2}{\partial A^{1}}
\frac{\partial A^1}{\partial Z^{1}}
\frac{\partial Z^1}{\partial W^{1}}
x
\(\dots\)
y
f^1
\begin{aligned}
& W^1 \\
\end{aligned}
\begin{aligned}
& W^2 \\
\end{aligned}
\begin{aligned}
& W^L \\
\end{aligned}
f^2
f^L
\mathcal{L}(g,y)
g
Z^L
A^2
Z^2
A^1
Z^1
\frac{\partial \mathcal{L}}{\partial W^1} = \frac{\partial Z^1}{\partial W^1} \cdot \frac{\partial A^1}{\partial Z^1} \cdot \frac{\partial Z^2}{\partial A^1} \cdot \color{#93c47d}{\frac{\partial \mathcal{L}}{\partial Z^2}}
\underbrace{\hspace{4.7cm}}
\frac{\partial Z^2}{\partial W^{2}}
\frac{\partial \mathcal{L}}{\partial W^2} = \frac{\partial Z^2}{\partial W^2} \cdot \color{#93c47d}{\frac{\partial \mathcal{L}}{\partial Z^2}}
backpropagation: reuse of computation
\frac{\partial \mathcal{L}(g,y)}{\partial g}
\frac{\partial g}{\partial Z^{L}}
\frac{\partial Z^3}{\partial A^{2}}\frac{\partial A^3}{\partial Z^{3}} \dots \frac{\partial Z^L}{\partial A^{L-1}}
\frac{\partial \mathcal{L}(g,y)}{\partial Z^2}
\underbrace{\hspace{4cm}}
\frac{\partial A^2}{\partial Z^{2}}
Training a neural network: the full loop
Initialize all weights \(W^1, W^2, \dots, W^L\) randomly
Repeat until stopping criterion:
- Forward pass: pick a data point \((x^{(i)}, y^{(i)})\), compute \(Z^1, A^1, Z^2, A^2, \dots, g^{(i)}\)
- Evaluate loss: compute \(\mathcal{L}(g^{(i)}, y^{(i)})\)
- One full backward pass: compute \(\nabla_{W^\ell} \mathcal{L}(g^{(i)}, y^{(i)})\) for every \(\ell = L, \dots, 1\), reusing cached results
In practice, mini-batches rather than pure SGD; in PyTorch, loss.backward() then optimizer.step().
e.g., three layers:
\begin{array}{rr} \frac{\partial \mathcal{L}}{\partial W^3} = & \frac{\partial Z^3}{\partial W^3} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}} \\[6pt] \phantom{\frac{\partial \mathcal{L}}{\partial W^2} =} & \phantom{\frac{\partial Z^2}{\partial W^2} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \\[6pt] \phantom{\frac{\partial \mathcal{L}}{\partial W^1} =} & \phantom{\frac{\partial Z^1}{\partial W^1} \cdot \frac{\partial A^1}{\partial Z^1} \cdot \frac{\partial Z^2}{\partial A^1} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \end{array}
\begin{array}{rr} \phantom{\frac{\partial \mathcal{L}}{\partial W^3} =} & \phantom{\frac{\partial Z^3}{\partial W^3} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \\[6pt] \frac{\partial \mathcal{L}}{\partial W^2} = & \frac{\partial Z^2}{\partial W^2} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}} \\[6pt] \phantom{\frac{\partial \mathcal{L}}{\partial W^1} =} & \phantom{\frac{\partial Z^1}{\partial W^1} \cdot \frac{\partial A^1}{\partial Z^1} \cdot \frac{\partial Z^2}{\partial A^1} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \end{array}
\begin{array}{rr} \phantom{\frac{\partial \mathcal{L}}{\partial W^3} =} & \phantom{\frac{\partial Z^3}{\partial W^3} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \\[6pt] \phantom{\frac{\partial \mathcal{L}}{\partial W^2} =} & \phantom{\frac{\partial Z^2}{\partial W^2} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}}} \\[6pt] \frac{\partial \mathcal{L}}{\partial W^1} = & \frac{\partial Z^1}{\partial W^1} \cdot \frac{\partial A^1}{\partial Z^1} \cdot \frac{\partial Z^2}{\partial A^1} \cdot \textcolor{#6aa84f}{\frac{\partial A^2}{\partial Z^2} \cdot \frac{\partial Z^3}{\partial A^2}} \cdot \textcolor{#3d85c6}{\frac{\partial g}{\partial Z^3} \cdot \frac{\partial \mathcal{L}}{\partial g}} \end{array}
- Update all weights at once: \(W^\ell \leftarrow W^\ell - \eta\, \nabla_{W^\ell} \mathcal{L}\) for all \(\ell\)
Outline
-
Backward pass (to learn parameters/weights)
- Backpropagation (gradient descent & the chain rule)
- Recursive reuse of computation
- Practical gradient issues and remedies
The flat branch
on the flat branch the ReLU's slope is 0, and one zero factor zeroes the whole product
at \(w = 4\) (the middlemen slide): \(3 \cdot 1 \cdot 12 = 36\)
\text{ReLU}
y
= wx
\Sigma
z
\frac{\partial g}{\partial z}
\frac{\partial z}{\partial w}
\nabla_{w} \mathcal{L}(g,y) = 3 \cdot 0 \cdot (-12) = 0
\frac{\partial \mathcal{L}}{\partial g}
g
\mathcal{L}(g, y)
= \text{ReLU}(z)
“\(z\) tells \(w\) how to change”
\(\dfrac{\partial z}{\partial w} = x = 3\)
“the guess tells \(z\) how to change”
\(\dfrac{\partial g}{\partial z} = \text{ReLU}'(-3) = 0\)
“the loss tells the guess how to change”
\(\dfrac{\partial \mathcal{L}}{\partial g} = 2(g - y) = -12\)
x
w
\phantom{= wx} = -3
= 0
\begin{array}{r}\text{1-d example:}\\ x = 3,\ y = 6,\ w = -1\end{array}
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = 1\)
\(W^2_2 = 1\)
\(\mathcal{L}(g, y) = (g - y)^2\)
More neurons, more layers
forward pass: evaluation
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = 1\)
\(W^2_2 = 1\)
\(\mathcal{L}(g, y) = (g - y)^2\)
\(12\)
\(12\)
\(6\)
\(6\)
\(18\)
\(18\)
\(144\)
More neurons, more layers
forward pass: evaluation
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = 1\)
\(W^2_2 = 1\)
\(12\)
\(12\)
\(6\)
\(6\)
\(18\)
\(18\)
The backward pass
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial g}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial g}{\partial z^2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 1 \cdot 24\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^2_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial W^2_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 12 \cdot \textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 288\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^2_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial W^2_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 6 \cdot \textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 144\)
\(\begin{aligned} \dfrac{\partial \mathcal{L}}{\partial g} &= 2(g - y) \\[2pt] &= 2(18 - 6) \\[2pt] &= 24 \end{aligned}\)
gradient w.r.t. 2nd layer weights:
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = 1\)
\(W^2_2 = 1\)
\(12\)
\(12\)
\(6\)
\(6\)
\(18\)
\(18\)
The backward pass
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial a^1_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial a^1_1}{\partial z^1_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^1_1}{\partial W^1_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^1_1}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 3 \cdot 1 \cdot 1 \cdot \textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 72\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial a^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial a^1_2}{\partial z^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^1_2}{\partial W^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 3 \cdot 1 \cdot 1 \cdot \textcolor{#6aa84f}{24}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 72\)
gradient w.r.t. 1st layer weights:
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = -1\)
\(W^2_1 = 1\)
\(W^2_2 = 1\)
\(12\)
\(12\)
\(-3\)
\(0\)
\(12\)
\(12\)
if ReLU input is negative, some weights won't update
\(\begin{aligned} \dfrac{\partial \mathcal{L}}{\partial g} &= 2(g - y) \\[2pt] &= 2(12 - 6) \\[2pt] &= 12 \end{aligned}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial g}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial g}{\partial z^2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 1 \cdot 12\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#6aa84f}{12}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial a^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial a^1_2}{\partial z^1_2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
👻 dead ReLU
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^1_2}{\partial W^1_2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial W^1_2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 3 \cdot \textcolor{#cc0000}{0} \cdot 1 \cdot \textcolor{#6aa84f}{12}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial W^2_2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial z^2}{\partial W^2_2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0} \cdot \textcolor{#6aa84f}{12}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0}\)
\(W^2_2 = 1\)
\(W^1_2 = -1\)
gradient w.r.t. red weights:
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = -1\)
\(W^2_2 = -1\)
\(12\)
\(12\)
\(6\)
\(6\)
\(-18\)
\(0\)
if \(z^2 < 0\), no weights get updated
\(\begin{aligned} \dfrac{\partial \mathcal{L}}{\partial g} &= 2(g - y) \\[2pt] &= 2(0 - 6) \\[2pt] &= -12 \end{aligned}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial g}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial g}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0} \cdot (-12)\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0}\)
👻 dead ReLU
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial W^2_k}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial W^2_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= a^1_k \cdot \textcolor{#cc0000}{0}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial W^1_k}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^1_k}{\partial W^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial a^1_k}{\partial z^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial a^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#cc0000}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 3 \cdot 1 \cdot (-1) \cdot \textcolor{#cc0000}{0}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#cc0000}{0}\)
\(W^2_2 = -1\)
\(W^2_1 = -1\)
\(W^1_2 = 2\)
\(W^1_1 = 4\)
gradient w.r.t. all weights:
\(k = 1\) or \(2\), the hidden neuron the weight connects to, so each row covers both weights of its layer
\mathcal{L}(g, y)
\Sigma
\Sigma
\text{ReLU}
\text{ReLU}
\begin{bmatrix}
a_{1}^1 \\[3ex]
a_2^1
\end{bmatrix}
\Sigma
g
\text{ReLU}
z^{2}
z_{1}^{1}
z_{2}^{1}
\(x = 3\)
\(y = 6\)
\(W^1_1 = 4\)
\(W^1_2 = 2\)
\(W^2_1 = -1\)
\(W^2_2 = -1\)
\(12\)
\(12\)
\(6\)
\(6\)
\(-18\)
\(-18\)
Residual (skip) connection
\(W^2_2 = -1\)
\(W^2_1 = -1\)
\(W^1_2 = 2\)
\(W^1_1 = 4\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial g}{\partial z^2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 1 + \text{ReLU}'(-18)\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 1\)
\(\begin{aligned} \dfrac{\partial \mathcal{L}}{\partial g} &= 2(g - y) \\[2pt] &= 2(-18 - 6) \\[2pt] &= -48 \end{aligned}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial g}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial g}{\partial z^2}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 1 \cdot (-48)\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= \textcolor{#6aa84f}{-48}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^2_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial W^2_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= a^1_k \cdot (\textcolor{#6aa84f}{-48})\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial \mathcal{L}}{\partial W^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}=\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^1_k}{\partial W^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial a^1_k}{\partial z^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\dfrac{\partial z^2}{\partial a^1_k}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\cdot\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}\textcolor{#6aa84f}{\dfrac{\partial \mathcal{L}}{\partial z^2}}\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 3 \cdot 1 \cdot (-1) \cdot (\textcolor{#6aa84f}{-48})\)
\(\vphantom{\dfrac{\partial \mathcal{L}}{\partial W^1_1}}= 144\)
even if \(z^2 < 0\), with skip connection, weights in earlier layers can still get updated
gradient w.r.t. all weights:
Going deep: vanishing and exploding gradients
backpropagation computes gradients via the chain rule — a product of many factors:
e.g. \(\displaystyle \quad \frac{\partial \mathcal{L}}{\partial W^1} \;=\; \frac{\partial Z^1}{\partial W^1} \cdot \underbrace{\frac{\partial A^1}{\partial Z^1} \cdot \frac{\partial Z^2}{\partial A^1} \cdots \frac{\partial A^{L-1}}{\partial Z^{L-1}} \cdot \frac{\partial Z^L}{\partial A^{L-1}}}_{{L-1}\text{ layers of multiplicative factors}} \cdot \frac{\partial g}{\partial Z^L} \cdot \frac{\partial \mathcal{L}}{\partial g}\)
Factors below 1: the product shrinks fast with depth, so gradients vanish
Factors above 1: the product grows fast with depth, so gradients explode
Common remedies:
- residual connections
- gradient clipping (if \(\|\nabla\| > \tau\), rescale \(\nabla \leftarrow \tau \frac{\nabla}{\|\nabla\|}\), keeping the direction)
- careful initialization, normalization: both keep factors near 1
Summary
backpropagation backward passbackpropagationchain rule
practical issues dead ReLUskip connectionvanishing / exploding gradient
Backpropagation is gradient descent plus the chain rule: each “tells” is one partial derivative, and a weight’s gradient multiplies the factors between it and the loss.
One backward sweep caches \(\partial \mathcal{L}/\partial Z\) at each layer, and every earlier weight reuses it.
One training step: forward pass, loss, backward pass, then all weights update at once (as in PyTorch).
On its flat branch a ReLU passes zero gradient: a dead neuron freezes its edges, and a dead output freezes the whole network.
Skip connections keep a slope-1 path for the gradient; long products of factors can still vanish or explode (clipping and careful initialization help).
6.390 IntroML (Fall26) - Lecture 6 Neural Networks II
By Shen Shen
