Neural Network
roychuang
參考我寒訓的講義：
Why Neural Network
機器學習的目的就是擬合出一個函數 f
用統計學的方法以外，科學家想到我們可以模擬人類大腦神經元的運作連結方式
深度學習 (Deep Learning)，一種機器學習的方法，模擬人類大腦神經元的運作模式，由許多神經細胞構成神經網路，經歴學習的過程。
深度學習又可以分為 Multilayer perceptron (MLP), Recurrent neural network (RNN), convolutional neural network (CNN) 等等。
Matrices
Matrix
Matrices
- 算是一種資料結構
- 用來把一堆數字包起來/一起做運算
Add
Matrices
矩陣加矩陣的話大小要一樣
Multiply a Scalar
Matrices
Vector & dot product
Matrices
Transpose
Matrices
Multiplication
Matrices
Multiplication
Matrices
Multiplication
Matrices
Multiplication
Matrices
Neural Network
Neural Network
Human Brain
Neural Network
\(x_1\)
\(x_2\)
\(x_3\)
\(y_1\)
\(y_2\)
Input Layer
Hidden Layers
Output
Layer
Neural Network (NN)
Neuron
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
\(z = x_1w_1 + x_2w_2 + b\)
Neural Network
Neuron
Neuron
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
Activation Function
Neural Network
Example
1
Activation : Sigmoid \(\sigma(z) = \frac{1}{1+e^{-z}} \)
-1
1
0
0
0
-2
2
1
-2
-1
1
2
-1
-2
1
3
-1
-1
4
4
0.98
-2
0.12
0.86
0.11
0.62
0.83
Neural Network
Training the Network
Training
Process
- 目標：給定輸入之後，可以得到期望的輸出。
- 先隨機給定一些權重 \(\vec{W}\) 和輸入 \(\vec{X }\)，接著透過誤差函數 (Loss Function) 比較輸出 \(\hat{y}\) 和正解 \(y\) 的差距 (Loss)，最後根據 Loss 去調整各個 Weight 和 Bias。
Training
Gradient Descent
-
調整 Weight 和 Bias 的過程稱為最佳化 (Optimization, 最常用的方法是梯度下降 Gradient Descent)。
-
Gradient Descent 指的是，把誤差畫成函數圖形以後，根據斜坡的方向，朝著更低的那個點走，以此來更新 Weights
Training
Backpropagation
- 加速 optimization
- 得到其中一個 Weight 的 Gradient 以後透過一層層回推的方式快速反推所有其他 Weight 的 Gradient
Derivative & Partial
Derivative & Partial
Chain Rule
Derivative & Partial
Gradient Descent
Gradient Descent
Loss
W
Gradient Descent
Loss
Target
W
Gradient Descent
Loss
Start From Here
W
Gradient Descent
Loss
W
Gradient Descent
Loss
W
x -= Slope * Learning Rate
Gradient Descent
Loss
W
Gradient Descent
Loss
W
Gradient Descent
Loss
Until Reaching the minimum
W
Gradient Descent
Backpropagation
Backpropagation
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
\(z = x_1w_1 + x_2w_2 + b\)
......
\(y_1\)
\(y_2\)
let loss function = C
Backpropagation
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
\(z = x_1w_1 + x_2w_2 + b\)
......
let loss function = C
\(x_1\)
Forward Pass
Backpropagation
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
\(z = x_1w_1 + x_2w_2 + b\)
......
let loss function = C
Forward Pass:
Store the outputs from previous neuron
Forward Pass
Backpropagation
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
Activation Function
a = \(\sigma(z)\)
\(W_3\)
\(W_4\)
......
......
\(z'' = aw_4 + ...\)
\(z' = aw_3 + ...\)
\(W_3\)
\(W_4\)
Backward Pass
Backpropagation
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
Activation Function
a = \(\sigma(z)\)
\(W_3\)
\(W_4\)
Backward Pass
Case 1 :
\(z'\)
\(z''\)
Softmax
\(z' = aw_3 + ...\)
\(z'' = aw_4 + ...\)
\(y_1\)
\(y_2\)
Backpropagation
Activation Function
Backward Pass
Case 2 :
Next layer is not output layer
\(W_1\)
\(W_2\)
b
\(x_1\)
\(x_2\)
\(z\)
a = \(\sigma(z)\)
\(W_3\)
\(W_4\)
......
......
Calculate \({\partial{C}}/{\partial{z'}}\) by recursion
Hand Written Number Classification with MLP
Convolutional Neural Network
CNN
CNN
