Introduction to Deep Learning
Contents
- Neural network's architecture overview
- Activation functions
- Backpropagation
Neural network's architecture overview
Architecture overview
The most basic component of an artificial neural network is the activation unit.
It is made of an input, or set of n inputs (which may include a constant bias term) an 'activation' function and an output.
X1
X_1
X2
X_2
…
\dots
Xn
X_n
∼
\sim
θ1
\theta_1
θ2
\theta_2
θn
\theta_n
X0
X_0
θ0
\theta_0
Oi
O_i
∑
\sum
Activation node
∼
\sim
∑
\sum
Multilayer network
When we stack this units together into layers, we get a multilayer artificial neural network
O1
O_1
O2
O_2
O3
O_3
∼
\sim
∑
\sum
∼
\sim
∑
\sum
∼
\sim
∑
\sum
∼
\sim
∑
\sum
∼
\sim
∑
\sum
∼
\sim
∑
\sum
∼
\sim
∑
\sum
X1
X_1
X2
X_2
X3
X_3
Learning rules
Classification example:
XOR function
Let us suppose that we want to create a two layer neural network able to classify these observations.
(0,1)→1
(0,1) \rightarrow 1
(1,1)→0
(1,1) \rightarrow 0
(1,0)→1
(1,0) \rightarrow 1
(0,0)→0
(0,0) \rightarrow 0
Learning rules
Classification example:
XOR function
Or equivalently, we want a neural network able to create a classification region such as the yellow one.
Learning rules
Classification example:
XOR function
Proposed solution
∼
\sim
∑
\sum
−0.5
-0.5
1
1
∼
\sim
∑
\sum
∼
\sim
∑
\sum
−1
-1
+1
+1
1
1
X1
X_1
X2
X_2
+1
+1
+1
+1
+1
+1
+1
+1
−0.5
-0.5
−1.5
-1.5
Learning rules
∼
\sim
∑
\sum
−0.5
-0.5
1
1
∼
\sim
∑
\sum
∼
\sim
∑
\sum
−1
-1
+1
+1
1
1
X1
X_1
X2
X_2
+1
+1
+1
+1
+1
+1
+1
+1
−0.5
-0.5
−1.5
-1.5
0
0
1
1
heaviside(0∗1+1∗1+(−1.5∗1))=heaviside(−0.5)=0
heaviside(0*1 + 1*1 + (-1.5*1))=heaviside(-0.5)=0
heaviside(0∗1+1∗1+(−0.5∗1))=heaviside(0.5)=1
heaviside(0*1 + 1*1 + (-0.5*1))=heaviside(0.5)=1
heaviside(−1∗0+1∗1+(−0.5∗1))=heaviside(0.5)=1
heaviside(-1*0 + 1*1 + (-0.5*1))=heaviside(0.5)=1
Learning rules
∼
\sim
∑
\sum
−0.5
-0.5
1
1
∼
\sim
∑
\sum
∼
\sim
∑
\sum
−1
-1
+1
+1
1
1
X1
X_1
X2
X_2
+1
+1
+1
+1
+1
+1
+1
+1
−0.5
-0.5
−1.5
-1.5
0
0
0
0
heaviside(0∗1+1∗0+(−1.5∗1))=heaviside(−1.5)=0
heaviside(0*1 + 1*0 + (-1.5*1))=heaviside(-1.5)=0
heaviside(0∗1+1∗0+(−0.5∗1))=heaviside(−0.5)=0
heaviside(0*1 + 1*0 + (-0.5*1))=heaviside(-0.5)=0
heaviside(−1∗0+1∗0+(−0.5∗1))=heaviside(−0.5)=0
heaviside(-1*0 + 1*0 + (-0.5*1))=heaviside(-0.5)=0
Activation functions
Activation function
More complex activation functions
Activation function
Backpropagation
Backpropagation
O1
O_1
O2
O_2
O3
O_3
Now our objective is to train our network with a gradient based method, and to somehow propagate the errors to the previous layers
Backpropagation
O1
O_1
O2
O_2
O3
O_3
Of course, with more complex architectures, the problem of computing gradients becomes an issue
Backpropagation
O1
O_1
O2
O_2
O3
O_3
Introduction to Deep Learning
Introduction to Deep Learning
By Luis Roman
Introduction to Deep Learning
- 1,264
