Long Short-Term Memory (LSTM)
X
X
+
X
Feed-forward neural network
input layer
hidden layer
output layer
Feed-Forward Neural Network
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
O2
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
O2
O3
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
O2
O3
O4
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
O2
O3
O4
O5
Recurrent Neural Network
Mon
Tues
Wed
Thurs
Fri
O1
O2
O3
O4
O5
output
Recurrent Neural Network
Recurrent Neural Network
short-term memory!
Final RNN hidden state
Vanishing Gradient Problem!
Final RNN hidden state
Forward Propagation
Forward Propagation
Forward Propagation
Forward Propagation
Forward Propagation
Forward Propagation
Forward Propagation
Forward Propagation
Error Estimation
Forward Propagation
Back Propagation
Error Estimation
Forward Propagation
Back Propagation
Error Estimation
Forward Propagation
Back Propagation
Error Estimation
Forward Propagation
Back Propagation
Error Estimation
Repeat!
Back Propagation
∇
Vanishing Gradient Problem!
∇
Back Propagation
Vanishing Gradient Problem!
∇
Back Propagation
Vanishing Gradient Problem!
∇
Back Propagation
Vanishing Gradient Problem!