Long Short-Term Memory (LSTM)

X

X

+

X

Feed-forward neural network

input layer

hidden layer

output layer

Feed-Forward Neural Network

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

O2

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

O2

O3

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

O2

O3

O4

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

O2

O3

O4

O5

Recurrent Neural Network

Mon

Tues

Wed

Thurs

Fri

O1

O2

O3

O4

O5

output

Recurrent Neural Network

Recurrent Neural Network

short-term memory!

Final RNN hidden state

Vanishing Gradient Problem!

Final RNN hidden state

Forward Propagation

Forward Propagation

Forward Propagation

Forward Propagation

Forward Propagation

Forward Propagation

Forward Propagation

Forward Propagation

Error Estimation

Forward Propagation

Back Propagation

Error Estimation

Forward Propagation

Back Propagation

Error Estimation

Forward Propagation

Back Propagation

Error Estimation

Forward Propagation

Back Propagation

Error Estimation

Repeat!

Back Propagation

∇

Vanishing Gradient Problem!

∇

Back Propagation

Vanishing Gradient Problem!

∇

Back Propagation

Vanishing Gradient Problem!

∇

Back Propagation

Vanishing Gradient Problem!