Can we create a system, in which feeding an image, we can generate a reasonable caption in plain english ? 

Input to the system:

Output : 

• A group of teenage boys on a road jumping joyfully.

• Four boys running and jumping.

• Four kids jumping on the street with a blue car in the back.

• Four young men are running on a street and jumping for joy.

• Several young man jumping down the street.

Let's See, How we can accomplish this!

For image classification, we will use VGG 16 layer model, popularly known as VGG16.

• Very simple and homogeneous.

• Better accuracy because of increased depth of the network by adding more convolutional layers.

• Which is feasible due to the use of very small (3 × 3) convolution filters in all layers.

CNN

For sequence modelling, we use Recurrent Neural Network

P(next word | previous words)

RNN

Merging CNN with RNN

Training Example

Before:

h0 = max(0, Wxh * x0)

Now:

h0 = max(0, Wxh * x0
           + Wih * v)

RNN vs LSTM

• We will use a Long-Short Term Memory (LSTM) net, which has shown state-of-the art performance on sequence tasks such as translation and sequence generation. 

• Also due to its ability to deal with vanishing and exploding gradients, the most common challenge in designing and training RNNs.

RNN vs LSTM (contd.)

h1 = max(0, Wxh * x1 + Whh * h0)

LSTM changes the form of the equation for h1 such that:

1. More expressive multiplicative interactions.

2. Gradients flow nicer.

3. Network can explicitly decide to reset the hidden state.

• LSTM model trained to predict word of sentence after it has seen image as well as previous words.

• Use BPTT (Backprop through time) to train.

• Loss function used : negative log likelihood as usual.

Model Summary

LSTM model combined with a CNN image embedder and word embeddings. The unrolled connections between the LSTM memories are in blue and they correspond to the recurrent connections. All LSTMs share the same parameters.