Neural Artistic Style Transfer

ABSTRACT

Neural Artistic Style Transfer helps us create images with different combinations of Contents and Patterns. Content Images and Pattern Images can be according to the choice of the user. User can include any kind of images whatever he likes. It is based on Convolutional Neural Network i.e. CNN which helps in Image processing. Also it allows us to extract separately the content and style of the image. There can be different Pattern Images so that we can build a particular image with different patterns. Prisma is a mobile app that allows to transfer the style of one image to the content of another.

INTRODUCTION

CNNs are like Neural Network made up of learnable weights and biases. It works on multi-channeled images. Convolution Layer is the main Building block of CNN and each layer consists of a number of filters. It also consists of a pooling layer which reduces the spatial size of the representation i.e. computations. CNN takes an input image, detects objects, process it and classify it under certain categories. Computers sees an input image as array of pixels in the form of h x w x d(h = height, w = width, d = dimention).

Convolution Layer is the first layer to extract features from an input image. It is a mathematical operation that takes two inputs such as image matrix and a filter, which led to generation of "Feature Matrix" after the multiplication of both. Different filters build different covolution of an image such as blur, sharpen, etc.

Pooling Layer helps in reducing the number of parameters when the images are too large. There are different types of pooling i.e. Max, Average and Sum. Max takes the largest element from the generated feature map and in the same way Average and sum.

Gram Matrix is the results from multiplying a matrix with the transpose of itself. It is used to extract the style of an image.

APPROACH

I have used Convolutional Neural Network i.e. CNN to implement this idea. CNN is the right choice for image processing. Also, it allows us to extract separately the content and the style of an image. For Content Image CNN gets the higher layer because at First layers almost whole content is available while we want only semantic concepts which can be achieved only by higher layers. For Style Image, correlations between different types of neurons in the network are computed using Gram Matrix.

Objective

Generate an image which is a combination of both Content Image and  Pattern Image.

Methodology

Algorithm to extract content of Content Image:

Step 1 : An Image is provided as input

Step 2 : The image is converted into array of matrix of pixels

Step 3 : The matrix is passed through trained CNN and the Convolution takes place. Feature Matrix is extracted with the help of Filter and Image matrix.

Step 4 : Pooling of the Image takes place so that the image doesn't become too large

Step 5 : Again Step 3 and Step 4 are repeated for a limited number of times depends on the user.

Methodology

Algorithm to extract style of Style Image:

Step 1 : An Image is provided as input.

Step 2 : The image is converted into array of pixels.

Step 3 : Calculate the Transpose of the Matrix.

Step 4 : The Gram Matrix is calculated with the help of multiplication of original Matrix and Transpose matrix. It is used to extract the style of the style image as it removes Raw data and keeps only important information.

Conclusion

Able to generate an image which is a combination of content image and pattern image

References

1. https://arxiv.org/pdf/1508.06576v2.pdf
2. https://medium.com/@RaghavPrabhu/understanding-of-convolutional-neural-network-cnn-deep-learning-99760835f148
3. https://towardsdatascience.com/neural-style-transfer-tutorial-part-1-f5cd3315fa7f
4. https://towardsdatascience.com/artistic-style-transfer-b7566a216431
5. https://towardsdatascience.com/neural-networks-intuitions-2-dot-product-gram-matrix-and-neural-style-transfer-5d39653e7916
6. https://medium.com/data-science-group-iitr/artistic-style-transfer-with-convolutional-neural-network-7ce2476039fd
7. http://www.subsubroutine.com/sub-subroutine/2016/11/12/painting-like-van-gogh-with-convolutional-neural-networks