An autoencoder is a special type of feed forward neural network which does the following

Encodes its input \(\text x_ \text i\) into a hidden representation \(\text h\)

The model is trained to minimize a certain loss function which will ensure that \(\hat \text{x}_\text{i}\) is close to \(\text{x}_\text i\) (we will see some such loss functions soon)

Decodes the input again from this hidden representation

Let us consider the case where

\(\text {dim}\)(\(\textbf h) < \text {dim} (\textbf x_ \textbf i)\)

If we are still able to reconstruct \(\hat \text{x}_\text{i}\) perfectly from \(\text h\), then what does it say about \(\text h\)? 

Do you see an analogy with PCA?

\(\text h\) is a loss-free encoding of \(\text{x}_\text i\). It captures all the important characteristics of \(\text{x}_\text i\)

In such a case the autoencoder could learn a trivial encoding by simply copying \(\text{x}_\text{i}\) into \(\text h\) and then copying \(\text h\) into \(\hat \text{x}_\text{i}\)

Let us consider the case where

\(\text {dim}\)(\(\textbf h) \ge \text {dim} (\textbf x_ \textbf i)\)

Do you see an analogy with PCA?

Such an identity encoding is useless in practice as it does not really tell us anything about the important characteristics of the data

   Choice of \(f(\textbf x_ \textbf i)\) and \(g(\textbf x_ \textbf i)\)

   Choice of loss function

Suppose all our inputs are binary (each \(x_{ij} \in {0,1}\))

Which of the following functions would be most apt for the decoder?

Logistic as it naturally restricts all
outputs to be between 0 and 1

Suppose all our inputs are real (each \(x_{ij} \in \mathbb R\))

Which of the following functions would be most apt for the decoder?

What will logistic and tanh do?

They will restrict the reconstructed \(\hat \text{x}_\text{i}\) to lie between \([0,1]\) or \([-1,1]\) whereas we want \(\hat \text{x}_\text{i}\) \(\in \mathbb R^n\)

Consider the case when the inputs are real
valued

The objective of the autoencoder is to reconstruct ^xi to be as close to xi as possible

We can then train the autoencoder just like a regular feedforward network using back- propagation

This can be formalized using the following objective function:

Note that the loss function is shown for only one training example.

We have already seen how to calculate the expression in the boxes when we learnt backpropagation

Consider the case when the inputs are binary

We use a sigmoid decoder which will produce outputs between 0 and 1, and can be interpreted as probabilities.

For a single n-dimensional \(i^{th}\) input we can use the following loss function

Again we need a formula for \(\frac{\partial \mathscr L(\theta)}{\partial W^*}\) and \(\frac{\partial \mathscr L(\theta)}{\partial W}\) to use backpropagation

What value of \(\hat x_{ij}\) will minimize this function?

If \(x_{ij} = 1 \) \(?\)

Indeed the above function will be minimized when \(\hat x_{ij} = x_{ij} !\) 

If \(x_{ij} = 0 \) \(?\)

We have already seen how to calculate the expressions in the square boxes when we learnt BP

The first two terms on RHS can be computed as:

use a linear encoder

use a linear decoder

use squared error loss function

normalize the inputs to

The operation in the bracket ensures that the data now has \(0\) mean along each dimension \(j\) (we are subtracting the mean)

Let \(X'\) be this zero mean data matrix then what the above normalization gives us is \(X = \frac{1}{\sqrt m} X'\)

Now \((X)^TX = \frac{1}{\sqrt m} (X')^TX'\) is the covariance matrix (recall that covariance matrix plays an important role in PCA)

First we will show that if we use linear decoder and a squared error loss function then

The optimal solution to the following objective function

is obtained when we use a linear encoder.

This is equivalent to

(just writing the expression (1) in matrix form and using the definition of \(\Vert A \Vert _ F\)) (we are ignoring the biases)

From SVD we know that optimal solution to the above problem is given by

By matching variables one possible solution is

Thus \(H\) is a linear transformation of \(X\) and \(W=V_{., \leq k}\)

From SVD, we know that \(V\) is the matrix of eigen vectors of \(X^TX\)

From PCA, we know that P is the matrix of the eigen vectors of the covariance matrix

We saw earlier that, if entries of \(X\) are normalized by

then \(X^TX\) is indeed the covariance matrix

Thus, the encoder matrix for linear autoencoder (\(W\)) and the projection
matrix (\(P\)) for PCA could indeed be the same. Hence proved

use a linear encoder

use a linear decoder

use a squared error loss function

and normalize the inputs to

Here, (as stated earlier) the model can simply learn to copy \(\text{x}_\text{i}\) into \(\text h\) and then copying \(\text h\) into \(\hat \text{x}_\text{i}\)

While poor generalization could happen even in undercomplete autoencoders it is an even more serious problem for overcomplete auto encoders

To avoid poor generalization, we need to introduce regularization

Another trick is to tie the weights of the encoder and decoder 

i.e., \(W^* = W^T\)

This helps because the objective is still to reconstruct the original (uncorrupted) \(\textbf x_i\)

For example,

Where \(W_1\) is the trained vector of weights connecting the input to the first hidden neuron

What values of \(\textbf x_i\) will cause  \(\textbf h_1\) to be maximum (or maximally activated)

Suppose we assume that our inputs are normalized so that \(\Vert \textbf x_i \Vert =1\)

Solution: \(\textbf x_i = \cfrac{W_1}{\sqrt {W_1^TW_1}}\)

Let us plot these images (\(\textbf x_i\)'s) which maximally activate the first \(k\) neurons of the hidden representations learned by a vanilla autoencoder and different denoising autoencoders

These \(\textbf x_i\)'s are computed by the above formula using the weights \((W_1, W_2 ... W_k)\) learned by the respective autoencoders

The hidden neurons of the denoising AEs seem to act like pen-stroke detectors (for example, in the highlighted neuron the black region is a stroke that you would expect in a '0' or a '2' or a '3' or a '8' or a '9')

As the noise increases the filters become more wide because the neuron has to rely on more adjacent pixels to feel confident about a stroke

Another way of corrupting the inputs is to add
a Gaussian noise to the input

\(\tilde x_{ij} = x_{ij} + \mathcal N (0,1)\) 

We will now use such a denoising AE on a different dataset and see their performance

PCA does not give such edge detectors

A hidden neuron with sigmoid activation will have values between 0 and 1

We say that the neuron is activated when its output is close to 1 and not activated when its output is close to 0.

A sparse autoencoder tries to ensure the neuron is inactive most of the times.

A sparse autoencoder uses a sparsity parameter \(\rho\) (typically very close to 0, say, 0.005) and tries to enforce the constraint \(\hat \rho_l = \rho\)

One way of ensuring this is to add the following term to the objective function

When will this term reach its minimum value and what is the minimum value? Let us plot it and check.

The function will reach its minimum value(s) when \(\hat \rho_l = \rho\).

\(\mathscr L (\theta)\) is the squared error loss or cross entropy loss and \(\Omega (\theta)\) is the sparsity constraint.

We already know how to calculate \(\frac{\partial \mathscr L (\theta)}{\partial W}\)

Let us see how to calculate \(\frac{\partial \mathscr \Omega (\theta)}{\partial W}\)

Finally,

(and we know how to calculate both terms on R.H.S)

Can be re-written as

By Chain rule:

For each neuron \(l \in 1...k\) in hidden layer, we have

(see next slide)

It does so by adding the following regularization term to the loss function

where, \(J_\textbf x (\textbf h)\) is the Jacobian of the encoder.

Let us see how it looks like.

In other words, the (\(j, l\)) entry of the Jacobian captures the variation in the output of the \(l^{th}\) neuron with a small variation in the \(j^{th}\) input.

Consider \(\frac{\partial h_1}{\partial x_1}\), what does it mean if \(\frac{\partial h_1}{\partial x_1} = 0\) ?

It means that this neuron is not very sensitive to variations in the input \(x_1\).

But doesn't this contradict our other goal of minimizing \(\mathscr L (\theta)\) which requires \(\mathbf h\) to capture variations in the input.

By putting these two contradicting objectives against each other we ensure that \(h\) is sensitive to only very important variations as observed in the training data.

Tradeoff - capture only very important variations in the data

\(\mathscr L (\theta)\) - capture important variations in data

\(\Omega(\theta)\) - do not capture variations in data

It makes sense to maximize a neuron to be sensitive to variations along \(\mathbf u_1\) 

At the same time it makes sense to inhibit a neuron from being sensitive to variations along \(\mathbf u_2\) (as there seems to be small noise and unimportant for reconstruction)

What does this remind you of ?

By doing so we can balance between the contradicting goals of good reconstruction and low sensitivity.

Regularization

Weight decaying

Sparse

Contractive