The network contains $\text L-1$ hidden layers (2, in this case) having $\text n$ neurons each

Finally, there is one output layer containing $\text k$ neurons (say, corresponding to $\text k$ classes)

$a_2$

$a_3$

Each neuron in the hidden layer and output layer can be split into two parts : pre-activation and activation ($a_i$ and $h_i$ are vectors)

                                                                                                                               pre-activation and activation ($a_i$ and $h_i$ are vectors)

                                                                         and activation ($a_i$ and $h_i$ are vectors)

($a_i$ and $h_i$ are vectors)

The input layer can be called the $0$-th layer and the output layer can be called the ($L$)-th layer

$W_i \in \R^{n \times n}$ and $b_i \in \R^n$ are the weight and bias between layers $i-1$ and $i  (0 < i < L$)

$W_L \in \R^{n \times k}$ and $b_L \in \R^k$ are the weight and bias between the last hidden layer and the output layer   ($L = 3$) in this case)

$a_1$

$h_L=\hat {y} = f(x)$

$h_2$

$h_1$

The activation at layer $i$ is given by

$h_i(x) = g(a_i(x))$

where $g$ is called the activation function (for example, logistic, tanh, linear, etc.)

The activation at the output layer is given by

$f(x) = h_L(x)=O(a_L(x))$

where $O$ is the output activation function (for example, softmax, linear, etc.)

To simplify notation we will refer to $a_i(x)$ as $a_i$ and $h_i(x)$ as $h_i$

$\hat y_i = f(x_i) = O(W_3 g(W_2 g(W_1 x + b_1) + b_2) + b_3)$

Model:

$\theta = W_1, ..., W_L, b_1, b_2, ..., b_L (L = 3)$

Algorithm: Gradient Descent with Back-propagation (we will see soon)

Objective/Loss/Error function: Say,

$min \cfrac {1}{N} \displaystyle \sum_{i=1}^N \sum_{j=1}^k (\hat y_{ij} - y_{ij})^2$

$\text {In general,}$ $min  \mathscr{L}(\theta)$

Parameters:

where $\mathscr{L}(\theta)$ is some function of the parameters

Recall our gradient descent algorithm

We can write it more concisely as

where $\nabla \theta_t = [\frac {\partial \mathscr{L}(\theta)}{\partial w_t},\frac {\partial \mathscr{L}(\theta)}{\partial b_t}]^T$

Now, in this feedforward neural network, instead of $\theta = [w,b]$ we have $\theta = [W_1, ..., W_L, b_1, b_2, ..., b_L]$

We can still use the same algorithm for learning the parameters of our model

$\nabla \theta$ is thus composed of

$\nabla W_1, \nabla W_2, ..., \nabla W_{L-1} \in \R^{n \times n}, \nabla W_L \in \R^{n \times k},$ 

$\nabla b_1, \nabla b_2, ..., \nabla b_{L-1} \in \R^{n}, \nabla b_L \in \R^{k}$ 

How to choose the loss function $\mathscr{L}(\theta)?$

$\nabla W_1, \nabla W_2, ..., \nabla W_{L-1} \in \R^{n \times n}, \nabla W_L \in \R^{n \times k},$ 

$\nabla b_1, \nabla b_2, ..., \nabla b_{L-1} \in \R^{n}, \nabla b_L \in \R^{k} ?$ 

How to compute $\nabla \theta$ which is composed of 

We will illustrate this with the help of two examples

Consider our movie example again but this time we are interested in predicting ratings

Here $y_i \in \R ^3$

The loss function should capture how much $\hat y_i$ deviates from $y_i$

If $y_i \in \R ^n$ then the squared error loss can capture this deviation

$\mathscr {L}(\theta) = \cfrac {1}{N} \displaystyle \sum_{i=1}^N \sum_{j=1}^3 (\hat y_{ij} - y_{ij})^2$

More specifically, can it be the logistic function?

No, because it restricts $\hat y_i$ to a value between $0$ & $1$ but we want $\hat y_i \in \R$

So, in such cases it makes sense to have '$O$' as linear function

$f(x) = h_L = O(a_L)$

$= W_Oa_L + b_O$

$\hat y_i = f(X_i)$ is no longer bounded between $0$ and $1$