RNN

Sequence Processing Tasks

Sequence tagging
- POS tagging
- Named Entity Recognition
- Sentence Compression
Sequence Transduction
- Machine Translation
- Speech Recognition
- Dialogue

Recurrent Neural Networks

$x_1$

$x_2$

$x_3$

$x_4$

$x_5$

$U$

$W$

$V$

$s_1$

$s_2$

$s_3$

$s_4$

$s_5$

$\hat{y}_1$

$\hat{y}_2$

$\hat{y}_3$

$\hat{y}_4$

$\hat{y}_5$

$s_i = \sigma (Ux_i + Ws_{i-1} + b)$
$\hat{y}_i = o(Vs_i +c)$

$s_0$

$W$

Find a cheap Chinese restaurant

$s_i = RNN (s_{i-1},x_i)$

VB DT JJ JJ NN

Recurrent Neural Networks

$x_i$

$U$

$s_t$

$\hat{y}_i$

$s_{t-1}$

$W$

$V \in \mathbb{R}^{36 \times d}$

$y_i = softmax(Vs_i +c)$
$softmax(a)_i = \frac{e^{a_i}}{\sum_j e^{a_j}}$
$\mathscr{L}_i$ = - $\sum_c y_{ci} log \hat{y}_{ci}$

HMM vs RNN

HMMs are simpler than RNN

36 Hidden States (POS Tags)
5000 words in vocabulary
HMM Parameters:
- Transition Parameters: 36*36 = 1296
- Emission Parameters: 36*5000 = 180000
- Total = 181296
RNN Parameters:
- $U$ = 5000*300 = 1500000
- $W$ = 300*300 = 90000
- $V$ = 300*36 = 10800
- Total = 1600800

HMMs have less parameters, hence require less data

HMM vs RNN

HMMs make a Markov Assumption

$P(y_t | y_{t-1},...,y_2,y_1) = P(y_t | y_{t-1})$

RNNs model the entire conditional dependency

$P(w_6 |w_5,w_4,..,w_1)$

$x_1$

$x_2$

$x_3$

$x_4$

$x_5$

$U$

$W$

$V$

$s_1$

$s_2$

$s_3$

$s_4$

$s_5$

$\hat{y}_5$

$s_0$

$W$

Find me a cheap Chinese

HMM vs RNN

HMMs are generative models

VB

DT

JJ

Find

a

cheap

$P(x) = \sum_y P(x,y)$

$= \sum_y P(x|y)P(y)$

$= \sum_y \prod_tP(x_t|y_t)\prod_tP(y_t|y_{t-1})$

HMM vs RNN

RNNs are primarily discriminative models

$x_1$

$x_2$

$x_3$

$x_4$

$x_5$

$U$

$W$

$V$

$s_1$

$s_2$

$s_3$

$s_4$

$s_5$

$\hat{y}_5$

$s_0$

$W$

$P(\hat{y}_5 |x_5,x_4,..,x_1)$