Speech Project

Week-6 Progress Report

B02901054 方為

B02901085 徐瑞陽

Kaldi Project

Special Project
2nd stage

Natural Language Processing

Recursive Neural Network

Evolution

Meaning of Word

1 - of - N encoding

再根據不同應用去train

一個Neural Network

Neighbor
word

Middle
word
prob.

e.g

Word2Vec

將單字轉為投射到實數向量空間

而且這個空間有許多好的性質

Text
coupus

A set of
vector

Meaning of sentence

sentence word sequence

\approx

\approx

Bag of words

Without considering the order

But order is matter @@"

Sequential Model

Tree Model

Implementation

RNN

Recurrent Neural Network

Can't catch
long sentence Meaning

CONS

Gradient Explosion or Vanish

LSTM

Long Short-Term Memory

強化版 Neuron Layer

Input gate :

Output gate :

Forget gate :

Memory Cell要被留下來的比例

Memory Cell要隨Input更新的比例

Memory Cell要呈現在output的比例

求導很複雜...

還好我們有theano ^.＜

\frac{\partial O^{r} }{\partial c_t} \approx f_t \odot \frac{\partial O^{r} }{\partial c_{t-1}}

\frac{\partial O^{r} }{\partial c_t} \approx f_t \odot \frac{\partial O^{r} }{\partial c_{t-1}}

c_t = i_t \odot g + f_t \odot c_{t-1}

c_t = i_t \odot g + f_t \odot c_{t-1}

簡化版error signal backpropogate

跟memory cell相關的所有參數

Tree - LSTM

Child-Sum Tree-LSTM

Dependency Tree

\tilde{h}_{see} =\left \{ h_{John} , h_{Bill}\right \}

\tilde{h}_{see} =\left \{ h_{John} , h_{Bill}\right \}

\tilde{h}_{j} = \sum_{k \in Child(j)} h_k

\tilde{h}_{j} = \sum_{k \in Child(j)} h_k

f_{jk} = \sigma\left ( W^{(f)}x_j + U^{(f)}h_k + b^{(f)}\right )

f_{jk} = \sigma\left ( W^{(f)}x_j + U^{(f)}h_k + b^{(f)}\right )

c_j = i_j \odot g + \sum_{k \in Child(j)} f_{jk} \odot c_k

c_j = i_j \odot g + \sum_{k \in Child(j)} f_{jk} \odot c_k

N-ary Tree-LSTM

Constituency Tree

i_j = \sigma\left ( W^{(i)}x_j +\sum_{l=1}^{N} U_{l}^{(i)} h_{jl} + b^{(i)}\right )

i_j = \sigma\left ( W^{(i)}x_j +\sum_{l=1}^{N} U_{l}^{(i)} h_{jl} + b^{(i)}\right )

c_j = i_j \odot g + \sum_{l=1}^{N} f_{jl} \odot c_{jl}

c_j = i_j \odot g + \sum_{l=1}^{N} f_{jl} \odot c_{jl}

是leaf 才有

x_j

x_j

o_j , f_j

o_j , f_j