Speech Project
Week-6 Progress Report
B02901054 方為
B02901085 徐瑞陽
Kaldi Project
Special Project
2nd stage
Natural Language Processing
Recursive Neural Network
Evolution
Meaning of Word
1 - of - N encoding
再根據不同應用去train
一個Neural Network
Neighbor
word
Middle
word
prob.
e.g
Word2Vec
將單字轉為投射到實數向量空間
而且這個空間有許多好的性質
Text
coupus
A set of
vector
Meaning of sentence
sentence word sequence
\approx
Bag of words
Without considering the order
But order is matter @@"
Sequential Model
Tree Model
Implementation
RNN
Recurrent Neural Network
Can't catch
long sentence Meaning
CONS
Gradient Explosion or Vanish
LSTM
Long Short-Term Memory
強化版 Neuron Layer
Input gate :
Output gate :
Forget gate :
Memory Cell要被留下來的比例
Memory Cell要隨Input更新的比例
Memory Cell要呈現在output的比例
求導很複雜...
還好我們有theano ^.＜
\frac{\partial O^{r} }{\partial c_t} \approx f_t \odot \frac{\partial O^{r} }{\partial c_{t-1}}
c_t = i_t \odot g + f_t \odot c_{t-1}
簡化版error signal backpropogate
跟memory cell相關的所有參數
Tree - LSTM
Child-Sum Tree-LSTM
Dependency Tree
\tilde{h}_{see} =\left \{ h_{John} , h_{Bill}\right \}
\tilde{h}_{j} = \sum_{k \in Child(j)} h_k
f_{jk} = \sigma\left ( W^{(f)}x_j + U^{(f)}h_k + b^{(f)}\right )
c_j = i_j \odot g + \sum_{k \in Child(j)} f_{jk} \odot c_k
N-ary Tree-LSTM
Constituency Tree
i_j = \sigma\left ( W^{(i)}x_j +\sum_{l=1}^{N} U_{l}^{(i)} h_{jl} + b^{(i)}\right )
c_j = i_j \odot g + \sum_{l=1}^{N} f_{jl} \odot c_{jl}
是leaf 才有
x_j
o_j , f_j
以此類推