4. Pipeline, Soft Pipeline, Mean Field, and Neural CRF

Pipeline

Fruit

flies

like

a

banana

A

N

V

D

N

Pipeline

Fruit

flies

like

a

banana

A

N

V

D

N

B-NP

I-NP

O

B-NP

I-NP

Pipeline

Fruit

flies

like

a

banana

B-NP

I-NP

O

B-NP

I-NP

Pipeline

Fruit

flies

like

a

banana

B-NP

I-NP

O

B-NP

I-NP

NP

NP

VP

S

A

N

V

D

N

Pipeline

Fruit

flies

like

a

banana

A

N

V

D

N

B-NP

I-NP

O

B-NP

I-NP

Pipeline

Fruit

flies

like

a

banana

A

N

V

D

N

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Pipeline

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Pipeline

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Pipeline

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Pipeline

Pipeline

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

A

N

V

D

N

f_{318}(\mathbf{P},[\mathbf{O}])=0.0
f_{298}(\mathbf{V},[\mathbf{O}])=1.0
f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0

Pipeline

Fruit

flies

like

a

banana

\left[ \begin{array}{c} 0.0\\ 1.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.0\\ 1.0\\ 0.0 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 1.0\\ 0.0\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0
f_{298}(\mathbf{V},[\mathbf{O}])=1.0
f_{318}(\mathbf{P},[\mathbf{O}])=0.0

Pipeline

Fruit

flies

like

a

banana

A

N

V

D

N

B-NP

I-NP

O

B-NP

I-NP

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Soft Pipeline

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Soft Pipeline

Fruit

flies

like

a

banana

\left[ \begin{array}{c} 0.1\\ 0.5\\ 0.1\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.0\\ 0.1\\ 0.6\\ 0.0 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.5\\ 0.2\\ 0.1\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.0\\ 0.1 \end{array} \right]
f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0
\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

f_{298}(\mathbf{V},[\mathbf{O}])=0.5
f_{318}(\mathbf{P},[\mathbf{O}])=0.2

Soft Pipeline

Fruit

flies

like

a

banana

B-NP

I-NP

O

B-NP

I-NP

\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.0\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.1\\ 0.5\\ 0.1\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.5\\ 0.2\\ 0.1\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.0\\ 0.1\\ 0.6\\ 0.0 \end{array} \right]

N

V

D

A

P

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Soft Pipeline

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

What about...

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

What about...

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Joint Modeling with Mean Field

Joint Modeling

Fruit

flies

like

a

banana

A

N

V

D

N

B-NP

I-NP

O

B-NP

I-NP

Factorial CRF

Fruit

flies

like

a

banana

A

N

V

D

N

B

I

O

B

I

p(y_{pos},y_{np}|\color{grey}{x})

Sutton, C, McCallum, A. & Rohanimanesh, K. (2007) Dynamic Conditional Random Fields

Factorial CRF

Fruit

flies

like

a

banana

p(y_{pos},y_{np})\approx q(y_{pos}) q(y_{np})
\min KL\big(q(y_{pos}) q(y_{np})||p(y_{pos},y_{np})\big)

A

N

V

D

N

B

I

O

B

I

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Mean Field

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

q(y_{np})
q(y_{pos})

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Mean Field

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

Iteration 0

Fruit

flies

like

a

banana

Mean Field

Iteration 0

\left[ \begin{array}{c} 0.1\\ 0.4\\ 0.1\\ 0.2\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.1\\ 0.1\\ 0.3\\ 0.2 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.3\\ 0.2\\ 0.1\\ 0.3\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.7\\ 0.2\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.0\\ 0.0\\ 0.2\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.1\\ 0.4\\ 0.5 \end{array} \right]
\left[ \begin{array}{c} 0.6\\ 0.3\\ 0.1 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} 0.4\\ 0.5\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.6\\ 0.1\\ 0.3 \end{array} \right]
\left[ \begin{array}{c} 0.1\\ 0.7\\ 0.2 \end{array} \right]

Fruit

flies

like

a

banana

Mean Field

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} 0.1\\ 0.4\\ 0.1\\ 0.2\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.1\\ 0.1\\ 0.3\\ 0.2 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.3\\ 0.2\\ 0.1\\ 0.3\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.7\\ 0.2\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.0\\ 0.0\\ 0.2\\ 0.0 \end{array} \right]

Iteration 1

q_{pos}

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Mean Field

\left[ \begin{array}{c} 0.1\\ 0.4\\ 0.5 \end{array} \right]
\left[ \begin{array}{c} 0.6\\ 0.3\\ 0.1 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} 0.4\\ 0.5\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.6\\ 0.1\\ 0.3 \end{array} \right]
\left[ \begin{array}{c} 0.1\\ 0.7\\ 0.2 \end{array} \right]

Iteration 1

Fruit

flies

like

a

banana

Mean Field

Iteration 1

\left[ \begin{array}{c} \color{grey}{0.1}\;0.0\\ \color{grey}{0.4}\;0.7\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.2}\;0.2 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.3}\;0.5\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.3}\;0.2\\ \color{grey}{0.2}\;0.2 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} \color{grey}{0.3}\;0.5\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.3}\;0.2\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.1\\ \color{grey}{0.7}\;0.7\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.8}\;0.8\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.0}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.1}\;0.0\\ \color{grey}{0.4}\;0.4\\ \color{grey}{0.5}\;0.6 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.6}\;0.8\\ \color{grey}{0.3}\;0.2\\ \color{grey}{0.1}\;0.0 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} \color{grey}{0.4}\;0.3\\ \color{grey}{0.5}\;0.6\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.6}\;0.7\\ \color{grey}{0.1}\;0.2\\ \color{grey}{0.3}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.1}\;0.0\\ \color{grey}{0.7}\;0.8\\ \color{grey}{0.2}\;0.2 \end{array} \right]

Fruit

flies

like

a

banana

Mean Field

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} 0.0\\ 0.7\\ 0.0\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.5\\ 0.0\\ 0.1\\ 0.2\\ 0.2 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.5\\ 0.1\\ 0.1\\ 0.2\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.1\\ 0.7\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.0\\ 0.0\\ 0.1\\ 0.1 \end{array} \right]

Iteration 2

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Mean Field

\left[ \begin{array}{c} 0.0\\ 0.4\\ 0.6 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.2\\ 0.0 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} 0.3\\ 0.6\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.7\\ 0.2\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.8\\ 0.2 \end{array} \right]

Iteration 2

Fruit

flies

like

a

banana

Mean Field

Iteration 2

\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.7}\;0.7\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.2}\;0.2 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.5}\;0.8\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.2}\;0.1 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} \color{grey}{0.5}\;0.7\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.2}\;0.2\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.0\\ \color{grey}{0.7}\;0.9\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.1}\;0.0 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.8}\;0.8\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.4}\;0.2\\ \color{grey}{0.6}\;0.8 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.8}\;0.9\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.0}\;0.0 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} \color{grey}{0.3}\;0.1\\ \color{grey}{0.6}\;0.8\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.7}\;0.9\\ \color{grey}{0.2}\;0.1\\ \color{grey}{0.1}\;0.0 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.8}\;0.8\\ \color{grey}{0.2}\;0.2 \end{array} \right]

Fruit

flies

like

a

banana

Mean Field

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} 0.0\\ 0.7\\ 0.0\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.0\\ 0.0\\ 0.1\\ 0.1 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.7\\ 0.0\\ 0.0\\ 0.2\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.1\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.8\\ 0.0\\ 0.0\\ 0.1\\ 0.1 \end{array} \right]

Iteration 3

Fruit

flies

like

a

banana

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

N

V

D

A

P

Mean Field

\left[ \begin{array}{c} 0.0\\ 0.2\\ 0.8 \end{array} \right]
\left[ \begin{array}{c} 0.9\\ 0.1\\ 0.0 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} 0.1\\ 0.8\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.9\\ 0.1\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.8\\ 0.2 \end{array} \right]

Iteration 3

Fruit

flies

like

a

banana

Mean Field

Iteration 3

\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.7}\;0.7\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.2}\;0.2 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.8}\;0.8\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.1}\;0.1 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} \color{grey}{0.7}\;0.7\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.2}\;0.2\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.9}\;0.9\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.0}\;0.0 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.8}\;0.8\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.0}\;0.0\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.2}\;0.2\\ \color{grey}{0.8}\;0.8 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.9}\;0.9\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.0}\;0.0 \end{array} \right]

 B 

I

 O 

 

\left[ \begin{array}{c} \color{grey}{0.1}\;0.1\\ \color{grey}{0.8}\;0.8\\ \color{grey}{0.1}\;0.1 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.9}\;0.9\\ \color{grey}{0.1}\;0.1\\ \color{grey}{0.0}\;0.0 \end{array} \right]
\left[ \begin{array}{c} \color{grey}{0.0}\;0.0\\ \color{grey}{0.8}\;0.8\\ \color{grey}{0.2}\;0.2 \end{array} \right]

Learning

x, y_{pos}, y_{np}
\textbf{w}
Given Find
\text{1. Randomly initialize }\textbf{w}
\text{2. Mean field inference to obtain }q_{pos}\text{ and }q_{np} \text{ (i.e., minimize KL)}.
\text{3. Update model parameter }\textbf{w}.
\text{4. Iterate step 2 and 3 until convergence}.

Decoding

x, \textbf{w}
y_{pos}, y_{np}
Given Find
\text{1. Mean field inference to obtain }q_{pos}\text{ and }q_{np} \text{ (i.e., minimize KL)}.
\text{2. Decode based on }q_{pos}\text{ and }q_{np}.

Neural CRF

Soft Pipeline

Fruit

flies

like

a

banana

\left[ \begin{array}{c} 0.1\\ 0.5\\ 0.1\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.0\\ 0.1\\ 0.6\\ 0.0 \end{array} \right]

N

V

D

A

P

\left[ \begin{array}{c} 0.5\\ 0.2\\ 0.1\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.0\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

f_{298}(\mathbf{V},[\mathbf{O}])=0.5
f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0
f_{318}(\mathbf{P},[\mathbf{O}])=0.2

Soft Pipeline

Fruit

flies

like

a

banana

\left[ \begin{array}{c} 0.1\\ 0.5\\ 0.1\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.0\\ 0.1\\ 0.6\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.5\\ 0.2\\ 0.1\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.0\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

1

2

3

4

5

\dim
f_{298}(\dim 2,[\mathbf{O}])=0.5
f_{318}(\dim 5,[\mathbf{O}])=0.2
f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0

Soft Pipeline

Fruit

flies

like

a

banana

\left[ \begin{array}{c} 0.1\\ 0.5\\ 0.1\\ 0.1\\ 0.2 \end{array} \right]
\left[ \begin{array}{c} 0.3\\ 0.0\\ 0.1\\ 0.6\\ 0.0 \end{array} \right]
\left[ \begin{array}{c} 0.5\\ 0.2\\ 0.1\\ 0.1\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 0.0\\ 0.0\\ 0.9\\ 0.0\\ 0.1 \end{array} \right]
\left[ \begin{array}{c} 1.0\\ 0.0\\ 0.0\\ 0.0\\ 0.0 \end{array} \right]

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

1

2

3

4

5

\dim
f_{298}(\dim 2,[\mathbf{O}])=0.5
f_{318}(\dim 5,[\mathbf{O}])=0.2
f_{124}(like,[\mathbf{O},\mathbf{B}])=1.0
\mathbf{f}= \left[ \begin{array}{cc} \mathbf{f}_{disc}\\ \mathbf{f}_{cont} \end{array} \right]

Word Embeddings

Fruit

flies

like

a

banana

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]

Pre-trained word embeddings (e.g. word2vec)

Pipeline with Embeddings

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]
L

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]
U

Pipeline with Embeddings

Pipeline with Embeddings

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]
\bigtriangledown \mathbf{f}_{cont}

Fine Tuning Embeddings

Neural CRF

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]

MLP

MLP

MLP

MLP

MLP

Neural CRF

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +2.8\\ +2.2\\ -5.5\\ \vdots\\ +2.5\\ -7.3 \end{array} \right]
\left[ \begin{array}{c} +4.0\\ +3.4\\ -3.2\\ \vdots\\ +6.1\\ -8.5 \end{array} \right]
\left[ \begin{array}{c} +7.7\\ -4.4\\ +6.4\\ \vdots\\ -2.2\\ +1.8 \end{array} \right]
\left[ \begin{array}{c} -1.8\\ +0.7\\ -4.2\\ \vdots\\ +1.7\\ -7.8 \end{array} \right]
\left[ \begin{array}{c} -1.3\\ +1.8\\ +1.9\\ \vdots\\ -0.8\\ -1.7 \end{array} \right]

MLP

MLP

MLP

MLP

MLP

\bigtriangledown
\bigtriangledown
\bigtriangledown
\bigtriangledown
\bigtriangledown

LSTM CRF

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +0.8\\ +0.2\\ -4.5\\ \vdots\\ -1.5\\ +8.3 \end{array} \right]
\left[ \begin{array}{c} -2.0\\ +2.7\\ -9.3\\ \vdots\\ +9.4\\ -5.4 \end{array} \right]
\left[ \begin{array}{c} +5.7\\ -2.4\\ +4.4\\ \vdots\\ -1.2\\ +0.8 \end{array} \right]
\left[ \begin{array}{c} +2.2\\ +0.2\\ -3.4\\ \vdots\\ -1.2\\ +7.2 \end{array} \right]
\left[ \begin{array}{c} +5.2\\ +1.2\\ -2.4\\ \vdots\\ +0.2\\ -1.2 \end{array} \right]

LSTM

LSTM

LSTM

LSTM

LSTM

LSTM CRF

Fruit

flies

like

a

banana

B

I

O

B

I

O

B

I

O

B

I

O

B

I

O

\left[ \begin{array}{c} +2.8\\ +2.2\\ -5.5\\ \vdots\\ +2.5\\ -7.3 \end{array} \right]
\left[ \begin{array}{c} +4.0\\ +3.4\\ -3.2\\ \vdots\\ +6.1\\ -8.5 \end{array} \right]
\left[ \begin{array}{c} +7.7\\ -4.4\\ +6.4\\ \vdots\\ -2.2\\ +1.8 \end{array} \right]
\left[ \begin{array}{c} -1.8\\ +0.7\\ -4.2\\ \vdots\\ +1.7\\ -7.8 \end{array} \right]
\left[ \begin{array}{c} -1.3\\ +1.8\\ +1.9\\ \vdots\\ -0.8\\ -1.7 \end{array} \right]
\bigtriangledown
\bigtriangledown
\bigtriangledown
\bigtriangledown
\bigtriangledown

LSTM

LSTM

LSTM

LSTM

LSTM

Overall Picture

Inference

\mathbf{w}
\max

MAP

Inference

Marginal

\log\sum\exp
\mathbf{w}

A Unified Framework

\mathbf{w}^{(k)}
\mathbf{w}^{(k+1)}
L
L
U
\Delta(y,y')

A Unified Framework

\mathbf{w}^{(k)}
\mathbf{w}^{(k+1)}
L
L
U
\Delta(y,y')

A Unified Framework

\mathbf{w}^{(k)}
\mathbf{w}^{(k+1)}
L
L
U
\Delta(y,y')

A Unified Framework

\mathbf{w}^{(k)}
\mathbf{w}^{(k+1)}
L
L
U
\Delta(y,y')
\text{GM, NN, Word Embeddings, ...}
\mathbf{f}