Language Segmentation of Twitter Tweets using Weakly Supervised Language Model Induction

David Alfter

15 September 2015

The Problem

N-GRAM Language Model

P(w_i|w_1,\ldots,w_{i-1}) =

P(w_i|w_1,\ldots,w_{i-1}) =

d_{w_{i-n+1},\ldots,w_i} \frac{C(w_{i-n+1},\ldots,w_i)}{C(w_{i-n+1},\ldots,w_{i-1})}

d_{w_{i-n+1},\ldots,w_i} \frac{C(w_{i-n+1},\ldots,w_i)}{C(w_{i-n+1},\ldots,w_{i-1})}

\alpha_{w_{i-n+1},\ldots,w_{i-1}} P(w_i|w_{i-n+2},\ldots,w_{i-1})

\alpha_{w_{i-n+1},\ldots,w_{i-1}} P(w_i|w_{i-n+2},\ldots,w_{i-1})

{

n-gram probability

Language Segmentation of Twitter Tweets using Weakly Supervised Language Model Induction David Alfter 15 September 2015