Making Sense of Text Data with Deep NLP

Presentation Link: http://bit.ly/deep_nlp_pydata

Code Link: https://github.com/smfullman/deep_nlp_pydata

Note for posterity: This presentation was given in 2018; much of it is still conceptually sound but there have been many advancements in NLP/deep learning since then (and the code examples have not been maintained).  If you are interested in learning more, check out the most recent versions of the courses mentioned in the last slide

Scott

• Main gig: Tribune doing data science
• Also: Work with startups focused on using machine learning (shameless plug: all of them are hiring)
• Previously: PwC Analytics/ML Lab
• Edu: Chicago Booth (MBA, Concentration: Statistics); University of Illinois Urbana-Champaign (Systems Engineering)

Goals

• Answer the question: "Why use deep NLP?"

• Introduce a framework for building deep NLP models, and demonstrate the power of incorporating external data into your task

• Give enough knowledge to get started, provide enough resources to fill in the gaps

• Note: There are a few major concepts flippantly introduced in this talk, denoted by a hand-wave: 👋.  Go ⬇️ in the slides for more information, references, links to resources, and code examples

Why use deep NLP?

• A smattering of what's possible:

  • Classification: group text into categories

  • Sentiment: decide the positive/negative view of text

  • Attribution: assign authorship, detect language type

  • Part of Speech: parse sentence structure

  • Language Modeling: score likelihood of sentences

  • Named Entity Recognition: identify types of words

  • Summarization, Translation: consolidate or transfer text information in a human readable way

  • Dialogue: formulate simple responses to conversation snippets

  • Question answering: formulate responses to questions 

  • Multi-modal transfer: caption images or audio transcripts

Warning

• Build simple models first!!! assess time / resource consumption versus gain tradeoff

• If you're not using deep learning for the data representations, transfer learning, or multi-output/multi-modal data, you probably don't need deep learning

Warm-up Example

• Task: Classify data info one of 4 categories

• Data: 800 examples, each example has two dimensions ('x1', 'x2'), evenly distributed across categories

• Metric: Accuracy

Warm-up Example

1) Data is set of feature vectors

Warm-up Example

1) Data is set of feature vectors

2) Model made up of layers, layers have weights and activation functions

3) Data transformed in model by series of functions, resulting in a final vector

Warm-up Example

1) Data is set of feature vectors

2) Model made up of layers, layers have weights and activation functions

3) Data transformed in model by series of functions, resulting in a final vector

4) Loss function compares model output to true output, quantifies "incorrectness"

Warm-up Example

1) Data is set of feature vectors

2) Model made up of layers, layers have weights and activation functions

3) Data transformed in model by series of functions, resulting in a final vector

4) Loss function compares model output to true output, quantifies "incorrectness"

5) Compute partial derivative of loss wrt weights, use optimizer to determine step size

Warm-up Example

1) Data is set of feature vectors

2) Model made up of layers, layers have weights and activation functions

3) Data transformed in model by series of functions, resulting in a final vector

4) Loss function compares model output to true output, quantifies "incorrectness"

5) Compute partial derivative of loss wrt weights, use optimizer to determine step size

6) Update model weights accordingly

Warm-up Example

1) Data is set of feature vectors

2) Model made up of layers, layers have weights and activation functions

3) Data transformed in model by series of functions, resulting in a final vector

4) Loss function compares model output to true output, quantifies "incorrectness"

5) Compute partial derivative of loss wrt weights, use optimizer to determine step size

6) Update model weights accordingly

7) Rinse and repeat until loss stop going down

Warm-up Example

Warm-up Example

Warm-up Example

Blue line = Accuracy

Orange line = Loss

Task: Movie Review Sentiment Classification

• Establish Baseline
• Prepare Data
• Build Model
• Train, Predict, and Measure

Establish a Baseline

• This should look familiar: count how many times you see a bi-gram, adjust by TF-IDF, push through a linear classifier with strong regularization

• Note the feature vector here is more than 200,000 dimensions due to combinatorial bi-gram problem 

Random baseline: 0.503
Ridge model accuracy: 0.873
Confusion Matrix:
[[11032  1468]
 [ 1702 10798]]
Report: 

             precision    recall  f1-score   support

          0       0.87      0.88      0.87     12500
          1       0.88      0.86      0.87     12500

avg / total       0.87      0.87      0.87     25000

Establish a Baseline

Prepare Data

• Classic NLP: lower, stem, remove stopwords, lemmatize, n-gram, dependency or part of speech

• With deep learning, we're asking the model to learn to compose features automatically

• Preparing text for deep learning model is not that different but there are some considerations:

  • Sequential tokens, hashing, and padding

  • Your tokenizer can influence performance greatly

  • Minimize vocab for memory and speed

  • Splitting train/val/test: be careful with sequential data; with large data sets, we don't actually need a lot in val and test (< ~10%)

Prepare Data

• Original string:

•  

• After tokenizer:

•  

• After int:

• After int:

• After pad:

Building a Model

• Our model will have four types of layers: embedding, convolutional, LSTM, and attention

• Let's build it layer by layer and talk about what they each do to transform the data

Building a Model: Embed

• How does a deep NLP model encode text information? We need to vectorize it, but how?

• Words are essentially categorical data so can we one-hot encode? Count/TF-IDF vector?

• Unfortunately both are too big, and don't allow for generalization.  We need a more compact input.

• Solution: Distributed representation of tokens, a.k.a word vectors or word embeddings

• Why? Pretty good representations of the word "meaning", while being much lower dimension, and you can learn on unlabeled data

Building a Model: Embed

• Main components remain the same: Input, Model layers, Output, Loss function, Optimizer, Compile

• Embedding layer takes care of the integer to word vector representation, the embedding weights are a matrix of word vectors

• Initially set trainable=False, vector weights wont update

Building a Model: Conv

• Convolutions have two main parameters:

  • 1) kernel size (how many words to view at once)

  • 2) filter size (how many different filters you want)

• Convolutions are powerful because they help us solve the n-gram problem, we can look at several words and represent them together, without increasing our vocab

Kernel Size: 2

Filter Size: 3

Building a Model: Conv

• Convolutions have two main parameters:

  • 1) kernel size (how many words to view at once)

  • 2) filter size (how many different filters you want)

• Convolutions are powerful because they help us solve the n-gram problem, we can look at several words and represent them together, without increasing our vocab

Kernel Size: 2

Filter Size: 3

Building a Model: LSTM

• LSTMs are is designed to pass along error signals better than stock RNNs (👋)

• The LSTM receives the input from the conv layer above, and also the previous step output

• The last hidden state (red box) can be used as a representation of the entire text, but this typically doesn't work well in practice because we are trying to compress too much information into too small a vector

Building a Model: Attention

• The attention layer looks at the every individual output from the LSTM layer and decides if it's relevant to the task

• The result is a summed vector the same size of the LSTM hidden state, but with information sampled throughout the sequence

• Our task requires just one vector to classify the whole review, but attention vectors can also be output every time step

Building a Model: Final

• We now have the major building blocks for creating a deep NLP model

• We can attach a classifier to the output of the LSTM/attention layers and have it predict the sentiment of the movie review

Building a Model: Final

• The LSTM is updated to return all hidden states with the return_sequences=True parameter

• The AttentionWithContext layer is a custom layer which attends to each hidden state from the LSTM and creates a single weighted sum vector

Language Model/Transfer Learning

• One of the huge advantages of deep learning models comes from the ability to learn generalized features from large amounts of "unlabeled" data and transfer that knowledge to a new task

• When you a model on billions of examples you get much better representation of the data in each layer

• When we want to apply this model "knowledge" to a new task, we can reuse the first few layers of a model, cut off the bottom, and then "glue" on a new task

Language Models

• In deep NLP, creating a language model is a common starting point for future tasks

• Language models attempt to answer: given some context sequence, what's the next word?  e.g., P(w|"who let the dogs")

• Since we only need strings of text to train a language model, we can use the billions of examples we find on the internet

Language Models

Language Models

Language Models

Original: 
"<sos> this is a charming version of the classic dicken 's tale . henry winkler makes a good showing as the scrooge character . even though you know what will happen this version has enough of a change to make it better that average . if you love a christmas carol in any version , then you will love this . <eos>"

Predicted:

"<sos> this is a charming version of the classic disney 's tale . henry nicholson makes a good showing as the character . even though you know what will happen this version has enough of a change to make it better that average . if you love a christmas carol in any version , then you will love this . <eos>"

Transfer Learning

Language Model

(reused weights)

Sentiment Model

(new weights)

Transfer Learning

• Reusing a model is as simple as picking the input and output layers from the original, and then inserting it into the new model

• Weights for reused layers will be transferred over while new layers will be randomly initialized

Transfer Learning

Accuracy: 0.91232
Confusion Matrix: 
[[11380  1120]
 [ 1072 11428]]
             precision    recall  f1-score   support

          0       0.91      0.91      0.91     12500
          1       0.91      0.91      0.91     12500

avg / total       0.91      0.91      0.91     25000

• F1 of 0.91 beats our baseline by a good margin, and we didn't really try to tune anything
• Current SOTA for this task is 0.95 using this general approach, more data, and some tricks with timing of freezing/unfreezing layers

Transfer Learning