NLP Foundations : Text Preprocessing & Vectorization

Business Scenario

Welcome!

In Lab 2 you built the network’s training loop step-by-step in Keras/TensorFlow without .fit(), separated the forward pass, loss, backpropagation, and gradient descent using tf.GradientTape

SmartCart's search bar receives raw, messy customer text -- search queries like "CHEAP wireless earbuds under 1000!!" and short reviews like "delivery was late but the laptop quality is good." Before any model (including the RNN in the next lab) can use this text, it has to be cleaned, tokenized, normalized, and turned into numbers.

Git Pull

git pull origin branchName

Pre-Lab Preparation

Topic : NLP Foundation

1) Introduction to NLP
2) Text Processing - (Cleaning, tokenization, lemmatization)
3) Vectorization - Bag of words(BoW) / TF-IDF
4) Word Embeddings

Natural Language Processing

NLP is the field concerned with getting computers to work with human language. Neural networks only understand numbers, so every NLP pipeline has to answer the same question: "how do we turn a messy sentence into numbers without throwing away its meaning?"

Setup

1

Task 1: Loading Dataset

import re
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

import nltk
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
from nltk.tokenize import word_tokenize

import tensorflow as tf
from tensorflow import keras
from keras.layers import TextVectorization, Embedding, GlobalAveragePooling1D, Dense
from keras import Sequential

from sklearn.preprocessing import LabelEncoder

np.random.seed(42)
tf.random.set_seed(42)

Load Dataset

2

Dataset : 

# One-time downloads for nltk's tokenizer, stopword list, and lemmatizer dictionary

nltk.download('punkt')
nltk.download('punkt_tab')
nltk.download('stopwords')
nltk.download('wordnet')
nltk.download('omw-1.4')
df = pd.read_csv('smartcart_customer_queries.csv')
print(df.shape)
df.head(10)
df['category'].value_counts()

Task 2 : Text Cleaning

def clean_text(text):
    text = str(text).lower()
    text = re.sub(r'[^a-z0-9\s]', ' ', text)
    text = re.sub(r'\s+', ' ', text).strip()
    return text

df['clean_text'] = df['query_text'].apply(clean_text)
df[['query_text', 'clean_text']].sample(5)

Lowercase everything, strip punctuation/special characters and collapse extra whitespace. Numbers are kept, since prices ("under 1000") are meaningful for SmartCart search.

Output

Task 3 : Tokenization

df['tokens'] = df['clean_text'].apply(word_tokenize)
df[['clean_text', 'tokens']].sample(5, random_state=1)

Split each cleaned string into individual word tokens with nltk's word_tokenize.

Output

Task 4 : Stopword Removal 

stop_words = set(stopwords.words('english'))

def remove_stopwords(tokens):
    return [t for t in tokens if t not in stop_words]

df['tokens_no_stop'] = df['tokens'].apply(remove_stopwords)
df[['tokens', 'tokens_no_stop']].sample(5, random_state=1)

Very common words ("the", "is", "a") carry little distinguishing meaning for search or classification, so they're usually dropped before vectorizing.

Output

Task 5 : Stemming Vs Lemmatization

Stemming (PorterStemmer) chops word endings using fixed rules — fast, but can produce forms that aren't real words ("running" → "run", but "organic" → "organ").

Lemmatization (WordNetLemmatizer) looks up the dictionary base form ("lemma") of a word — slower, but always produces a real word.

stemmer = PorterStemmer()
lemmatizer = WordNetLemmatizer()

sample_words = ['running', 'shoes', 'organic', 'cheaper', 'earbuds', 'jackets']

comparison = pd.DataFrame({
    'word': sample_words,
    'stemmed': [stemmer.stem(w) for w in sample_words],
    'lemmatized': [lemmatizer.lemmatize(w) for w in sample_words]})
comparison
def lemmatize_tokens(tokens):
    return [lemmatizer.lemmatize(t) for t in tokens]

df['final_tokens'] = df['tokens_no_stop'].apply(lemmatize_tokens)
df['processed_text'] = df['final_tokens'].apply(lambda toks: ' '.join(toks))
df[['query_text', 'processed_text']].sample(8, random_state=1)

Output

Full pipeline, end to end, on one example

sample_raw = df['query_text'].iloc[10]
print('1. Raw:       ', repr(sample_raw))
print('2. Cleaned:   ', clean_text(sample_raw))
toks = word_tokenize(clean_text(sample_raw))
print('3. Tokenized: ', toks)
toks_ns = remove_stopwords(toks)
print('4. No stopwords:', toks_ns)
toks_lem = lemmatize_tokens(toks_ns)
print('5. Lemmatized:', toks_lem)

Output

Task 6 : Vectorization : Bag-of-Words (BoW)

Text Vectorization builds the vocabulary 

processed_texts = df['processed_text'].tolist()

bow_vectorizer = TextVectorization(output_mode='count', max_tokens=2000)
bow_vectorizer.adapt(processed_texts)                      # builds the vocabulary

vocab = bow_vectorizer.get_vocabulary()
print('Vocabulary size:', len(vocab))
sample_queries = [processed_texts[10], processed_texts[50]]

for text in sample_queries:
    vec = bow_vectorizer([text])[0].numpy()
    nonzero = {vocab[i]: int(vec[i]) for i in np.nonzero(vec)[0]}
    print(f'Query: "{text}"  ->  BoW: {nonzero}')

Output :

Output :

Task 7 : Vectorization : TF - IDF

tfidf_vectorizer = TextVectorization(output_mode='tf_idf', max_tokens=2000)
tfidf_vectorizer.adapt(processed_texts)

tfidf_vocab = tfidf_vectorizer.get_vocabulary()
# Compare a common word vs. a rare word directly: how many queries each appears in,
# and the tf-idf weight that a single occurrence of that word gets.
common_word, rare_word = 'shoe', 'basmati'

for w in [common_word, rare_word]:
    if w in tfidf_vocab:
        doc_freq = sum(1 for t in processed_texts if w in t.split())
        tfidf_col = tfidf_vocab.index(w)
        single_word_weight = tfidf_vectorizer([w])[0, tfidf_col].numpy()
        print(f"'{w}': appears in {doc_freq} of {len(processed_texts)} queries "
              f"-> tf-idf weight per occurrence ~ {single_word_weight:.3f}")
    else:
        print(f"'{w}' not in vocabulary -- pick another word from `vocab` to compare")

Output :

Task 8 : Prepare Padded Sequence - Foundation of Embeddings

sequence_length = int(np.percentile(df['processed_text'].str.split().apply(len), 95))
print('Chosen padding length (95th percentile of query length):', sequence_length)

int_vectorizer = TextVectorization(
    output_mode='int', max_tokens=2000, output_sequence_length=sequence_length)
int_vectorizer.adapt(processed_texts)

X_sequences = int_vectorizer(np.array(processed_texts)).numpy()
print('Padded sequence matrix shape:', X_sequences.shape)
print('\nExample:')
print('  text:    ', processed_texts[10])
print('  sequence:', X_sequences[10])

BoW and TF-IDF both throw away word order. Embeddings need the opposite: each query kept as an ordered sequence of integer token IDs, padded to the same length so they can be batched. 

Output 

# Labels, ready for training a model on top of these sequences
label_encoder = LabelEncoder()
y = label_encoder.fit_transform(df['category'])
class_names = label_encoder.classes_
vocab_size = int_vectorizer.vocabulary_size()
print('Vocabulary size for embeddings:', vocab_size)
print('Classes:', class_names)

Output

Task 9 : Word Embeddings

An Embedding layer maps each token ID to a short, dense, trainable vector. Untrained, those vectors are random and meaningless; we train a tiny classifier on top of them so the embeddings learn to place words that show up in similar SmartCart queries close together in vector space.

embedding_dim = 16

embed_model = Sequential([
    Embedding(input_dim=vocab_size, output_dim=embedding_dim, input_length=sequence_length),
    GlobalAveragePooling1D(),          # average the word vectors in each query into one vector
    Dense(16, activation='relu'),
    Dense(3, activation='softmax')
])

embed_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', 
metrics=['accuracy'])
embed_model.summary()

Output 

history = embed_model.fit(X_sequences, y, epochs=30, validation_split=0.2, verbose=0)

print('Final training accuracy:  ', round(history.history['accuracy'][-1], 3))
print('Final validation accuracy:', round(history.history['val_accuracy'][-1], 3))

Calculating Accuracy

Output

Task 10 : Explore the Learned Embeddings

Pull out the embedding weight matrix and check whether SmartCart-relevant words that show up in similar contexts ended up with similar vectors -- something a raw BoW count could never tell us, since BoW treats every word as entirely independent from every other word.

embedding_weights = embed_model.layers[0].get_weights()[0]   # shape: (vocab_size, embedding_dim)
embed_vocab = int_vectorizer.get_vocabulary()

def word_vector(word):
    if word not in embed_vocab:
        return None
    return embedding_weights[embed_vocab.index(word)]

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-9)

pairs = [('shoe', 'sneaker'), ('cheap', 'budget'), ('organic', 'premium'),
         ('rice', 'sugar'), ('shoe', 'rice')]

for w1, w2 in pairs:
    v1, v2 = word_vector(w1), word_vector(w2)
    if v1 is not None and v2 is not None:
        print(f"cosine_similarity('{w1}', '{w2}') = {cosine_similarity(v1, v2):.3f}")
    else:
        print(f"'{w1}' or '{w2}' not in vocabulary -- try another pair from your dataset")

Output

Task 11 : Mini Text Analysis

new_query = "  Best WIRELESS Earbuds under 1500!! "

cleaned = clean_text(new_query)
tokens = word_tokenize(cleaned)
tokens_ns = remove_stopwords(tokens)
tokens_lem = lemmatize_tokens(tokens_ns)
processed = ' '.join(tokens_lem)

int_seq = int_vectorizer([processed])
predicted_probs = embed_model.predict(int_seq, verbose=0)[0]
predicted_class = class_names[np.argmax(predicted_probs)]

print('1. Raw:        ', repr(new_query))
print('2. Cleaned:    ', cleaned)
print('3. Tokens:     ', tokens)
print('4. No stopword:', tokens_ns)
print('5. Lemmatized: ', tokens_lem)
print('9. Padded seq: ', int_seq.numpy()[0])
print('10. Predicted category:', predicted_class)

Output

 

Great job!

You have successfully cleaned, tokenized, and normalized 420 SmartCart customer queries/reviews, converted text into Bag-of-Words, TF-IDF and padded integer sequences, explored learned word vectors using an Embedding layer and performed an end-to-end text analysis on a new customer query

Checkpoint

   Git Push

git push origin branchName

Next-Lab Preparation

Topic : Recurrent Neural Network

1) Understand RNN fundamentals and hidden-state mechanism.

2) Learn different types of RNN architectures.
3) Understand the sequential processing bottleneck.
4) Explore the vanishing gradient problem.