Content ITV PRO
This is Itvedant Content department
Language Architecture
Text preprocessing and Vectorization
Learning Outcome
4
Compare Count Vectorizer, TF-IDF and Word2Vec for turning text into vectors.
Differentiate between Stemming (chopping) and Lemmatization (contextual root-finding).
3
2
Apply tokenization and stop word removal to filter out linguistic noise
1
Explain why raw text must be cleaned before entering a ML model.
5
Pipe vectors into an ML algorithm for text classification.
The "Tweet" Dilemma
Look at these two sentences:
-----------------------------------------------------
| RAW HUMAN TEXT | CLEAN MACHINE TEXT |
-----------------------------------------------------
| "The SpaceX Falcon 9 launch | the spacex falcon 9 |
| was INCREDIBLE!!! 🚀 #Space" | launch was incredible|
| | space |
-----------------------------------------------------“Human Expression (Noisy Data)”
- CAPS ("INCREDIBLE!!!")
- Emoji 🚀
- Hashtag #Space
👤
Text Preprocessing Pipeline
🤖“Machine-Ready Data (Clean Text)”
- Plain lowercase text
- No punctuation
- No emoji
- Clean black font
“Human Expression (Noisy Data)”
- CAPS ("INCREDIBLE!!!")
- Emoji 🚀
- Hashtag #Space
👤
Text Preprocessing Pipeline
🤖“Machine-Ready Data (Clean Text)”
- Plain lowercase text
- No punctuation
- No emoji
- Clean black font
🔤 (lowercase)
❌ ! ? (remove punctuation)
🚫 😊 (remove emoji)
#❌ (remove hashtags)
Machines cannot understand emotion, tone, or emphasis — It only understand patterns and tokens.
Let’s move to understand Preprocessing.
We need a systematic way to strip away the "human emotion" and leave only the "core data."
This is where preprocessing comes into the picture.
The Preprocessing Pipeline
The Flow of Data:
Text preprocessing is not just one action; it is a step-by-step assembly line.
Inside the Concept
from nltk.tokenize import word_tokenize
text = "SpaceX is going to Mars!"
print(word_tokenize(text))
# Output: ['SpaceX', 'is', 'going', 'to', 'Mars', '!']
from nltk.tokenize import word_tokenize
text = "SpaceX is going to Mars!"
print(word_tokenize(text))
# Output: ['SpaceX', 'is', 'going', 'to', 'Mars', '!']
Stop Words Removal (Filtering the Noise)
The words like “is,” “the,” “and,” “a” appear frequently but add little meaning for ML models, so we remove them to save computational power.
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
clean_tokens = [w for w in tokens if w.lower() not in stop_words]
Stemming (The Blunt Axe)
Reducing words to their base or root form.
It uses strict, rule-based chopping. It just cuts off the end of words (like "-ing" or "-ed").
- Running -> Run
- Caring -> Car (Wait, "Car"? Yes, Stemming makes mistakes!)
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
print(stemmer.stem("running")) # Output: run
a
Lemmatization (The Smart Dictionary)
It reduces words to their root form (the "Lemma"), but uses a vocabulary and morphological analysis (context).
How it works: It knows that "better" is actually the root of "good", and "caring" comes from "care", not "car".
a
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
print(stemmer.stem("running")) # Output: run
Comparison: Stemming vs. Lemmatization
Which one to choose?
- Stemming: Fast, computationally cheap, but sometimes produces non-words (e.g., trouble -> troubl). Best for massive datasets like search engines.
- Lemmatization: Slower, computationally expensive, but produces actual dictionary words based on context. Best for Chatbots and Sentiment Analysis.
| Original Word | Stemming Output (Axe) | Lemmatization Output (Brain) |
|---|---|---|
| Caring | Car ❌ | Care ✅ |
| Geese | Gees ❌ | Goose ✅ |
Vectorization - Technique 1: Count Vectorizer (Bag of Words)
The simplest way to convert text. It just counts how many times each word appears.
It ignores word order and meaning, so “Good, not bad” and “Bad, not good” shows the same score.
Not all words are created equal. Counting isn't enough; we need to measure importance.
The Logic:
TF (Term Frequency): Rewards a word if it is used a lot in one specific document.
IDF (Inverse Document Frequency): Punishes a word if it is used in every document across the dataset.
Very common words get almost zero importance, while rare and specific words get higher importance.
-
Words like “the”, “and”, “is” appear in almost every sentence, so their weight becomes ~0 (very low importance).
-
Rare words like “algorithm” or “supernova” appear less often, so they get a higher score and more importance in the model.
Vectorization - Technique 2: TF-IDF (The Great Balancer)
Vectorization - Technique 3: Word2Vec (Capturing True Meaning)
Count Vectorizer and TF-IDF only count words and their frequency. They do not understand the meaning of words.
- Even though “Happy” and “Glad” have the same meaning, the model treats them as completely different
Word2Vec learns from the context of words. If words like “Happy” and “Glad” appear in similar sentences, it gives them similar numerical values.
Deep Dive: CBOW vs. Skip-Gram (The Word2Vec Engines)
Word2Vec learns meaning by playing a “fill in the blank” game using surrounding words.
There are two ways to do this.
1. CBOW (Continuous Bag of Words):
Predicts the target word from surrounding words.
Example: “The [---] sat on the mat.” → “Cat”
2. Skip-Gram:
Predicts the surrounding words from the target word.
Example: “Cat” → “The”, “sat”, “on”, “the”, “mat”
Bringing it Together: Text Classification
Once the text is converted into numbers (vectors), we use them to classify the data.
The Pipeline:
We feed the text vectors into a Machine Learning model (like Naive Bayes or Logistic Regression) to make a decision.
-
Sentiment Analysis: Positive (1) or Negative (0)
-
Spam Detection: Spam (1) or Ham (0)
Crucial Step:
Always evaluate the model using Accuracy, Precision, and Recall before deploying it.
Real-World Applications
Spam Email Detection: Protecting your inbox using TF-IDF and classification.
Chatbots & Search Engines:
Google uses dense vectors to understand search intent, not just keywords.
Movie / Product Reviews: E-commerce sites automatically sorting reviews by sentiment (Positive/Negative).
Shopping Recommendation Systems:
Suggest products based on the meaning of what you browse or read.
Summary
5
Word2Vec captures context, and the vectors power text classification.
4
TF-IDF boosts rare, meaningful words.
3
Count Vectorizer tallies frequency.
2
Clean tokens are converted to numbers (vectorization).
1
Raw text is cleaned: lowercase, tokenize, remove stop words, lemmatize
Quiz
Why do embeddings outperform Bag of Words for understanding meaning?
A. They count words more accurately
B. They capture word meaning
C. They remove stop words
D. They clean the raw text
Quiz
Why do embeddings outperform Bag of Words for understanding meaning?
A. They count words more accurately
B. They capture word meaning
C. They remove stop words
D. They clean the raw text
GenAI - Text Preprocessing & Vectorization
By Content ITV
