An Introduction to Natural Language Processing

Anjali Bhavan

What is NLP?

Applications

1. Spell check

2. Chatbots

3. Search engines

4. Voice assistants

5. Translation

 

 

6. Spam filtering

7. Advertising

8. Governance

9. Social media: analytics, detection etc.

Some Common NLP Tasks

1. Speech Recognition

2. Sentiment Analysis

3. Topic Modeling

4. Language generation

5. Conversational Systems

6. Part of Speech Tagging

7. Named Entity Recognition

8. Speech/text conversion

9. Question Answering

10. Text Classification

NLP System Pipeline

1. Preprocessing and analysis of raw text/speech

2. Extracting features and information

3. Building models for learning extracted features

Preprocessing

1. Tokenization

2. Stop words/punctuation removal

3. Stemming

4. Lemmatization

5. Vectorization

Feature Extraction

1. Various numeric features can be extracted

2. As can word-based features!

 

Model Building

Common NLP Toolkits/Libraries

1. NLTK

2. SpaCy

3. AllenNLP

4. Pytorch

5. Huggingface

6. Tensorflow

 

 

 

7. CoreNLP

8. Gensim

Language Modeling

Limitations, Consequences etc.