IBM Watson Natural Language Processing

Upkar Lidder

Developer Advocate, IBM

> ulidder@us.ibm.com

> @lidderupk

> upkarlidder.com

IBM Developer

IBM Signup - http://bit.ly/hacker-ibm

Workshop Material - http://bit.ly/hacker-nlp

Wifi - HD-Events / oz5l8x

Prerequisites

@lidderupk

IBM Developer

1. Create IBM Cloud Account using THIS URL

3. If you already have an account, use the above URL to sign into your IBM Cloud account.

2. Check your email and activate your account. Once activated, log back into your IBM Cloud account using the link above.

http://bit.ly/hacker-ibm

Call for Code 2019

@lidderupk

IBM Developer

Call for Code 2019 - Get started quickly

https://developer.ibm.com/callforcode/

```
Background
```
```
Research
```
```
Data
```

```
Videos
```
```
Solutions
```
```
Blogs/Articles
```
```
Code patterns
```

@lidderupk

IBM Developer

IBM Code Patterns

@lidderupk

IBM Developer

NL on IBM Cloud

@lidderupk

IBM Developer

Watson Assistant

Watson Assistant lets you build conversational interfaces into any application, device, or channel.

Watson Studio

Everything AI and ML with IBM. Project based development environment with access to assets like Cloud Object Store, Machine Learning Services, AutoAI, SPSS, Jupyter Notebooks, etc.

Discovery

Cognitive search and content analysis in your unstructured documents.

Natural Language Classifier

Create custom classifier for your domain data. You create a classifier instance by providing a set of representative strings and a set of one or more correct classes for each training.

NL on IBM Cloud

@lidderupk

IBM Developer

Watson Studio & Watson Machine Learning

@lidderupk

IBM Developer

Watson Studio

IBM Watson Studio

@lidderupk

IBM Developer

IBM Watson Studio

@lidderupk

IBM Developer

Watson Custom Speech to Text Model

@lidderupk

IBM Developer

https://github.com/IBM/Train-Custom-Speech-Model

Workshop - Goals

@lidderupk

IBM Developer

Successfully Create, Store and Deploy a Natural Language Classifier Model on IBM Cloud using Watson Studio and Watson Machine Learning Services.

The Classifier will categorize an email as ham or spam.

Question - ham or spam

@lidderupk

IBM Developer

Congrats! 1 year special cinema pass for 2 is yours. call 09061209465 now! C Suprman V, Matrix3, StarWars3, etc all 4 FREE! bx420-ip4-5we. 150pm. Dont miss out!

Aft i finish my lunch then i go str down lor. Ard 3 smth lor. U finish ur lunch already?

Will purchase d stuff today and mail to you. Do you have a po box number?

I won a coupon for $400. I am happy.

Question - ham or spam

@lidderupk

IBM Developer

Congrats! 1 year special cinema pass for 2 is yours. call 09061209465 now! C Suprman V, Matrix3, StarWars3, etc all 4 FREE! bx420-ip4-5we. 150pm. Dont miss out!

Aft i finish my lunch then i go str down lor. Ard 3 smth lor. U finish ur lunch already?

Will purchase d stuff today and mail to you. Do you have a po box number?

I won a coupon for $400. I am happy.

Supervised Learning - or Unsupervised Learning

@lidderupk

IBM Developer

Our model by end of the workshop - ham or spam

@lidderupk

IBM Developer

Preprocessing for NLP - remove "noise"

@lidderupk

IBM Developer

Tokenization

Input: Friends, Romans, Countrymen, lend me your ears;

Friends

Output:

Romans

Countrymen

lend

your

ears

Remove Stop Words

https://nlp.stanford.edu/IR-book/html/htmledition/contents-1.html

Noise

<div class="sl-block-content" data-placeholder-tag="p" data-placeholder-text="Text" style="z-index: 15;">
   <p>
      <span style="font-size:1.0em">
         <span style="text-align:start">Input: Friends, Romans, Countrymen, lend me your ears; </span>
      </span>
   </p>
</div>

Preprocessing for NLP - remove "noise"

@lidderupk

IBM Developer

Stemming & Lemmatization

car, cars, car's, cars'

car

am, are, is

https://nlp.stanford.edu/IR-book/html/htmledition/contents-1.html

Vectorizing - models do not understand text vectors

@lidderupk

IBM Developer

Bag of Words

Term Frequency

tf-idf - Term Frequency, Inverse Document Frequency

raw count of a term in a document, i.e., the number of times that term toccurs in document d

The inverse document frequency is a measure of how much information the word provides, i.e., if it's common or rare across all documents.

But First - prepare the data

@lidderupk

IBM Developer

import pandas as pd

# create a dataframe from the data file
df = pd.read_csv('SpamHam-Train-new.csv')

# extract the spam and ham data into separate dataframes
df_spam = df[df['Ham-Spam'] == 'spam']
df_ham = df[df['Ham-Spam'] == 'ham']

# take out 5 examples to test for spam
df_test_spam = df_spam.sample(5)
df_spam = df_spam.drop(index=df_test_spam.index)

# take out 5 examples to test for ham
df_test_ham = df_ham.sample(5)
df_ham = df_ham.drop(index=df_test_ham.index)

# print out the number of rows in each frame
print('#rows in df_spam: ' + str(df_spam.shape[0]))
print('#rows in df_test_spam: ' + str(df_test_spam.shape[0]))
print('#rows in df_spam: ' + str(df_ham.shape[0]))
print('#rows in df_test_spam: ' + str(df_test_spam.shape[0]))

# save each data frame to a file of it's own
df_spam.to_csv('./spam_train.csv', index=False)
df_test_spam.to_csv('./spam_test.csv', index=False)
df_ham.to_csv('./ham_train.csv', index=False)
df_test_ham.to_csv('./ham_test.csv', index=False)

#rows in df_spam: 668
#rows in df_test_spam: 5
#rows in df_spam: 4320
#rows in df_test_spam: 5

#rows in df_spam: 668
#rows in df_test_spam: 5
#rows in df_spam: 4320
#rows in df_test_spam: 5

Steps

@lidderupk

IBM Developer

Sign up / Log into IBM Cloud - http://bit.ly/hacker-ibm
Create Watson Studio Service.
Sign into Watson Studio and create a new Data Science Project. It also creates a Cloud Object Store for you.
Upload csv data to your project - spam and ham.
Add a new Natural Language Classifier to your project.
Create a NLC model.
Add labelled classes and training data to the model.
Test your model in Jupyter Notebook !

Step 1 - sign up/ log into IBM Cloud

@lidderupk

IBM Developer

http://bit.ly/hacker-ibm

Step 2 - locate Watson Studio in Catalog

@lidderupk

IBM Developer

Step 3 - create Watson Studio instance

@lidderupk

IBM Developer

@lidderupk

IBM Developer

Resources List - If you want to look at all your services

Step 4 - launch Watson Studio

@lidderupk

IBM Developer

Step 5 - create a new project

@lidderupk

IBM Developer

Step 6 - pick Standard project

@lidderupk

IBM Developer

Step 6a - pick US South as region if asked

@lidderupk

IBM Developer

Step 7 - give the project a name and assign COS

@lidderupk

IBM Developer

Step 7a - you may have to add a new storage from this page

@lidderupk

IBM Developer

Step 8 - open asset tab

@lidderupk

IBM Developer

Step 9 - drag and drop data file into Load Assets

@lidderupk

IBM Developer

You will need ham-train and spam-train files

Step 10 - add Natural Language Classifier model to the project

@lidderupk

IBM Developer

Step 10a - add Natural Language Classifier service

@lidderupk

IBM Developer

Step 11 - create ham and spam classes

@lidderupk

IBM Developer

Step 12 - add training data

@lidderupk

IBM Developer

Step 13 - train the model

@lidderupk

IBM Developer

Step 14 - service credentials for the Notebook from NLC service

@lidderupk

IBM Developer

Step 14 - service credentials for the Notebook from NLC service

@lidderupk

IBM Developer

{
  "classifier_id": "2ef34ex558-nlc-1192",
  "url": "https://gateway.watsonplatform.net/natural-language-classifier/api/v1/classifiers/2ef34ex558-nlc-1192",
  "collection": [
    {
      "classes": [
        {
          "class_name": "ham",
          "confidence": 0.9951709163892192
        },
        {
          "class_name": "spam",
          "confidence": 0.004829083610780819
        }
      ],
      "text": "Let's go to Alaska this weekend !",
      "top_class": "ham"
    },
    {
      "classes": [
        {
          "class_name": "spam",
          "confidence": 0.97534853311385
        },
        {
          "class_name": "ham",
          "confidence": 0.02465146688614998
        }
      ],
      "text": "You have won a trip to Alaska !!",
      "top_class": "spam"
    }
  ]
}

Output from Notebook

@lidderupk

IBM Developer

Optional - web application using classifier

@lidderupk

IBM Developer

https://github.com/IBM/nlc-email-phishing

Optional - web application using classifier

@lidderupk

IBM Developer

Workshop

Thank you

Let's chat !

@lidderupk

IBM Developer

Upkar Lidder, IBM

@lidderupk

https://github.com/lidderupk/

ulidder@us.ibm.com

IBM Developer SF

https://www.meetup.com/IBM-Developer-SF-Bay-Area-Meetup/

NLP - IBM Watson Workshop

By Upkar Lidder

NLP - IBM Watson Workshop

1,062

Upkar Lidder

Upkar Lidder is a Full Stack Developer and Data Wrangler with a decade of development experience in a variety of roles. He can be seen speaking at various conferences and participating in local tech groups and meetups.

IBM Watson Natural Language Processing

IBM Developer

Prerequisites

Call for Code 2019

Call for Code 2019 - Get started quickly

IBM Code Patterns

NL on IBM Cloud

Watson Assistant

Watson Studio

Discovery

Natural Language Classifier

NL on IBM Cloud

Watson Studio & Watson Machine Learning

Watson Studio

IBM Watson Studio

IBM Watson Studio

Watson Custom Speech to Text Model

Workshop - Goals

Question - ham or spam

Question - ham or spam

Supervised Learning - or Unsupervised Learning

Our model by end of the workshop - ham or spam

Preprocessing for NLP - remove "noise"

Preprocessing for NLP - remove "noise"

Vectorizing - models do not understand text vectors

But First - prepare the data

Steps

Step 1 - sign up/ log into IBM Cloud

Step 2 - locate Watson Studio in Catalog

Step 3 - create Watson Studio instance

Resources List - If you want to look at all your services

Step 4 - launch Watson Studio

Step 5 - create a new project

Step 6 - pick Standard project

Step 6a - pick US South as region if asked

Step 7 - give the project a name and assign COS

Step 7a - you may have to add a new storage from this page

Step 8 - open asset tab

Step 9 - drag and drop data file into Load Assets

Step 10 - add Natural Language Classifier model to the project

Step 10a - add Natural Language Classifier service

Step 11 - create ham and spam classes

Step 12 - add training data

Step 13 - train the model

Step 14 - service credentials for the Notebook from NLC service

Step 14 - service credentials for the Notebook from NLC service

Output from Notebook

Optional - web application using classifier

Optional - web application using classifier

Thank you

Let's chat !

NLP - IBM Watson Workshop

More from Upkar Lidder