Loading deck

Graduate Affairs Chatbot

Team

  • Rohan Kshirsagar (TL)
  • Akshay Nagpal
  • Yogeshwar Mutneja

Use case

Problem

  • Graduate Affairs deals with on-boarding graduate students into Columbia 
  • Deal with ~ 50 emails per day
  • Have two staff members
  • Tiring menial work

Sample Question 

I have submitted the I20 requests documents to your university for nearly one month.
Could you help me to check if my I20 has been sent to me or if other documents needed to apply for my I20?

 

Sample Answer

For questions related to the visa application process, your I-20, or Optional Practical Training, contact isso@columbia.edu.

Key insight

  • Many of these questions and answers are repetive
  • Students are too lazy find the answer online 
  • Current methods in Machine Learning and NLP can automate this task

Goal

  • To reduce the burden of the staff by answering a portion of these emails automatically 

Stakeholder Agreement

 

We have agreement from the Columbia Graduate Student Affairs to work on this 

Question Answer Dataset

They will provide an anonymized question answer dataset from their emails

Technical goals

  • Automatically train a conversational agent from an annotated dataset of question and answers
  • Have confidence attached to every response

Data

  • Built in Google sheets
  • ~ 200 Question/Answer pairs
  • Manually annotated for Intents and Entities
  • ~ 15 Links provided
  • Built a web scraping tool to get more data.
  • Built a feedback page to collect more data and customer feedback. 

Intent

Expresses the main goal or theme of the statement 

Entities

key parameters to the intent

Is there a deadline at which the checklist must be completed?

 

If I decide to submit my transcripts before the registration via mail, is there a deadline to it before the registration of classes?

 

Architecture

Conversation Message Flow 

Message

Preprocessing

Watson

Conversation

Intent

Entities

Answer building

Case normalization, character removal, tokenization, sentence splitting, question/ date recognition

Send response

Retrieve Message Flow 

Message

Preprocessing

Watson

Retrieve (no rank)

Candidate Links

Link Filtering

Tokenization, Extracting Noun Phrases

Return top link

e.g. Remove links from Twitter

Algorithms

  • Distinguish intents vs background
  • Entity extraction
  • Engine
  • Solr population

How do we populate Watson Conversation with intents?

  • For each (email,intent) in dataset:
    • Split email into sentences
    • for sentence in email
      • if sentence is question, output question + intent
    • if no sentence is identified as question, save email + intent
  • Link to Code

How do we populate Watson with entities?

  • Finding noun phrases using nltk
  • fuzzy string matching (e.g. facebook group and housing group)
  • word2vec (e.g. vaccination and Immunization)
  • conditional probability, frequency, mutual information stats  (e.g. housing and UAH)
  • Link to Code

Message Processing

Engine

  • Use Watson to determine entities
  • Find question in message
    • send question to Watson to determine intent
  • if no intent in question
    • send message to Watson to determine intent
  • if still no intent, return entities
  • based on intent + set (entities), determine most likely response greedily by starting with the most entities possible and removing one by one
  • Link to code

Solr Document Population

  • Determines links based on depth parameter using BFS
  • Scrapes text from links using Beautiful Soup
  • Renders json in solr format 
  • Uploads to Watson Retrieve Solr instance 

Front ends

  • FB Messenger
  • Email (Microsoft Outlook)
  • iframe (embeddable within any web page)

Facebook Messenger

Email graduateaffairsbot@outlook.com

Evaluation

Ideal case

  • User feedback
  • Analyze rate of users successfully were serviced by our bot
  • We will have users in the spring semester

Intent, Entities, Response

  • On a test sample:
  • Intent Correct if  predicted intent == labeled intent
  • Entity Correct if (predicted entities).contains(labeled entities)
  • Response Correct if fuzzy_match(predicted, labeled) > threshold

Using Watson (40 test samples)

  • Entity score 0.87
  • Intent score 0.95
  • Response score: 0.62
  • Evaluation code: https://github.com/rohankshir/graduate-affairs-bot/blob/master/evaluation.py

Sample error

  • I have been admitted to the dual MS in Journalism and Computer Science. I am an american citizen, though I have been in India for the past 14 years, so I'm a little confused as to which checklist to follow. Should I follow the International Students' checklist or the regular Students' checklist?
  • Intent Label: Misc  
  • Entity Label: Checklist

Sample intent error

  • I have been admitted to the dual MS in Journalism and Computer Science. I am an american citizen, though I have been in India for the past 14 years, so I'm a little confused as to which checklist to follow. Should I follow the International Students' checklist or the regular Students' checklist?
  • Intent Label: Misc  ==> Predicted Label: Deadline
  • Entity Label: Checklist ==> Predicted Label: Checklist

Using Nearest Neighbors

  • Preprocess input using sklearn and nltk
  • Vectorize each question and find nearest neighbor using Euclidian distance
  • Not Integrated at the moment, current score is ~ .45 on dataset with 24 intents

USER SATISFACTION QUOTIENT

  • Asked students to act as dummy users.
  • Recorded their feedback on a scale of 1 to 4 stars based on how satisfactory the response they received was.
  • 4 star: perfect answers that perfectly answer the question
  • 3 stars: partially correct answer with a relevant link.
  • 2 stars: don't answer the question but are relevant to the topic.
  • 1 star: incorrect answer.
  • Received 3 stars and above around 40% of the time

Demo

Thanks