Building Smart Bots

Hi!

Agenda

• My first (dummy) bots

• Initial attempts

• Cassandra helping to colect data

• Spark helping to analyse it

• Machine Learning making smart bots

• Epic Fails

• Resources

Why bots?

Several actions can be automated

Examples

• Chat responses
• Reminders
• Default answers
• Discard unimportant emails
• Social media feed
• Collect GIFs
• Announce meetups
• etc

My first (dumb) bots

• Cron job on Linux
• Periodically send troll messages in Turkish to undergrad students mail group (admins thought system had been hacked)

The troll-less case: Hangouts

People talking to me

• Ask questions
• Ask more questions!
• Randomly talk to me, since I was a teacher at the time

Options

• Impolitely ignore
• Spend too much time answering

How can I automate a chat?

Take #1

Automatically respond to everyone

Évariste

Nope.

Create contextual answers.

Take #2

Évariste

It's better.

Évariste

if {
} else {
  if {
  } else {
    if {
    } else {
      if {
...

Take #3

Refine the cases and answers

Need something smarter to handle the answers.

Mom

Take #4

Select people to *not* generate automatic answers. ("whitelist")

New Contacts

Take #5

It's a lot of information

Brute force may not be good.

You can loose a lot of info

Évariste

Analyse previous chats and copy behaviours

Take #6

Read all my chat records and collect data

Cassandra can store this information <3

chat_by_category

chat_by_language

chat_by_allow

contacts

contact_tracking

Cool! I have data! 

Quick Q/A

Quick Q/A

Quick Q/A

API reads chat

Hangouts

Parse and create objects

Write in Cassandra

Quick Q/A

API reads chat

Hangouts

Parse and create objects

Write in Cassandra

New messages - Improving

Reads from Cassandra

Produces enhanced object 

Response

Queue

Queue

Quick Q/A

Great. How can I automate this EVEN MOAR?

Évariste

Évariste

Évariste

The more information I collect, the more precise are the responses

Too much data for my "home-made" analyser

Spark comes to rescue :)

Take #7

Analyse the information that I have on Cassandra with spending too much time on boilerplate stuff

Instant benefits from Spark

• Simplify my ugly parsers

API reads chat

Hangouts

*Raw* data to spark

Write raw in Cassandra

text.flatMap(line => line.split(" ")).
map(word => (word, 1)).
reduceByKey(_ + _)

Instant benefits from Spark

• Simplify my ugly parsers

Significant decrease for writing time!

Source: http://www.slideshare.net/EvanChan2/cassandra-day-2014-interactive-analytics-with-cassandra-and-spark

Instant benefits from Spark

• Simplify my ugly code to categorise

API reads chat

Hangouts

*Raw* data

Write raw in Cassandra

Good.

Évariste - Monday, 8:45

Évariste - Wednesday, 7:45

Hey Hanneli, sup?

What time is the meetup?

Hi

Évariste - Wednesday, 7:45

7PM

Where is it?

Paulista Ave, 2001

He always asks where is the meetup.

Hey Hanneli, sup?

What time is the meetup?

Hi

Évariste - Wednesday, 7:45

Learn to improve the answers.

Machine Learning

Take #8

Machine Learning

Take #8

Supervised model

Predict - He asked "what time", he will ask "where"

Person started conversation complaining, he/she might be angry

Reinforcement

Person started conversation with trivial question that could be found on Google. Should I answer or ignore it? 

Mathematics to make things better

• Regression models
• Generate equations to parametrise behaviours
• Matrixes
• Analytical Geometry
• Topology (!!)

Which topics are you considering?

• Have I had any previous contact with the person on the internet?
• Does this person have Twitter? (quick search before deciding; 123.people is a good start)
• Does it look like Spam? 
• Is he/she trying to contact me in other social media?
• Did he/she try to send me emails?
• Does the message have any blacklisted sentences? *

There is no default cookbook for the algorithms, but the idea may be applied in other areas

With Machine Learning

API reads chat

Hangouts

*Raw* data

Write raw in Cassandra

filter and process with Spark

Produces enhanced object 

Queue

Queue

MOAR bots

Take #9

Meetups

• Answer messages
• Tweet and announce
• Find venues
• Schedule new events

Motivational bots

• Send positive messages!

Github activity bot

• Monitor my commits, terminal actions
• Monitor my new PRs, automatically comments on some tasks on Pivotal
• Monitor build status
• Comments on PRs

Email responder

• Automatically answer some emails
• Search for new Spam
• Automatically create filers to Ads

Epic Fails

Take #10

Because bots are not so perfect (yet)

API reads chat

Hangouts

*Raw* data to spark

Write raw in Cassandra

filter and process with Spark

Produces enhanced object 

Response

Queue

Queue

Take #11

Swiping the endpoint addresses by mistake.

API reads chat

Hangouts

*Raw* data to spark

Write raw in Cassandra

filter and process with Spark

Produces enhanced object 

Response

Queue

Queue

Take #12

Weird answers. Friend called Paul Regus (PR)

Take #13

Trust Problem

Hey Hanneli

Hi

It that you?

Yes

How can I be sure it's not the bot?

It is not

Take #14

Topics and references

• XMPP
• Apps Script
• Parser and Regex
• Spark + Cassandra course
• Presentation
• ML free content
• Quora Thread for ML
• Event driven, notifications.
• Math and computer theory may be helpful

Take #15

Discussion points

• Security - it is sensitive information! OAuth, cryptography and 2FA are a must.
• Bias - How can you be sure you are being impartial? Should we be impartial all the time? 
• Remove all possible sensitive info and open source it; how can I determine if all algorithms that are specific to myself were removed?
• Migrate legacy code to new languages
• Functional Programming makes things much more clear
• Human factors - people get super sad/angry when they find out it was a bot.

Special Thanks

• @planetcassandra
• @lafp, @romulostorel        and @pedrofelipee       (GIFs)
• Bots <3

Thank you :)

Questions?

hannelita@gmail.com

@hannelita