Algorithm Challenge

MQ Tutors Association

First Approach

• Machine Learning (unsupervised)
• nltk library
• trained bullying data from kaggle
• LIWC emotion analysis (tweets are too short!)

The Data

1,000,000 tweets

 

tweets per user:

• 3 users with 5 tweets
• 12 users with 4 tweets
• 122 users with 3 tweets
• 2809 users with 2 tweets
• 996899 users with just 1 tweet

Retweets!

• RT @featureswitches: Our exclusive offer for #earlyadopters is coming soon on @getworm #lovethyuser
• RT @sr_newsuk: Study shows human sacrifice was less likely in more equal societies
• RT @dwthood: @journeymanhisto Right! Combining new "science" with older paternalism? Both modern and not modern?'
• RT @erikw: Such a great tool to analyze my #socialmedia activities! Find @kukupost on @GetWorm
• RT @ivebeenkirked: @SethFarre no not really more science fiction/comedy/romance. Haha. Perks of Being a Wallflower, About Time &  Kill your'

Filtering Approach

Data Preparation:

 

• 500k tweets
• remove retweets
• remove bot-like accounts
• ended up with 40% of the tweets

First Filter:

Sentiment Analysis

• choose tweets with more negative than positive words
• used word lexicon by Liu and Hu, containing 6800 positive and negative words
  (https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon)

Second Filter:
Target-based Approach

• Identify groups of traditionally vulnerable minorities, based on ethnicity, religion, or sexual orientation
• Custom-made dictionary containing these identities

asshat

asshole

bitch

chink

cholo

coon

cultist

<censored>

<censored>

 

Custom Wordlist

<censored>
<censored>

faggot

<censored>

<censored>

<censored>

freak

fundie

gay

 

jap

jew

moslem

mussie

etc

Output

23936 Negative Tweets

99858 Neutral Tweets

485 Negative and Insulting

191 Targeted Bullying

4 Attacking Ethnicity

22 Attacking Orientation

4 Attacking Religion

 

maybe if u werent such an annoying c*** u would of got further in the competition at least is a funny c***

"gay people are gastapo definitely no abusive gay political ampamp activist orgns like stonewall emphatically yes"

 

"youre an idiot atheists have only one collective position there is no god"

 

"how is it possible that you only have 908 followers this gay boy from sydney aches 4 u wadeinsydney"

in no particular order he is sexist racist bigot lying douchebag ampamp a moron from what ive read so far

 

"no you moron most were german citizens and what do you have against you male refugees fleeing forced conscription"

youre the king of crap because of 1 moron its time to attack the bulldogs clubhow do we you or

"youre always a unicorn nnbut instead of a horn its a functioning dick im calling you a dickhead"

forbzy you always have been a funny c*** haha I keep laughing at curry hahah

I think i am an idiot I didnt realise they still used the lucastilm brand they just own it

I mean hell be insane after he finds out what jon and rob did on ark so

Questions