Algorithm Challenge

MQ Tutors Association

First Approach

Machine Learning (unsupervised)
nltk library
trained bullying data from kaggle
LIWC emotion analysis (tweets are too short!)

The Data

1,000,000 tweets

tweets per user:

3 users with 5 tweets
12 users with 4 tweets
122 users with 3 tweets
2809 users with 2 tweets
996899 users with just 1 tweet

Retweets!

RT @featureswitches: Our exclusive offer for #earlyadopters is coming soon on @getworm #lovethyuser
RT @sr_newsuk: Study shows human sacrifice was less likely in more equal societies
RT @dwthood: @journeymanhisto Right! Combining new "science" with older paternalism? Both modern and not modern?'
RT @erikw: Such a great tool to analyze my #socialmedia activities! Find @kukupost on @GetWorm
RT @ivebeenkirked: @SethFarre no not really more science fiction/comedy/romance. Haha. Perks of Being a Wallflower, About Time &amp; Kill your'

Filtering Approach

Data Preparation:

500k tweets
remove retweets
remove bot-like accounts
ended up with 40% of the tweets

First Filter:

Sentiment Analysis

choose tweets with more negative than positive words
used word lexicon by Liu and Hu, containing 6800 positive and negative words
(https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html#lexicon)

Second Filter:
Target-based Approach

Identify groups of traditionally vulnerable minorities, based on ethnicity, religion, or sexual orientation
Custom-made dictionary containing these identities

asshat

asshole

bitch

chink

cholo

coon

cultist

Custom Wordlist

faggot

freak

fundie

gay

jap

jew

moslem

mussie

etc

Output

23936 Negative Tweets

99858 Neutral Tweets

485 Negative and Insulting

191 Targeted Bullying

4 Attacking Ethnicity

22 Attacking Orientation

4 Attacking Religion

maybe if u werent such an annoying c*** u would of got further in the competition at least is a funny c***

"gay people are gastapo definitely no abusive gay political ampamp activist orgns like stonewall emphatically yes"

"youre an idiot atheists have only one collective position there is no god"

"how is it possible that you only have 908 followers this gay boy from sydney aches 4 u wadeinsydney"

in no particular order he is sexist racist bigot lying douchebag ampamp a moron from what ive read so far

"no you moron most were german citizens and what do you have against you male refugees fleeing forced conscription"

youre the king of crap because of 1 moron its time to attack the bulldogs clubhow do we you or

"youre always a unicorn nnbut instead of a horn its a functioning dick im calling you a dickhead"

forbzy you always have been a funny c*** haha I keep laughing at curry hahah

I think i am an idiot I didnt realise they still used the lucastilm brand they just own it

I mean hell be insane after he finds out what jon and rob did on ark so

CENSORED