Stance detection with
Weakly-Supervised NLP
Automatic animations between code
Dr. Srijith Rajamohan
Problem
-
Natural Language Understanding for determining political affiliation
- Understanding intent is difficult
-
Augmented Intelligence
- Human-in-the-loop: Results of DNNs used to create projections that assist humans in classifying documents
-
Interpretable
- Self-attention gives insight into the decision-making process of a DNN
Problem
- Can we minimize the human effort for labeling?
- Are there certain types of dimensionality reduction techniques that result in better projections for this purpose?
- How effective are embeddings at learning intent from noisy text?
Solution Overview
Use PySpark to clean the data
Project affiliation in a 2D space similar to a form of Aspect-Based Sentiment Analysis (ABSA)
Self-attention based BiLSTM with pretrained static and contextual embeddings (Elmo)
Evaluate visualization/cognitive efficiencies of various dimensionality reduction techniques
Interactive web application to help correctly label this weakly-supervised data
Gather social media posts related to certain political hashtags, along with user metadata
Data Ingestion Pipeline
- VT cloud server
- 18 cores,192GB RAM, 200 + 500GB volume
- Conda environments for package management
- Python RQ
- Redis-based framework for job scheduling
- Uses Tweepy for interaction with Twitter
- Downloads and stores tweets corresponding to certain hashtags in timestamped files
- MongoDB setup for interacting with the data
- Metabase used as a dashboard for this DB
- Interactive filtering, visualizing and exploratory analysis from local machine
Data Cleaning
- Weakly-supervised data
- Noisy labels because tweet hashtags do not correspond to affiliation
- PySpark for data preprocessing
- John Snow Labs Sparknlp library
- Data cleaning
- Remove hyperlinks, emoticons
- Remove entries with empty Twitter User Descriptions (TUDs)
- No stemming or lemmatization was done
- Not recommended for contextual embeddings
Aspect-Based Sentiment Analysis
- Different from sentiment analysis
- Learn an entity’s opinion towards a number of topics that constitute a person’s political affiliation here
- Different from simply expressing a positive or negative sentiment
- ABSA also referred to as stance detection
- Frame this as inter-point distance in n-dimensional space
- Stance detection performed on Twitter User Description
- Training done on these TUDs
Training
- PyTorch code runs on GPUs
- 4 Volta GPU node with 16GB per-GPU memory
- Workflow automated with Airflow
- PyTorch code:
- Model training
- Generates metrics for accuracy, F1-score and ROC scores
- Dimension reduction for visualization
- Plot.ly used for generating the graphs
- Generating the plots from the metric files generated by PyTorch
- PyTorch code:
- Hyperparameter optimization done with Comet.ml
Corpus Statistics
- About 120k documents in the training/validation/test corpus
- Imbalanced dataset: 90k/30k for classes
- Torchtext module
- Use F1-score as the metric for assessment
Network Architecture
Embeddings
- Static embeddings
- Glove
- Glove.twitter
- Charngrams
- 100-dimensional embeddings
- Contextual embeddings
- Elmo
- Full training data
- All the data we have
- Limited training data
- Randomly sampled 5k training set
- Hand-curated set of 1219
General Workflow
Hyperparameter Optimization
APP DEMO
Projections - Interaction
- Entities are color-coded to represent the two ideologies
- Projections are color and opacity coded for affiliation in corpus, and strength of classification respectively
- Proximity in the two-dimensional space can be used to represent relative ’political affiliation’ without having to quantify it subjectively.
- Types of identifiable errors
- Type a error due to weakly-supervised label inaccuracy
- Type b error due to inaccuracy of the predicted label by the model
- Outliers: do not belong in that particular group
ELMo (Embeddings from Language Models)
- Contextual embeddings that address polysemy, used to capture semantic meaning
- Each token embedding is a function of the entire sentence
- Bidirectional language modeling
- Character-based and can use morphological cues to form representations of out-of-vocabulary words
- Shown to work well on small datasets
Active Learning
- Identify misclassifications as outliers -
- "Use supervision whenever possible"
- Active learning used to correct labels -
- Done by expert who is provided guidance on where to look
- In the MetaCost[ref] algorithm, normal class elements that have a reasonable probability are relabeled
- Cost-based relabeling can worsen the error
- Do this visually using the entity/cluster proximity
- How we define reasonable probability is the key...
- Entities with the greatest uncertainty
A Note on Softmax Scores
- On using softmax scores as a confidence score
- Softmax score as a 'correctness likelihood'
- Calibration is not done here
- Ideally reliability curves should be plotted
- There is some empirical evidence of the efficacy of naive use of softmax scores
A Note on Softmax Scores
- On using softmax scores as a confidence score
- Softmax score as a 'correctness likelihood'
- Calibration is not done here, ideally reliability curves should be plotted
- There is some empirical evidence on the efficacy of naive use of softmax scores
-
However, we don't need an exact estimate here
- Neural networks with softmax tend to overestimate the confidence of their predictions
- We just need to know that the network is not confident, i.e. anything that is not a high value is of interest
Incorrect Classification - Type (a)
The classifier prediction correctly identifies the true label - incorrect document label
Incorrect Classification - Outlier
Classifier output is incorrect, but the projection coordinates identify the incorrect document label
Attention
- Given a sentence of 'n' words and each word represented by 'd' dimension, each sentence is now a matrix (n,d)
- Hidden layer outputs of BiLSTM (u) for a sentence has output 'H'(n,2u)
- Linear combination of the 'n' vectors to form the attention vector 'm' (1,2u) and attention weights vector 'a' (1,n)
- 'da' is a hyperparameter
- w_{s2} is a vector of size 'da'
- W_s1 (da,2u)
- These weights are learned through training
\textbf{a} = softmax(w_{s2} tanh( W_{s1} H^T)) \\
\textbf{m} = \textbf{a} H
Attention Layer for Interpretability
Weights correspond to the inferred importance of the words for the classifier
Dimensionality Reduction for Visualization
- PCA
- linear dimensionality reduction with vanilla PCA
- MDS
- Low-dimensional projections that attempts to maintain the inter-point distance in high-dimensional space
- Similarity calculated using euclidean distance
- Isomap
- Similar to MDS, similarity calculated with geodesic distances
- Can capture non-linear manifold structure
- t-SNE
- Similarities as joint probabilities
- Good separation of points
MDS
- MDS
- Low-dimensional projections that attempts to maintain the inter-point distance in high-dimensional space
- Similarity calculated using euclidean distance (other distance metrics can also be used here)
- If [X] is the high dimensional matrix of observations
- $[Z]$ is a low-dimensional we hope to obtain through MDS.
\textbf{Stress}^2 = \sum_{i=1}^{nrows} \sum_{j=i+1}^{nrows}
\biggl( ([ X_i - X_j ] [X_{i} - X_{j}]^{T} - [Z_i - Z_j][Z_{i} - Z_{j}]^T) \biggr)
Isomap
- Similar to MDS
- Geodesic distance is used to compute similarities of points
- Helps to capture the curvature of the manifold
- Approximate distances since we don't know the true manifold
- Build a neighborhood graph using the following formula
- Using the nearest neighbor graph, use Dijkstra's algorithm to find the shortest path for pairs
- Apply MDS using the above computed distances
t-SNE
- Similarities as joint probabilities
- Minimize the KL divergence between high-d (P) and low-d (Q) joint probabilities
- For N points, and a high-d space represented by the vector X and low-d space represented by Y
- The joint probabilities in high-d space to measure similarity
- Sigma is the bandwidth of the gaussian kernel
e^{-d(x_i, x_j)^2 / 2\sigma_i^2} / \sum_{k \neq i}e^{-d(x_i, x_k)^2 / 2\sigma_i^2} \\ \\
p_{ij} = \dfrac{p_{i|j} + p_{j|i}}{2N}
q_{ij} = \dfrac{(1 + \lVert y_i - y_j \rVert^2)^{-1}}{\sum_{k \neq l}(1 + \lVert y_l - y_k \rVert^2)^{-1}}
The heavy tails of the Student-t kernel allow you to transform the small inter-point distances in high-d to points farther apart in low-d (good separation)
Dimensionality Reduction Techniques
1. PCA
2. MDS
3. Isomap
4. t-SNE
Results
Ansible Notebooks for Deployment
# Python code
#
- hosts: all
tasks:
- name: ping all hosts
ping:
- name: Supervisor install
become: yes
apt:
name: supervisor
state: latest
tags:
- supervisor_installMake your code shine with highlighting + Avbcuto-Animate
Part II - Graph Analytics on Social Networks
Mapping Right-wing Extremism
- Network mapping of right-wing figures and their supporters
- Identify influential associates and communities
- First (RQ, Tweepy) and second degree network information
- About 1 million files spread across 24 folders
- Graph of about 72 million nodes and 1 billion edges
Network Visualization
Make your code shine with highlighting + Avbcuto-Animate
First degree network
Second degree networks: Incremental visualization of large networks
Network Visualization
Exploratory analysis: Identifying general trends in network relationships
Graph Visualization - Top 100
Workflow
Read 1 million txt files of friends and followers info across 24 folders
Generate edges and extract metrics
Exploratory analysis and visualizations
Incremental visualization of network in graphtools
Compute Pagerank and centrality measures for all nodes
Interactive filtering of Pagerank results in Pyspark shell
Visualization of the subgraph generated
spark = SparkSession(sc)
sc.setLogLevel("INFO")
sqlcontext = pyspark.sql.SQLContext(sc)
data = read_input()
df = get_edges(data)
df = swap_edges_for_relationship(df)
v_name, userinfo = get_vertices(data)
g = GraphFrame(v_name, df)
results = g.pageRank(resetProbability=0.01, maxIter=1)
go = g.outDegrees
gi = g.inDegrees
gd = g.degrees
g_degree = gi.join(go, on='id', how='full').join(gd, on='id', how='full')GraphFrames: Performing Pagerank
res_read_cache = res_read_cache.withColumn('friends_of_central_figure',
find_elem_fr_udf(res_read_cache.id))
res_read_cache_filter = res_read_cache.filter(res_read_cache
['friends_of_central_figure'].isNotNull())
res_read_cache_filter = res_read_cache_filter.cache()
... operation to trigger caching ...
# Faster interactive querying
res_read_cache_filter.sort(size(col("friends_of_central_figure")),
ascending=True).show()
res_read_cache_filter.sort(size(col("friends_of_central_figure")),
ascending=False).show()
# Send output files
rclone sync /home/vt/page_rank_mapped_query_single_reordered.out
remote_google:liuqing_processed/query_reordered -v
Fast Interactive Querying
res_read_cache = res_read_cache.withColumn('friends_of_central_figure',
find_elem_fr_udf(res_read_cache.id))
res_read_cache_filter = res_read_cache.filter(res_read_cache
['friends_of_central_figure'].isNotNull())
res_read_cache_filter = res_read_cache_filter.cache()
... operation to trigger caching ...
# Faster interactive querying
res_read_cache_filter.sort(size(col("friends_of_central_figure")),
ascending=True).show()
res_read_cache_filter.sort(size(col("friends_of_central_figure")),
ascending=False).show()
# Send output files
rclone sync /home/vt/page_rank_mapped_query_single_reordered.out
remote_google:liuqing_processed/query_reordered -v
Send Results
Work in progress...
Tweet evaluation
- Cardniffnlp Tweeteval
- Twitter-roberta-base masked language model embeddings
- Task-specific embeddings
- Four tasks were evaluated
- Hate speech
- Offensive tweets
- Emotion recognition
- Pytorch Distributed Data Parallel code using HuggingFace
Thank you!
Myslides
By sjster
Myslides
Presentation slides for Stance detection
