Snorkel

Weak Supervision for NLP Tasks

 

HK ML Meetup

July 2020

Overview

• Problem Space
• Weak Supervision
• Snorkel
• Demo
• Lessons Learned
• Questions (time permitting)

• Problem: For supervised ML, collecting labels can be extremely costly and/or prohibitively time consuming

• Can we somehow encode guidelines for labeling data, and rapidly apply them to large amounts of unlabeled data?

• Potential Solution: Weak Supervision - "noisy, limited, or imprecise sources are used to provide supervision signal for labeling large amounts of training data in a supervised learning setting"

• In short: collect a bunch of "noisy" labels using low cost shortcuts and then sort out the problems that arise with this approach later (e.g., conflicts, overlaps)

 

• How to create noisy labels:
  • Encode domain knowledge from experts as labeling "rule"
  • Collect labels from mechanical turks / non-experts
  • Use related information (e.g., knowledge bases) and some knowledge transfer to label
  • Use specialized models for sub-tasks

• Python library with suite of tools to assist with weak supervision tasks; mostly focused on NLP
• Started by Alex Ratner while at Stanford University, has since grown into a very active open source project
• Used in industry[1] to great effect

• High level process:[1]
  • Incorporate domain knowledge into labeling functions
  • Resolve overlaps and conflicts with a label model
  • Use weighted labels to train final model

• Problem: Classic NLP IMDB movie review sentiment; given review text, determine if review is positive or negative

 

• Twist:
  • Let's Assume we only start with 1000 labels; will use as test set
  • Will use Snorkel to create the rest of our labeled data

• Create a labeling function
• Apply labeling function to unlabeled data
• Iterate on labeling functions
• Create a label model to resolve overlaps and conflicts[1]
• Filter out any rows with no information
• Train classification model (with or without probability weighted labels)
• Follow along with the code here

• Other features
  • Spacy Integration: can use NER, PoS tools to help build labels
  • Transformation Functions: can create data augmentation functions to enhance data (e.g., synonym replacement)
  • Sliced-based Learning: focus on subsets of classes / specific subproblems and weight importance

• Overall: 👍 recommended, worth at least exploring if you have high cost labeling scenarios

• Potential to be useful in low-data scenarios, establishing baselines, small performance boosts on existing models

• Great to gain a deeper understanding of a new problem space and/or new data

• Can use other models as labeling functions, can combine signals

• Can be used to pull in new data modes to existing models (e.g., caption text for image)

• Works well for multi-task / ancillary tasks

• Works well in conjunction with active learning

• Return on time investment has high variance, not a slam dunk; getting to a useful output usually requires many iterative cycles

• Performance gains will depend on your size of unlabeled data, quality of label functions, and ability to incorporate weighted labels

• Need to do some accounting for sub-class scenarios; don't want to skew the distributions with homogeneous labeling functions. Ideally LF are:

  • many (more than 20 is good), and diverse

  • mostly correct (50%+ accuracy), and conditionally independent

• Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

• Doesn't work well with tasks such as NER, where you need context

• Best to use in conjunction with other orthogonal methods

• Weak supervision can be a useful tool in your ML toolkit, helping to lower the cost and reduce the time needed to collect labeled data

• Snorkel is a well engineered, open source library that will help with the nuts and bolts of collecting noisy labels and augmenting your training data

• You will get the most return on your time in scenarios where the problem space is new/novel, where expert knowledge is scarce / costly, or where there are large volumes of unlabeled data

Questions

References

• Snorkel Docs
• Weak Supervision: A New Programming Paradigm for Machine Learning
• Data Programming: Creating Large Training Sets, Quickly
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Training Complex Models with Multi-Task Weak Supervision