Keshav's 3 Pass Method

3 Pass Method

• Pass 1 [ 5 - 10 mins ]
• Pass 2 [ 1 hour ]
• Pass 3 [ few hours? ]

3 Pass Method

1. Get the gist of it ( bird's eye view )
2. Grasp Contents ( but not the details )
3. Understand in depth ( put it all together )

3 Pass Method

• Each pass has a specific goal
• Builds on top of the previous

Pass 1

• Quick Scan
• Bird's Eye view ( the gist of it )
  • Decide if you should keep going
• Enough for papers
  • not in your research area

Pass 1

• Read Carefully
  • Title
  • Abstract
  • Introduction
  • Conclusion
• Section, Subsection Titles
• Glance through
  • equations
  • references

Pass 1

• References
  • Tick off the papers that you are familiar with

Pass 1

• References
  • Tick off the papers that you are familiar with

Pass 1

• Objectives : 5 C's
  • Category
  • Context
    • related papers
  • Correctness
  • Contributions
  • Clarity

Pass "dos"

• Read with greater care
• Takes about an hour for experienced reader
• Enough for papers that are interesting
  • but not your research speciality
    • Computer Vision?

Pass 2

• Objective
  • Summarise the paper to someone
    • with supporting evidence

Pass 2

• Highlight key points
• Make comments
  • Terms you don't understand
  • Questions for the author
• Use comments to write a review

Pass 2

• Figures, Graphs and such
• Tick relevant unread references

Hold on!

What if I still don't get it?

Like, at all?

Bulk of it is just incomprehensible

Why?

• New Subject Matter
• Unfamiliar Terminology/acronyms
• Weird unfamiliar techniques?
• Is it badly written?
• Or you are just tired man!

What to do about it?

• F**k it! Let's go bowling
• Or.. may be
  • Read background and get back to it
  • Power through pass 3

Pass 3

• 2 to many hours
• Great attention to detail
  • Challenge every assumption
• How would you present the same idea?

Pass 3

• Make the same assumptions
  • Recreate the work
• Compare with the actual paper
• Identify
  • Innovations
  • Hidden Failings
  • Assumptions

Pass 3

• Goal
  • Reconstruct the entire structure of the paper from memory
  • Identify Weak and Strong points

Bonus Round

How to do a Literature Survey?

Literature Survey

• Google Scholar / Semantic Scholar
  • bunch of carefully-chosen keywords
• 3-5 highly cited papers

Literature Survey

• One pass on each paper
• Find shared citations, repeated author names
  • Key papers
    • Download them
  • Key researchers
• Find top conferences

Literature Survey

If you are lucky enough to find a survey paper,

You are done.

Literature Survey

• Papers from
  • Recent Proceedings
  • Key Papers from Key Researchers
  • Shared Citations
• Combine them
  • And it becomes the first version of the survey

ULMFiT

Universal Language Model Fine-Tuning for Text Classification 

Universal Language Model Fine-Tuning for Text Classification

Abstract

Inductive transfer learning has greatly impacted computer vision, but existing approaches in NLP still require task-specific modifications and training from scratch. We propose Universal Language Model Fine-tuning (ULMFiT), an effective transfer learning method that can be applied to any task in NLP, and introduce techniques that are key for fine-tuning a language model. Our method significantly outperforms the state-of-the-art on six text classification tasks, reducing the error by 18-24% on the majority of datasets. Furthermore, with only 100 labeled examples, it matches the performance of training from scratch on 100x more data. We open-source our pretrained models and code.

Ideas

Inductive vs Transductive Transfer

Ideas

Language Modeling is the ideal source task.

Imagenet for NLP

Language Modeling

• Common / General Linguistic Features
  • Long-term dependencies
  • Hierarchical Relations
  • Sentiment
• Key component of every other NLP task

Steps

1. General Domain LM Pretraining
2. Target-task LM Fine Tuning
3. Target-task Classifier Fine Tuning

Dataset

• Pre-training on Wikitext-103
  • ~28K Wiki Articles
  • 103 Million words

Discriminative Fine-tuning

• Key Idea
  • Different layers capture different types of information
  • Fine-tune to different extents
  • Tune layers with different learning rate
• A variant of SGD to exploit this insight

Discriminative Fine-tuning

Slanted Triangular LR (STLR)

• Ideally
  • Quickly converge to suitable region of parameter space
• Variant of Triangular LR*
  • Short Increase
  • Long Decay Period

Slanted Triangular LR (STLR)

STLR

Target Task Classifier      Fine Tuning

• Aggressive Fine Tuning leads to
  • Catastrophic Forgetting

Gradual Unfreezing

• To overcome Catastrophic Forgetting
• "chain-thaw"?
  • that sounds interesting
• Unfreeze the model in iterations
  • Start with the Last Layer
  • Last Layer
    • Least General Knowledge

Gradual Unfreezing

• Start with the Last Layer
• Fine-tune for 1 Epoch
• Unfreeze next lower layer
• Fine-tune for 1 Epoch
• ...

Classifier Fine Tuning

• Add two Linear Layers
  • Only parameters learned from scratch
• Batch Normalization
• Drop out
• ReLU Activation for the first linear layer
• Softmax at the end

Classifier Fine Tuning

• "concat-pooling"?
  • that sounds interesting

Results