Existing strategies for applying pre-trained language representations to downstream
tasks:

1. feature-based

• ELMo

2. fine-tuning

• Generative Pre-trained Transformer (OpenAI GPT)

Limitation to learn general language representations:

• standard language models are unidirectional

• choice of architectures used during pre-training is limited

Parsing / composition is a framework for language understanding.

Compositionality principle: states that the meaning of word compounds is derived from the meaning of the individual words, and the manner in which those words are combined.

Hierarchical structure of language: states that through analysis, sentences can be broken down into simple structures such as clauses. Clauses can be broken down into verb phrases and noun phrases and so on.

Let us take a quick look at two different parse trees derived from a sentence "Bart watched a squirrel with binoculars".

Source: Understanding BERT Transformer: Attention isn’t all you need | by Damien Sileo | synapse_dev | Medium

Successive compositions yield the meaning of the sentence.

Example:

composing “a” and “squirrel”, “watched” with “a squirrel”, “watched a squirrel” and “with binoculars”

Composition relies on the result of parsing to determine what ought to be composed.

Composition and parsing are both hard tasks, and they need one another.

BERT is a new language representation model based on two steps.

1. pre-training

2. fine-tuning

Source: [1810.04805] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arxiv.org)

Pre-training:

• model is trained on unlabeled data over different pre-training tasks.

Fine-tuning:

• the BERT model is first initialized with the pre-trained parameters, and all of the parameters are fine-tuned using labeled data from the downstream tasks.

• unified architecture across different tasks.

Let us consider a question-answering example for the downstream task and WordPiece embeddings (Wu et al.,2016) with a 30,000 token vocabulary.

Example: <Question, Answer>

[CLS]: a special classification token which is the first token of every sequence.

E: input embedding

C: final hidden vector of the special [CLS] token, \( C \in \mathbb R^H\)

\(T_i\): final hidden vector for the \(i^{th}\) input token, \( T_i \in \mathbb R^H\)

Input: "this movie is great"

• a special “<mask>” token for 80% of the time
  • (e.g., “this movie is <mask>”);

If “great” is selected to be masked and predicted, then it will be replaced with:

• a random token for 10% of the time
  • (e.g., “this movie is drink”);

• the unchanged label token for 10% of the time
  • (e.g., “this movie is great”).

Input: sentences A and B

• 50% of the time B is the actual next sentence that follows A
  • labeled as IsNext

When generating sentence pairs for pretraining,

• 50% of the time it is a random sentence from the corpus
  • labeled as NotNext

At the input, sentence A and sentence B from pre-training are analogous to

• sentence pairs in paraphrasing
• hypothesis-premise pairs in entailment
• question-passage pairs in question answering
• a degenerate text pair in text classification or sequence tagging

At the output,

• the token representations are fed into an output layer for tokenlevel tasks, such as sequence tagging or question answering
• [CLS] representation is fed into an output layer for classification, such as entailment
  or sentiment analysis.

\(\mathbf {BERT_{BASE}}\)

\(\mathbf {BERT_{LARGE}}\)

1. L=12
2. H=768
3. A=12,
4. Total Parameters= 110M

1. L=24
2. H=1024
3. A=16,
4. Total Parameters= 340M

These models are fine-tuned on General Language Understanding Evaluation (GLUE) benchmark.