Transformers Lunch & Learn

Part 1: Background and The Transformer

Hamish Dickson - Oct 2024

Over the next 3 weeks

Question: How would you Approach translation?

What's the German for cake?

Was heißt auf Deutsch Kuchen?

???

Question: How would you Approach translation?

What's the German for cake?

Was heißt auf Deutsch Kuchen?

???

First attempts were ngram-to-ngram translations

What's the German for cake?

Was heißt auf Deutsch Kuchen?

This doesn't work well for lots of reasons, even if you could build these mappings you lack context

Question: How would you Approach translation?

What's the German for cake?

Was heißt auf Deutsch Kuchen?

???

Google Translate famously used seq2seq in 2014

Idea is

• have a model "roll over" the input text in order
• build up a representation of the text as we go
• "unroll" using a second model to a new language
• new output model for each language

Question: How would you Approach translation?

inputs

Question: How would you Approach translation?

I did some slides on this a few years ago

Missing piece: encoding/decoding text

Question: how do we go from words to something a ML model can use?

Our models use tensors, how do we turn words into tensors?

• train a model so that when words are similar they are close in a vector space, different either perpendicular or in the opposite direction from the origin
• use this as a lookup at the beginning
• lots of preprocessing to crowbar our sentence into a pretrained model

Missing piece: encoding/decoding text

A highly mocked talk about buffalos

So does it work?

YES! Very well

This is how Google Translate worked for about 4 years (although with tricks, bidirectional models and stacking)

But there are some issues

• note we have encoded this sequential bias into how we teach the model
• this bias helps us learn fast (we have told the model part the answer) but also it's not quite how language works
• this is very very hard to scale up
• w2v is similar, but worse, it's a whole other network

important modelling trick

So does it work?

Note, if you have a problem where:

• You have some sequential problem
• You don't have BUCKETS of similar data to pretrain with
• You want something fast (these go as O(N))

An RNN can still easily beat a transformer

Why do we care about all of this?

There is a huge problem with this design, it doesn't scale

• Remember we roll over each word, one at a time
• it's hard to imagine scaling this over multiple machines
• one machine means it takes a very long time to train a model

Attention is all you Need

• Google Brain paper from 2017
• Honestly, the paper is underwhelming
• Paper was an obscure translation paper (English to German)
• Cool fact: they did this diagram in Google Slides

Attention is all you Need

Introduced lots of ideas

• Ditch w2v, instead "tokenise" the input text (basically an ordinal encoding)
• At the same time as training the rest of the model train the token -> vector encoding
• Ditch the sequential encoding (ie the "rolling over/out")
• Instead we provide the whole sequence at once, this is super important

So how does this work?

Part 1: Embeddings

import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained("bert-base-uncased")

tokenizer.convert_ids_to_tokens(tokenizer("let's talk about bpe")['input_ids'])

So how does this work?

Part 1: Embeddings

Some tips:

• On open source models you can normally add or remove tokens
• This encoding scheme can cause real issues with numbers, the transformer probably won't understand "123" is close to "12" and "4"
• (maybe in part 2: you can force some tokens to be generated)

So how does this work?

Part 2: Attention

Idea behind attention is each token keeps a mapping of the "important" tokens around it

So in this example the attention mechanism learns "it" has a strong connection with "The animal"

So how does this work?

Part 2: Attention

For each token in the sequence we create:

1. A Query vector
2. A Key vector
3. A Value vector

This is a sort of scaling factor for our task - eg is the token a noun or not

This is a learnable representation of asking "what other tokens are important?"

A learnable way of describing the token for other tokens to use

So how does this work?

Part 2: Attention

Dot product between Q and K to 

norm factor, mostly to do with how we start training the model

our value scaling

So how does this work?

Part 2.5: The transformer block

We then do this several times (heads) and layer this with an FC layer

So how does this work?

Part 3: Encoder (BERT et al)

We're nearly done, don't worry

What's the German for cake?

embed

N layers of attention + FC networks

So how does this work?

Part 4: Decoder (GPT et al)

[start token]

So how does this work?

Part 4: Decoder (GPT et al)

embedded context

tokenize

embed

attention block

do this for N layers

FC layer the size of the vocab

Pick the most likely next token from here

Text

So how does this work?

Part 4: Decoder (GPT et al)

embedded context

tokenize

embed

attention block

do this for N layers

FC layer the size of the vocab

Pick the most likely next token from here

I guess downsides of this Architecture?

1. Each token attends to all others, that's O(N^2) RNNs are O(N)
2. You need a lot of compute and data to get this to train well. Realistically not possible before GPUs + AlexNet
3. Very difficult to train from scratch, we know much more now about how to do this, but jeepers it's still hard
4. Before this SOTA models could be trained on commercial desktops, now you need 60k H100s (meta-ai has two of these clusters)
5. Tokenization + positional embeddings a weak point

Final Last thoughts

Idea behind the original paper was to:

• build a seq2seq model without the sequential bias
• got scaling up for free

Next time:

• Show you how to use the encoder and decoder on their own
• Some code
• Talk about pretraining and fine tuning