Representation Learning

Building Useful Internal Knowledge

Shen Shen

Today's Roadmap

Representations

Layer-by-layer geometry and transfer

Learning Without Labels

Autoencoders, bottlenecks, and masking

Word Embeddings

Prediction, similarity, and soft lookup

Recent Ideas

Masking, contrastive learning, multimodality

Some image credits for Parts 1–2: visionbook.mit.edu

\dots

layer

linear combo

activations

\dots

layer

\dots

x_1

x_2

x_d

input

\Sigma

f(\cdot)

\Sigma

f(\cdot)

\Sigma

f(\cdot)

\Sigma

f(\cdot)

\Sigma

f(\cdot)

\Sigma

f(\cdot)

\Sigma

f(\cdot)

neuron

learnable weights

hidden

output

asymptotically, can approximate any function!*

\Sigma

\dots

under some technical conditions (e.g. nonlinearity can't be polynomial)

Two different ways to visualize a function

e.g. the identity function

Representation transformations for a variety of neural net operations

and stack of neural net operations

)

wiring graph

equation

mapping 1D

mapping 2D

What does training a neural net classifier look like?

Training data

z_1

a_1

z_2

{z}_1=\text { linear }(x) \in \mathbb{R^2}

{a}_1=\text { ReLU}(z_1)

g=\text {softmax}(z_2)

{z}_2=\text { linear }(a_1) \in \mathbb{R^2}

x\in \mathbb{R^2}

maps from complex data space to desired target space

Deep neural nets transform datapoints, layer by layer

Each layer is a different representation, aka embedding, of the data

From data to latent embeddings: representation learning

(From latent embeddings to data: generative modeling)

Representation learning

Generative modeling

🧠

humans also learn representations

[Bartlett, 1932]
[Intraub & Richardson, 1989]

Compact (minimal)
Explanatory (roughly sufficient)
Disentangled (independent factors)
Interpretable (understandable by humans)
Make subsequent problem solving easy

[See “Representation Learning”, Bengio 2013, for more commentary]

Good representations are:

ImageNet also taught us that labeling 14 million images by hand is brutal.

Label prediction (supervised learning)

Label

$$x$$

$$y$$

Features

Labels $y$ are expensive…

Unlabeled features $x$ are abundant

Can we learn useful things with just $x$?

Compact (minimal)
Explanatory (roughly sufficient)
Disentangled (independent factors)
Interpretable (understandable by humans)
Make subsequent problem solving easy

[See “Representation Learning”, Bengio 2013, for more commentary]

Auto-encoders explicitly aims

\left\{ \begin{array}{l} \\ \\ \end{array} \right.

\left\{ \begin{array}{l} \\ \\ \\ \end{array} \right.

these may just emerge as well

Good representations are:

Unsupervised Learning (feature reconstruction)

Features

Reconstructed Features

$$x$$

$$\hat{x}$$

[https://www.behance.net/gallery/35437979/Velocipedia]

Auto-encoder

"What I cannot create, I do not understand." Feynman

compact representation/embedding

Auto-encoder

\underbrace{\hspace{1cm}}

encoder

decoder

bottleneck

Auto-encoder

\tilde{x}=\text{NN}(x;W)

\min_{W} ||x - \tilde{x}||^2

Data space:

(high dimensional, irregular)

Encoder

Decoder

Representation space

(low dimensional, regular)

[Typically, encoders can serve as a translator to get "good representations", whereas decoders can serve as "generative models"]

But it's easy to "cheat" with auto-encoders

A bottleneck that's too wide

Perfect reconstruction, nothing learned. It just photocopies x.

[Vincent et al, Extracting and composing robust features with denoising autoencoders, ICML 08]

But it's easy to "cheat" with auto-encoders

[Steck 20, Zhang et al 17]

the reconstruction error is fine

but not very useful

decoder

encoder

Unsupervised Learning (feature reconstruction)

Features

Reconstructed Features

$$x$$

$$\hat{x}$$

$${x}$$

Self-supervised Learning (partial feature reconstruction)

Partial

Features

Other Partial Features

Hard reconstruction, forces understanding.

Masked Auto-Encoder (Vision)

[He, et al. Masked Autoencoders Are Scalable Vision Learners, 2021]

[Zhang, Isola, Efros, ECCV 2016]

e.g. masking channels

1. Masking

predict color from gray-scale

[Zhang, Isola, Efros, ECCV 2016]

1. Masking

Often, what we will be “tested” on is not what we were trained on.

"common"-sense representation

task-specific prediction

Final-layer adaptation: freeze $f$, train a new final layer to new target data

"common"-sense representation

task-specific prediction

Finetuning: initialize $f’$ as $f$, then continue training for $f'$ as well, on new target data

"common"-sense representation

task-specific prediction

Large Language Models (LLMs) are trained in this self-supervised way

Scrape the internet for plain texts.
Cook up “labels” (prediction targets) from these texts.
Convert “unsupervised” problem into “supervised” setup.

"To date, the cleverest thinker of all time was Issac. "

feature

label

To date, the

cleverest

\dots

To date, the cleverest

thinker

To date, the cleverest thinker

was

\dots

To date, the cleverest thinker of all time was

Issac

[video edited from 3b1b]

Word embedding

dot-product similarity

[video edited from 3b1b]

dot-product similarity

For now, let's look at how good embeddings enable "soft" dictionary look-up:

dict_en2fr = { 
  "apple" : "pomme",
  "banana": "banane", 
  "lemon" : "citron"}

Good word embeddings space is equipped with sensible dot-product similarity

Key

Value

apple

pomme

$:$

banane

banana

$:$

citron

lemon

$:$

dict_en2fr = { 
  "apple" : "pomme",
  "banana": "banane", 
  "lemon" : "citron"}

query = "lemon" 
output = dict_en2fr[query]

apple

pomme

banane

citron

banana

lemon

Key

Value

lemon

$:$

Query

Output

citron

dict_en2fr = { 
  "apple" : "pomme",
  "banana": "banane", 
  "lemon" : "citron"}

query = "orange" 
output = dict_en2fr[query]

Python would complain. 🤯

orange

apple

pomme

banane

citron

banana

lemon

Key

Value

$:$

Query

Output

???

What if we run

But we can probably see the rationale behind this:

Query

Key

Value

Output

orange

apple

$:$

pomme

banana

$:$

banane

lemon

$:$

citron

dict_en2fr = { 
  "apple" : "pomme",
  "banana": "banane", 
  "lemon" : "citron"}

query = "orange" 
output = dict_en2fr[query]

0.1

pomme

0.1

banane

0.8

citron

0.1

pomme

0.1

banane

0.8

citron

via these merging percentages [0.1 0.1 0.8] made sense

We put (query, key, value) in "good" embeddings in our human brain

such that "merging" the values

Query

Key

Value

Output

orange

apple

$:$

pomme

0.1

pomme

0.1

banane

0.8

citron

banana

$:$

banane

lemon

$:$

citron

orange

0.1

pomme

0.1

banane

0.8

citron

apple

banana

lemon

orange

0.8

0.1

pomme

banane

citron

very roughly, the attention mechanism in transformers automates this process.

e.g. medical imaging

[Zhou+,'22; Chen+,'22; Huang+,'22; An+,'22]

1. Masking

e.g. 3d geometry

[Pang+, '22; Liang+, '22; Min+, '22; Krispel+, '22]

1. Masking

e.g. audio

[Baade+, '22; Chong+, '22; Niizumi, '22; Huang+, '22]

1. Masking

e.g. graphs

[Tan+, '22; Zhang+, '22; Hou+, '22; Li+, '22]

1. Masking

e.g. robotics

[Xiao+, '22; Radosavovic+, '22; Seo+, '22;]

1. Masking

[Feichtenhofer, et al., "Masked Autoencoders As Spatiotemporal Learners", NeurIPS 2022]

1. Masking

[Feichtenhofer, et al., "Masked Autoencoders As Spatiotemporal Learners", NeurIPS 2022]

1. Masking

[He, et al. Masked Autoencoders Are Scalable Vision Learners, 2021]

masking ratio

transfer accuracy

masking 75% is
optimal for images

1. Masking

95% masked

98% masked

Similar empirical studies shows 15% as optimal for languages, and 95% for videos

1. Masking

The allegory of the cave

2. Contrastive learning

[images credit: visionbook.mit.edu]

2. Contrastive learning

[images credit: visionbook.mit.edu]

2. Contrastive learning

[Chen, Kornblith, Norouzi, Hinton, ICML 2020]

SimCLR animation

2. Contrastive learning

3. Multi-modality

[images credit: visionbook.mit.edu]

[Owens et al, Ambient Sound Provides Supervision for Visual Learning, ECCV 2016]

e.g. video, audio, images

[Owens et al, Ambient Sound Provides Supervision for Visual Learning, ECCV 2016]

What did the model learn?

[Owens et al, Ambient Sound Provides Supervision for Visual Learning, ECCV 2016]

e.g. image classification (done in the contrastive way)

[Radford et al, Learning Transferable Visual Models From Natural Language Supervision, ICML, 2011]

e.g. image classification (done in the contrastive way)

[Radford et al, Learning Transferable Visual Models From Natural Language Supervision, ICML, 2011]

[https://arxiv.org/pdf/2204.06125.pdf]

e.g Dall-E: text-image generation

[Slide Credit: Yann LeCun]

Summary

Neural networks are not just calculators; each layer reshapes data into a new representation.
Good representations are compact, explanatory, reusable, and useful for transfer.
Autoencoders learn representations by reconstructing inputs, but plain reconstruction can cheat.
Self-supervised masking makes reconstruction harder and turns unlabeled data into a training signal.
Word embeddings, contrastive learning, and multimodality all learn geometries where meaning is useful.