\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

This is self attention with only "1 head" (like having only 1 conv filter)

\mathbf v^{\textcolor{skyblue}{(h)}} (\mathbf x_i) = W^T_{v, \textcolor{skyblue}{h}} \mathbf x_i, \quad \mathbf q^{\textcolor{skyblue}{(h)}} (\mathbf x_i) = W^T_{q, \textcolor{skyblue}{h}} \mathbf x_i, \quad \mathbf k^{\textcolor{skyblue}{(h)}} (\mathbf x_i) = W^T_{k, \textcolor{skyblue}{h}} \mathbf x_i
\alpha_{i, j}^{\textcolor{skyblue}{(h)}} = \text{softmax}_j \left( \frac{\langle \mathbf q^{\textcolor{skyblue}{(h)}}(\mathbf x_i),\: \mathbf k^{\textcolor{skyblue}{(h)}}(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \textcolor{skyblue}{\sum_{h=1}^{H} W^T_{c, h}} \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j) ^{\textcolor{skyblue}{(h)}}
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

Multi-headed attention with H heads (similar to conv. filter dimension)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_{v, \textcolor{skyblue}{h}}, W_{q, \textcolor{skyblue}{h}}, W_{k, \textcolor{skyblue}{h}} \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query for each head.

Attention weights calculate how similar the query of each position is to the keys of all other positions for each head.

For each position and each head, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
\mathbf v (\mathbf x_i) = W^T_v \mathbf x_i, \quad \mathbf q (\mathbf x_i) = W^T_q \mathbf x_i, \quad \mathbf k (\mathbf x_i) = W^T_k \mathbf x_i
\alpha_{i, j} = \text{softmax}_j \left( \frac{\langle \mathbf q(\mathbf x_i),\: \mathbf k(\mathbf x_j) \rangle}{\sqrt{d'}} \right)
\mathbf u_i = \sum_{j=1}^{n} \alpha_{i, j} \mathbf v(\mathbf x_j)
\mathbf u_i' = \text{LayerNorm}(x_i + \mathbf u_i; \gamma_1, \beta_1)
\mathbf z_i = W_2^T \text{ReLU} (W_1^T \mathbf u_i')
\mathbf z_i' = \text{LayerNorm}(\mathbf u_i' + \mathbf z_i; \gamma_2, \beta_2)

This block constitutes self-attention. In ordinary attention, keys/queries take different inputs.

W_v, W_q, W_k \in \mathbb R^{d \times d'}
\text{LayerNorm}(\mathbf z; \mathbf \gamma, \mathbf \beta) = \mathbf \gamma \frac{\mathbf z - \mathbf \mu_z}{\sigma_z} + \mathbf \beta

Each position gets a value, key, and query vector

Attention weights calculate how similar the query of each position is to the keys of all other positions

For each position, value vectors are linearly combined weighted by attention.

Add output to residual input + normalize

Feed through linear / nonlinearity

Add another residual connection + normalize

Note: often add positional encoding into x

Transformer block maps

\mathbf x \in \mathbb R^{n \times d} \to \mathbb R^{n \times d}
Made with Slides.com