LLM Inference
9th July 2026
LLM Decoding
https://poloclub.github.io/transformer-explainer/
KV Cache
Demo: Observing the generation
TPOT
TTFT
Prefill Phase
Decode Phase
ms
ms
Decode
Prefill
Decode
Prefill
Compute Heavy
Bandwidth Heavy
Decode
Prefill
Compute Heavy
Bandwidth Heavy
Compute Bound
Memory Bound
Roofline Curve
Roofline Curve
FLOPs/ sec
FLOPs/ byte
Roofline Curve
FLOPs/ sec
FLOPs/ byte
Peak FLOPs
Roofline Curve
FLOPs/ sec
FLOPs/ byte
Peak FLOPs
Bandwidth
Roofline Curve
FLOPs/ sec
FLOPs/ byte
Peak FLOPs
Bandwidth
Roofline Curve
Roofline Curve
Prefill
Decode
KV Cache
KV Cache
KV Cache
KV Caching
<Token>
q
k
v
KV Caching
<Token>
q
k
v
KV Cache= 2
KV Caching
<Token>
q
k
v
KV Cache= 2 * dim
dim
KV Caching
KV Cache= 2 * dim * num heads
KV Caching
KV Cache= 2 * dim * num heads * num layers
Layer 1
Layer 1
Layer 2
Layer n
Quantization
| Format | Bit |
|---|---|
| FP64 | 8 |
| FP32 | 4 |
| FP16/BF16 | 2 |
| FP8 | 1 |
| FP4 | 0.5 |
KV Caching
KV Cache= 2 * dim * num heads * num layers * kv data type
<Token>
KV Caching
KV Cache= 2 * dim * num heads * num layers * kv data type * seq length
<Token>
<Token>
<Token>
<Token>
<Token>
......
KV Caching
https://huggingface.co/meta-llama/Llama-2-7b
KV Caching
https://huggingface.co/meta-llama/Llama-2-7b
Demo
KV Caching
KV Cache= 2 * dim * num heads * num layers * kv data type * seq length
KV Caching
KV Caching
7B vs 15B
7B vs 15B
Inference Engines
CPU based Inference
GPU based Inference
