# Attention variants: MHA, GQA, MQA, MLA and sliding windows

**AI Engineering Visualised** · Context · Film, 2:36 · Updated 2026-09-11

Multi-head, grouped-query, multi-query, multi-head latent and sliding-window attention are different ways of sharing or shrinking the keys and values a model stores per token. The choice decides how much KV cache memory a long context costs.

> Press play and watch a KV-cache warehouse fill an H100: full attention needs 68.7 GB at Llama 3.1 8B’s full context, grouped-query attention cuts it to 17.2 GB, one shared head to a thirty-second, DeepSeek-V3’s latent attention compresses 71 times, and a sliding window caps it at 537 MB, all computed from the models’ published configs.

Watch the film: https://understandingdata.com/ai-engineering-visualised/attention-variants/
Video file: https://assets.understandingdata.com/viz/attention-variants/video/film.mp4?v=349bf8bf9b

What are MHA, GQA, MQA and MLA? They are ways of arranging a transformer’s attention heads, and what separates them is how much memory each token costs. Every token a model has read leaves a key vector and a value vector behind in every layer, for every key/value head: the KV cache. It grows with every token of context, and on long prompts it, not the weights, is what runs a GPU out of memory. The film draws it as a warehouse: one shelf per key/value head, one cell per stretch of context, and an H100’s 80 GB as a glass tank on the right.

Full multi-head attention (MHA) gives every query head its own key and value head. On the shape of Llama 3.1 8B (32 layers, 32 heads of 128 numbers), that is 2 × 32 × 32 × 128 × 2 bytes = 524,288 bytes a token in BF16, as in Llama 2 7B. At Llama 3.1 8B’s full context of 131,072 tokens the cache would be 68.7 GB, and beside 16.1 GB of weights it does not fit on an 80 GB H100.

Grouped-query attention (GQA), which Llama 3.1 8B and 70B and Mistral 7B use, lets several query heads share one key/value head. Llama 3.1 8B has 32 query heads over 8 key/value heads, groups of four, so a token costs 131,072 bytes and the full context 17.2 GB. Multi-query attention (MQA), as in Falcon-7B, shares one key/value head across all of them: a thirty-second of the cache. The GQA paper (Ainslie et al., 2023) found that MQA loses quality, while grouped-query attention stays close to full attention at close to MQA’s speed, which is why GQA became the common default.

Multi-head latent attention (MLA), from DeepSeek-V2 and used in DeepSeek-V3, does not share heads; it compresses. Each token is stored as one latent vector of 512 numbers plus a 64-number position key per layer, and every head’s key and value are re-expanded from the latent when attention reads it. At DeepSeek-V3’s size that is 70,272 bytes a token against 4,997,120 for full attention: 71 times less, for a model with 128 attention heads. A sliding window instead caps how many tokens are kept: Mistral 7B v0.1 keeps the last 4,096 in each layer, so its cache stops at 537 MB however long the chat runs, and a fact 5,000 tokens back can no longer be attended to directly.

The memory decides how many people one GPU can serve. After Llama 3.1 8B’s 16.1 GB of weights, an H100 has 63.9 GB left for caches: three 32,000-token chats with full attention, 15 with GQA, and 30 with the cache stored in FP8, which halves every number here. When paused, the film lets you pick a variant, a context length and a cache precision, and the shelves, the tank and the formula follow; the KV cache per token for Llama 3.1 8B matches the 128 KiB the prefill and decode film uses.

The honest caveats. The warehouse keeps Llama 3.1 8B’s shape and swaps only the key/value heads, so its MHA and MQA figures describe that shape, not a trained model; the MLA figure on that shape uses the DeepSeek-V2 paper’s sizing (a latent of four head widths plus half a head for position) and is illustrative, while the MLA chapter prints DeepSeek-V3’s real numbers. Latent attention also trades memory for arithmetic: every read has to project the latent back up to each head’s key and value, though the DeepSeek-V2 paper notes those up-projections can be folded into the query and output projections at inference. Serving systems add overheads the formula leaves out: paged caches round up to blocks, activations and workspace need memory too, and quantising the cache to FP8 usually costs a little accuracy. Quality differences between the variants depend on training, and the film quotes the GQA paper rather than measuring them.

## The maths

### The KV cache, per token

```latex
\text{bytes per token} \;=\; 2 \times L \times H_{kv} \times d_h \times b \;=\; 2 \times 32 \times 8 \times 128 \times 2 \;=\; 131{,}072
```

A key and a value (the 2) for every layer L and every key/value head H_kv, each d_h numbers of b bytes. Llama 3.1 8B has 32 layers, 8 key/value heads of 128, stored in BF16. The attention pills change H_kv (32, 8 or 1); the cache switch changes b (2 for BF16, 1 for FP8).

### Grouped and multi-query attention

```latex
H_{kv}^{\text{MHA}} = H_q = 32, \qquad H_{kv}^{\text{GQA}} = \frac{H_q}{g} = \frac{32}{4} = 8, \qquad H_{kv}^{\text{MQA}} = 1
```

Grouping g query heads onto each key/value head divides the cache by g. Full attention at Llama 3.1 8B’s shape needs 524,288 bytes a token, grouped queries 131,072, a single shared head 16,384: a thirty-second.

### Multi-head latent attention

```latex
\text{bytes per token} \;=\; L\,(d_c + d_h^{R})\,b \;=\; 61 \times (512 + 64) \times 2 \;=\; 70{,}272 \quad\text{vs}\quad 61 \times 128 \times (192 + 128) \times 2 \;=\; 4{,}997{,}120
```

DeepSeek-V3 caches one compressed latent c of d_c = 512 numbers and one shared position key of 64 per layer, instead of 128 heads’ keys and values: 71 times less than full attention at its own size. On read, each head’s key and value are re-expanded from the latent by learned up-projections.

### A sliding window

```latex
\text{cache} \;=\; 2 L H_{kv} d_h b \times \min(n, W) \;=\; 131{,}072 \times 4{,}096 \;\approx\; 537\ \text{MB}
```

Mistral 7B v0.1 keeps only the last W = 4,096 tokens in each layer, so its cache stops growing at 537 MB however long the chat n runs. The price is that a token more than 4,096 back cannot be attended to directly; information from it can only arrive relayed through the layers above.

## Transcript

### 0:00 Out of memory

Long prompts run a GPU out of memory, and the weights are not to blame: every token a model reads leaves a key and a value behind in every layer, the KV cache, and it grows with every token.

### 0:16 Full attention

With full multi-head attention on the Llama 3.1 8B shape, all 32 heads in all 32 layers keep their own key and value: 524 kilobytes a token, so 131,072 tokens need 68.7 gigabytes, and with 16 gigabytes of weights that will not fit on an 80 gigabyte H100.

### 0:42 Grouped queries

Llama 3.1 8B actually uses grouped-query attention: four query heads share each key and value head, so eight shelves replace 32, a token costs 131 kilobytes, and the full context needs 17.2 gigabytes.

### 1:01 One shared head

Multi-query attention takes sharing to the limit with a single key and value head, a thirty-second of the cache, but with every head reading the same keys and values the GQA paper found quality drops, so eight groups became the usual compromise.

### 1:18 Compress, not share

DeepSeek-V3 compresses instead of sharing heads: multi-head latent attention squeezes each token into a 512-number latent plus a 64-number position key, 70 kilobytes a token, 71 times less than full attention at its size.

### 1:37 Sliding window

A sliding window bounds the cache instead: Mistral 7B keeps only the last 4,096 tokens in each layer, 537 megabytes however long the chat, but a fact 5,000 tokens back can no longer be attended to directly.

### 1:56 Users per GPU

That memory decides how many people one GPU can serve: after 16 gigabytes of weights an H100 has 64 gigabytes left, enough for three 32,000-token chats with full attention, 15 with GQA, and 30 once the cache is stored in FP8.

### 2:17 Count KV heads

So when long context runs out of memory, count key and value heads, not query heads, store the cache in FP8, and choose grouped queries, a latent or a window by how far back the model really needs to look.

## Related terms

- [Context window](https://understandingdata.com/ai-coding-dictionary/context-window/)
- [Attention](https://understandingdata.com/ai-coding-dictionary/attention/)
- [Cache tokens](https://understandingdata.com/ai-coding-dictionary/cache-tokens/)
- [Prefix cache](https://understandingdata.com/ai-coding-dictionary/prefix-cache/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)

## More visualisations

- [KV cache and paging](https://understandingdata.com/ai-engineering-visualised/kv-cache/)
- [Prefill vs decode: time to first token and tokens per second](https://understandingdata.com/ai-engineering-visualised/prefill-decode/)
- [FlashAttention](https://understandingdata.com/ai-engineering-visualised/flash-attention/)
- [Attention head fingerprints](https://understandingdata.com/ai-engineering-visualised/attention-heads/)
- [Prompt caching: reuse the stable prefix](https://understandingdata.com/ai-engineering-visualised/prompt-caching-prefix/)
- [The transformer block, exploded](https://understandingdata.com/ai-engineering-visualised/transformer-block/)

---

Source: https://understandingdata.com/ai-engineering-visualised/attention-variants/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.