Attention variants: MHA, GQA, MQA, MLA and sliding windows

What are MHA, GQA, MQA and MLA? They are ways of arranging a transformer’s attention heads, and what separates them is how much memory each token costs. Every token a model has read leaves a key vector and a value vector behind in every layer, for every key/value head: the KV cache. It grows with every token of context, and on long prompts it, not the weights, is what runs a GPU out of memory. The film draws it as a warehouse: one shelf per key/value head, one cell per stretch of context, and an H100’s 80 GB as a glass tank on the right.

Full multi-head attention (MHA) gives every query head its own key and value head. On the shape of Llama 3.1 8B (32 layers, 32 heads of 128 numbers), that is 2 × 32 × 32 × 128 × 2 bytes = 524,288 bytes a token in BF16, as in Llama 2 7B. At Llama 3.1 8B’s full context of 131,072 tokens the cache would be 68.7 GB, and beside 16.1 GB of weights it does not fit on an 80 GB H100.

Grouped-query attention (GQA), which Llama 3.1 8B and 70B and Mistral 7B use, lets several query heads share one key/value head. Llama 3.1 8B has 32 query heads over 8 key/value heads, groups of four, so a token costs 131,072 bytes and the full context 17.2 GB. Multi-query attention (MQA), as in Falcon-7B, shares one key/value head across all of them: a thirty-second of the cache. The GQA paper (Ainslie et al., 2023) found that MQA loses quality, while grouped-query attention stays close to full attention at close to MQA’s speed, which is why GQA became the common default.

Multi-head latent attention (MLA), from DeepSeek-V2 and used in DeepSeek-V3, does not share heads; it compresses. Each token is stored as one latent vector of 512 numbers plus a 64-number position key per layer, and every head’s key and value are re-expanded from the latent when attention reads it. At DeepSeek-V3’s size that is 70,272 bytes a token against 4,997,120 for full attention: 71 times less, for a model with 128 attention heads. A sliding window instead caps how many tokens are kept: Mistral 7B v0.1 keeps the last 4,096 in each layer, so its cache stops at 537 MB however long the chat runs, and a fact 5,000 tokens back can no longer be attended to directly.

The memory decides how many people one GPU can serve. After Llama 3.1 8B’s 16.1 GB of weights, an H100 has 63.9 GB left for caches: three 32,000-token chats with full attention, 15 with GQA, and 30 with the cache stored in FP8, which halves every number here. When paused, the film lets you pick a variant, a context length and a cache precision, and the shelves, the tank and the formula follow; the KV cache per token for Llama 3.1 8B matches the 128 KiB the prefill and decode film uses.

The honest caveats. The warehouse keeps Llama 3.1 8B’s shape and swaps only the key/value heads, so its MHA and MQA figures describe that shape, not a trained model; the MLA figure on that shape uses the DeepSeek-V2 paper’s sizing (a latent of four head widths plus half a head for position) and is illustrative, while the MLA chapter prints DeepSeek-V3’s real numbers. Latent attention also trades memory for arithmetic: every read has to project the latent back up to each head’s key and value, though the DeepSeek-V2 paper notes those up-projections can be folded into the query and output projections at inference. Serving systems add overheads the formula leaves out: paged caches round up to blocks, activations and workspace need memory too, and quantising the cache to FP8 usually costs a little accuracy. Quality differences between the variants depend on training, and the film quotes the GQA paper rather than measuring them.