WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/attention-variants/

1
00:00:00.047 --> 00:00:15.552
Long prompts run a GPU out of memory, and the weights are not to blame: every token a model reads leaves a key and a value behind in every layer, the KV cache, and it grows with every token.

2
00:00:15.552 --> 00:00:41.969
With full multi-head attention on the Llama 3.1 8B shape, all 32 heads in all 32 layers keep their own key and value: 524 kilobytes a token, so 131,072 tokens need 68.7 gigabytes, and with 16 gigabytes of weights that will not fit on an 80 gigabyte H100.

3
00:00:41.969 --> 00:01:01.056
Llama 3.1 8B actually uses grouped-query attention: four query heads share each key and value head, so eight shelves replace 32, a token costs 131 kilobytes, and the full context needs 17.2 gigabytes.

4
00:01:01.056 --> 00:01:18.343
Multi-query attention takes sharing to the limit with a single key and value head, a thirty-second of the cache, but with every head reading the same keys and values the GQA paper found quality drops, so eight groups became the usual compromise.

5
00:01:18.343 --> 00:01:37.125
DeepSeek-V3 compresses instead of sharing heads: multi-head latent attention squeezes each token into a 512-number latent plus a 64-number position key, 70 kilobytes a token, 71 times less than full attention at its size.

6
00:01:37.125 --> 00:01:56.035
A sliding window bounds the cache instead: Mistral 7B keeps only the last 4,096 tokens in each layer, 537 megabytes however long the chat, but a fact 5,000 tokens back can no longer be attended to directly.

7
00:01:56.035 --> 00:02:17.292
That memory decides how many people one GPU can serve: after 16 gigabytes of weights an H100 has 64 gigabytes left, enough for three 32,000-token chats with full attention, 15 with GQA, and 30 once the cache is stored in FP8.

8
00:02:17.292 --> 00:02:33.214
So when long context runs out of memory, count key and value heads, not query heads, store the cache in FP8, and choose grouped queries, a latent or a window by how far back the model really needs to look.