WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/prefill-decode/

1
00:00:00.140 --> 00:00:14.425
Paste a long document into a chat and you wait for the first word, then the rest arrive at a steady pace, because answering is two jobs, prefill and decode, each limited by a different part of the GPU.

2
00:00:14.425 --> 00:00:33.009
Prefill pushes the whole prompt through in one pass: 2,000 tokens of Llama 3.1 8B is 33 trillion operations, and at an H100's 989 teraflops that takes about 34 milliseconds, with every core busy.

3
00:00:33.009 --> 00:00:51.705
Decode is a drip: each new token rereads all 16 gigabytes of weights for just 16 billion operations, so even at 3.35 terabytes a second an H100 needs 4.8 milliseconds a token, about 209 a second.

4
00:00:51.705 --> 00:01:07.335
That is the part people miss: decode does one operation per byte it moves and sits low on the roofline's memory slope, while prefill does 2,000 and sits on the flat compute roof, past the ridge at 295.

5
00:01:07.335 --> 00:01:22.796
So servers batch decode: 64 users share one read of the weights, the step grows only from 4.8 to 5.4 milliseconds, and the chip turns out over 11,000 tokens a second instead of 209.

6
00:01:22.796 --> 00:01:45.119
The failure mode is buying arithmetic for chat: an H100 has three times an A100's teraflops but only 1.6 times its bandwidth, so one user's decode gets 1.6 times faster, while a 32,000-token prompt still waits 0.8 seconds, a third of that in attention.

7
00:01:45.119 --> 00:02:03.531
Measured on a real endpoint, BF16 Llama 3.1 8B on CoreWeave, the first word came after about 0.3 seconds for a short prompt or one of 2,000 tokens, mostly network, and 0.74 seconds for 16,000.

8
00:02:03.531 --> 00:02:18.608
Decode ran at 156 tokens a second, then 13 percent slower with the long prompt, because each step also rereads that prompt's 2.1 gigabytes of KV cache, 13 percent more bytes than the weights alone.

9
00:02:18.608 --> 00:02:30.981
So treat them as two budgets: cut time to first token by trimming and caching the prompt, and buy tokens per second with bandwidth, quantisation and batching, not with faster arithmetic.