What is the difference between prefill and decode? When a language model answers, it does two different jobs. Prefill reads your whole prompt in one pass and builds the model's working memory of it, the KV cache; the time it takes is most of the time to first token. Decode then writes the answer one token at a time, each step reading the whole model again; its speed is the tokens per second you watch stream in. The two are limited by different parts of the GPU, which is why a long prompt makes you wait for the first word and then the rest arrive at a steady pace.
The film follows one GPU, an H100, serving Llama 3.1 8B in BF16: 8.03 billion parameters, 16 GB of weights, drawn as the wall behind the chip. Prefill is the sluice: 2,000 prompt tokens flood through together, every weight is read once for all of them, and the chip does 33 trillion operations in about 34 milliseconds with every core busy. Decode is the tap: each drop is one token, and each one needs the whole wall read again for only 16 billion operations. At 3.35 TB/s that is 4.8 milliseconds a token, about 209 a second at best, while most of the chip's arithmetic sits idle.
The roofline explains it. Divide a step's operations by the bytes it moves and you get its arithmetic intensity. The H100's ridge is at 295 operations per byte: below it the chip waits on memory, above it on arithmetic. Decode at batch 1 does one operation per byte and sits low on the memory slope; prefill of 2,000 tokens does about 2,000 and sits on the flat compute roof. So the two metrics answer to different hardware: time to first token to arithmetic (and, for long prompts, to attention, which grows with the square of the prompt), tokens per second to bandwidth.
That is also why serving systems batch decode. When 64 users decode together, one read of the weights serves all 64: the step grows only from 4.8 to 5.4 milliseconds while the chip produces over 11,000 tokens a second. Each sequence still reads its own KV cache, so batching climbs the slope more slowly than you might hope; even at 295 users this model stays memory bound. Continuous batching and paged KV caches exist to keep that batch full. Quantising the weights to 8 bits halves the bytes and roughly doubles batch-1 decode.
The measurements are real. Through OpenRouter, with the provider pinned to CoreWeave's BF16 endpoint, the first word arrived after about 0.3 seconds for a 34-token or a 2,081-token prompt (network and queueing dominate at that size) and after 0.74 seconds for 16,361 tokens. Decode ran at 156 tokens a second, about three quarters of an H100's ceiling, and 13 percent slower with the long prompt, matching the 13 percent of extra bytes its 2.1 GB of KV cache adds to every step. The same model on another provider's FP8 endpoint decoded at 48 tokens a second: the speed you get is whatever bandwidth and batch share the provider gives you.
The honest caveats. The roofline is a ceiling, not a prediction: real kernels reach perhaps 40 to 70 percent of peak arithmetic and 70 to 90 percent of peak bandwidth, and the film's arithmetic ignores activation traffic and kernel overheads. The provider does not publish its hardware, its batch size or its scheduler, so the measured numbers include queueing and other users' load, and comparing them with an H100's ceiling assumes a GPU the provider does not name. INT8 here means weight-only quantisation with arithmetic still in BF16.
The maths
- The roofline
Every step is limited by whichever takes longer: the arithmetic at the chip's peak rate, or the bytes at its memory bandwidth. Below 295 operations per byte an H100 waits on memory (the slope on the terrain); above it, on arithmetic (the flat roof). The GPU pills swap in the A100 (153) and the RTX 4090 (164).
- Prefill: the whole prompt in one pass
P is the parameter count, N the prompt length, L the layers and d the hidden size. Two operations per weight per token, plus causal attention, which grows with the square of the prompt: a thirtieth of the work at 2,000 tokens, a third of it at 32,768. Drag the prompt slider to move the prefill point along the roof.
- Decode: one token, every weight
Each step reads every weight once for the whole batch B, and each sequence's KV cache of C tokens at k = 128 KiB per token. At batch 1 that is one operation per byte, far down the slope. The batch slider shares the weight read across more users, which is what moves decode up the slope.
- Why the cache slows decode
A 16,361-token prompt adds 2.1 GB of keys and values that every decode step rereads. That is 13 percent more bytes than the weights alone, and the measured decode rate on the live endpoint fell by 13 percent, from 156 to 136 tokens a second.
Related terms
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- Input tokensInput tokens are the tokens you send in a request: the system prompt, the conversation history, loaded files, and tool definitions. You are billed for them, and they count against the context window.
- Output tokensOutput tokens are the tokens a model generates in its response, including any hidden reasoning. They are usually priced higher than input tokens, and turning up effort produces more of them.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.