Inference

Prefill vs decode: time to first token and tokens per second

Press play and watch one GPU serve Llama 3.1 8B: prefill floods the whole prompt through at once and is limited by arithmetic, while decode drips one token at a time, rereads all 16 GB of weights for each, and is limited by memory bandwidth, with real timings measured on a live endpoint.

GPU
Weights
Prompt tokens
Batch
Loading 3D view

Two waitsPaste a long document into a chat and you wait for the first word, then the rest arrive at a steady pace, because answering is two jobs, prefill and decode, each limited by a different part of the GPU.

Narrated with James Phoenix's AI voice.

33.5 ms2,000 tokens, compute bound
Time to first token
209 tok/sper user, memory bound
Decode
16.1 GBBF16
Weights read per step
295ops per byte, H100 SXM
Ridge

Two waits: the pause before the first word is prefill; the steady stream after it is decode.

What is the difference between prefill and decode? When a language model answers, it does two different jobs. Prefill reads your whole prompt in one pass and builds the model's working memory of it, the KV cache; the time it takes is most of the time to first token. Decode then writes the answer one token at a time, each step reading the whole model again; its speed is the tokens per second you watch stream in. The two are limited by different parts of the GPU, which is why a long prompt makes you wait for the first word and then the rest arrive at a steady pace.

The film follows one GPU, an H100, serving Llama 3.1 8B in BF16: 8.03 billion parameters, 16 GB of weights, drawn as the wall behind the chip. Prefill is the sluice: 2,000 prompt tokens flood through together, every weight is read once for all of them, and the chip does 33 trillion operations in about 34 milliseconds with every core busy. Decode is the tap: each drop is one token, and each one needs the whole wall read again for only 16 billion operations. At 3.35 TB/s that is 4.8 milliseconds a token, about 209 a second at best, while most of the chip's arithmetic sits idle.

The roofline explains it. Divide a step's operations by the bytes it moves and you get its arithmetic intensity. The H100's ridge is at 295 operations per byte: below it the chip waits on memory, above it on arithmetic. Decode at batch 1 does one operation per byte and sits low on the memory slope; prefill of 2,000 tokens does about 2,000 and sits on the flat compute roof. So the two metrics answer to different hardware: time to first token to arithmetic (and, for long prompts, to attention, which grows with the square of the prompt), tokens per second to bandwidth.

That is also why serving systems batch decode. When 64 users decode together, one read of the weights serves all 64: the step grows only from 4.8 to 5.4 milliseconds while the chip produces over 11,000 tokens a second. Each sequence still reads its own KV cache, so batching climbs the slope more slowly than you might hope; even at 295 users this model stays memory bound. Continuous batching and paged KV caches exist to keep that batch full. Quantising the weights to 8 bits halves the bytes and roughly doubles batch-1 decode.

The measurements are real. Through OpenRouter, with the provider pinned to CoreWeave's BF16 endpoint, the first word arrived after about 0.3 seconds for a 34-token or a 2,081-token prompt (network and queueing dominate at that size) and after 0.74 seconds for 16,361 tokens. Decode ran at 156 tokens a second, about three quarters of an H100's ceiling, and 13 percent slower with the long prompt, matching the 13 percent of extra bytes its 2.1 GB of KV cache adds to every step. The same model on another provider's FP8 endpoint decoded at 48 tokens a second: the speed you get is whatever bandwidth and batch share the provider gives you.

The honest caveats. The roofline is a ceiling, not a prediction: real kernels reach perhaps 40 to 70 percent of peak arithmetic and 70 to 90 percent of peak bandwidth, and the film's arithmetic ignores activation traffic and kernel overheads. The provider does not publish its hardware, its batch size or its scheduler, so the measured numbers include queueing and other users' load, and comparing them with an H100's ceiling assumes a GPU the provider does not name. INT8 here means weight-only quantisation with arithmetic still in BF16.

The maths

The roofline
time  =  max ⁣(operationspeak FLOP/s,  bytes movedbandwidth),ridge  =  989 TFLOP/s3.35 TB/s    295 ops per byte\text{time} \;=\; \max\!\left(\frac{\text{operations}}{\text{peak FLOP/s}},\; \frac{\text{bytes moved}}{\text{bandwidth}}\right), \qquad \text{ridge} \;=\; \frac{989\ \text{TFLOP/s}}{3.35\ \text{TB/s}} \;\approx\; 295\ \text{ops per byte}

Every step is limited by whichever takes longer: the arithmetic at the chip's peak rate, or the bytes at its memory bandwidth. Below 295 operations per byte an H100 waits on memory (the slope on the terrain); above it, on arithmetic (the flat roof). The GPU pills swap in the A100 (153) and the RTX 4090 (164).

Prefill: the whole prompt in one pass
Tfirst    2PN+2LdN2peak  =  2(8.03×109)(2000)+2(32)(4096)(2000)2989×1012    34 msT_{\text{first}} \;\approx\; \frac{2PN + 2L d N^2}{\text{peak}} \;=\; \frac{2(8.03\times10^{9})(2000) + 2(32)(4096)(2000)^2}{989\times10^{12}} \;\approx\; 34\ \text{ms}

P is the parameter count, N the prompt length, L the layers and d the hidden size. Two operations per weight per token, plus causal attention, which grows with the square of the prompt: a thirtieth of the work at 2,000 tokens, a third of it at 32,768. Drag the prompt slider to move the prefill point along the roof.

Decode: one token, every weight
tokenssecond    bandwidthweight bytes+BCk  =  3.35 TB/s16.06 GB    209(B=1, C=0)\frac{\text{tokens}}{\text{second}} \;\le\; \frac{\text{bandwidth}}{\text{weight bytes} + B\,C\,k} \;=\; \frac{3.35\ \text{TB/s}}{16.06\ \text{GB}} \;\approx\; 209 \quad (B = 1,\ C = 0)

Each step reads every weight once for the whole batch B, and each sequence's KV cache of C tokens at k = 128 KiB per token. At batch 1 that is one operation per byte, far down the slope. The batch slider shares the weight read across more users, which is what moves decode up the slope.

Why the cache slows decode
Ckweight bytes  =  16,361×131,07216.06×109    13%\frac{C\,k}{\text{weight bytes}} \;=\; \frac{16{,}361 \times 131{,}072}{16.06\times10^{9}} \;\approx\; 13\%

A 16,361-token prompt adds 2.1 GB of keys and values that every decode step rereads. That is 13 percent more bytes than the weights alone, and the measured decode rate on the live endpoint fell by 13 percent, from 156 to 136 tokens a second.

Related terms

More visualisations

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help