# Prefill vs decode: time to first token and tokens per second

**AI Engineering Visualised** · Inference · Film, 2:34 · Updated 2026-09-11

LLM inference has two phases. Prefill processes the whole prompt in one parallel pass and sets the time to first token; decode then produces one token at a time, rereading the model's weights for each, which is why tokens per second is bounded by memory bandwidth.

> Press play and watch one GPU serve Llama 3.1 8B: prefill floods the whole prompt through at once and is limited by arithmetic, while decode drips one token at a time, rereads all 16 GB of weights for each, and is limited by memory bandwidth, with real timings measured on a live endpoint.

Watch the film: https://understandingdata.com/ai-engineering-visualised/prefill-decode/
Video file: https://assets.understandingdata.com/viz/prefill-decode/video/film.mp4?v=92404b6457

What is the difference between prefill and decode? When a language model answers, it does two different jobs. Prefill reads your whole prompt in one pass and builds the model's working memory of it, the KV cache; the time it takes is most of the time to first token. Decode then writes the answer one token at a time, each step reading the whole model again; its speed is the tokens per second you watch stream in. The two are limited by different parts of the GPU, which is why a long prompt makes you wait for the first word and then the rest arrive at a steady pace.

The film follows one GPU, an H100, serving Llama 3.1 8B in BF16: 8.03 billion parameters, 16 GB of weights, drawn as the wall behind the chip. Prefill is the sluice: 2,000 prompt tokens flood through together, every weight is read once for all of them, and the chip does 33 trillion operations in about 34 milliseconds with every core busy. Decode is the tap: each drop is one token, and each one needs the whole wall read again for only 16 billion operations. At 3.35 TB/s that is 4.8 milliseconds a token, about 209 a second at best, while most of the chip's arithmetic sits idle.

The roofline explains it. Divide a step's operations by the bytes it moves and you get its arithmetic intensity. The H100's ridge is at 295 operations per byte: below it the chip waits on memory, above it on arithmetic. Decode at batch 1 does one operation per byte and sits low on the memory slope; prefill of 2,000 tokens does about 2,000 and sits on the flat compute roof. So the two metrics answer to different hardware: time to first token to arithmetic (and, for long prompts, to attention, which grows with the square of the prompt), tokens per second to bandwidth.

That is also why serving systems batch decode. When 64 users decode together, one read of the weights serves all 64: the step grows only from 4.8 to 5.4 milliseconds while the chip produces over 11,000 tokens a second. Each sequence still reads its own KV cache, so batching climbs the slope more slowly than you might hope; even at 295 users this model stays memory bound. Continuous batching and paged KV caches exist to keep that batch full. Quantising the weights to 8 bits halves the bytes and roughly doubles batch-1 decode.

The measurements are real. Through OpenRouter, with the provider pinned to CoreWeave's BF16 endpoint, the first word arrived after about 0.3 seconds for a 34-token or a 2,081-token prompt (network and queueing dominate at that size) and after 0.74 seconds for 16,361 tokens. Decode ran at 156 tokens a second, about three quarters of an H100's ceiling, and 13 percent slower with the long prompt, matching the 13 percent of extra bytes its 2.1 GB of KV cache adds to every step. The same model on another provider's FP8 endpoint decoded at 48 tokens a second: the speed you get is whatever bandwidth and batch share the provider gives you.

The honest caveats. The roofline is a ceiling, not a prediction: real kernels reach perhaps 40 to 70 percent of peak arithmetic and 70 to 90 percent of peak bandwidth, and the film's arithmetic ignores activation traffic and kernel overheads. The provider does not publish its hardware, its batch size or its scheduler, so the measured numbers include queueing and other users' load, and comparing them with an H100's ceiling assumes a GPU the provider does not name. INT8 here means weight-only quantisation with arithmetic still in BF16.

## The maths

### The roofline

```latex
\text{time} \;=\; \max\!\left(\frac{\text{operations}}{\text{peak FLOP/s}},\; \frac{\text{bytes moved}}{\text{bandwidth}}\right), \qquad \text{ridge} \;=\; \frac{989\ \text{TFLOP/s}}{3.35\ \text{TB/s}} \;\approx\; 295\ \text{ops per byte}
```

Every step is limited by whichever takes longer: the arithmetic at the chip's peak rate, or the bytes at its memory bandwidth. Below 295 operations per byte an H100 waits on memory (the slope on the terrain); above it, on arithmetic (the flat roof). The GPU pills swap in the A100 (153) and the RTX 4090 (164).

### Prefill: the whole prompt in one pass

```latex
T_{\text{first}} \;\approx\; \frac{2PN + 2L d N^2}{\text{peak}} \;=\; \frac{2(8.03\times10^{9})(2000) + 2(32)(4096)(2000)^2}{989\times10^{12}} \;\approx\; 34\ \text{ms}
```

P is the parameter count, N the prompt length, L the layers and d the hidden size. Two operations per weight per token, plus causal attention, which grows with the square of the prompt: a thirtieth of the work at 2,000 tokens, a third of it at 32,768. Drag the prompt slider to move the prefill point along the roof.

### Decode: one token, every weight

```latex
\frac{\text{tokens}}{\text{second}} \;\le\; \frac{\text{bandwidth}}{\text{weight bytes} + B\,C\,k} \;=\; \frac{3.35\ \text{TB/s}}{16.06\ \text{GB}} \;\approx\; 209 \quad (B = 1,\ C = 0)
```

Each step reads every weight once for the whole batch B, and each sequence's KV cache of C tokens at k = 128 KiB per token. At batch 1 that is one operation per byte, far down the slope. The batch slider shares the weight read across more users, which is what moves decode up the slope.

### Why the cache slows decode

```latex
\frac{C\,k}{\text{weight bytes}} \;=\; \frac{16{,}361 \times 131{,}072}{16.06\times10^{9}} \;\approx\; 13\%
```

A 16,361-token prompt adds 2.1 GB of keys and values that every decode step rereads. That is 13 percent more bytes than the weights alone, and the measured decode rate on the live endpoint fell by 13 percent, from 156 to 136 tokens a second.

## Transcript

### 0:00 Two waits

Paste a long document into a chat and you wait for the first word, then the rest arrive at a steady pace, because answering is two jobs, prefill and decode, each limited by a different part of the GPU.

### 0:14 Prefill

Prefill pushes the whole prompt through in one pass: 2,000 tokens of Llama 3.1 8B is 33 trillion operations, and at an H100's 989 teraflops that takes about 34 milliseconds, with every core busy.

### 0:33 Decode

Decode is a drip: each new token rereads all 16 gigabytes of weights for just 16 billion operations, so even at 3.35 terabytes a second an H100 needs 4.8 milliseconds a token, about 209 a second.

### 0:52 The roofline

That is the part people miss: decode does one operation per byte it moves and sits low on the roofline's memory slope, while prefill does 2,000 and sits on the flat compute roof, past the ridge at 295.

### 1:07 Batching

So servers batch decode: 64 users share one read of the weights, the step grows only from 4.8 to 5.4 milliseconds, and the chip turns out over 11,000 tokens a second instead of 209.

### 1:23 Wasted money

The failure mode is buying arithmetic for chat: an H100 has three times an A100's teraflops but only 1.6 times its bandwidth, so one user's decode gets 1.6 times faster, while a 32,000-token prompt still waits 0.8 seconds, a third of that in attention.

### 1:45 Measured

Measured on a real endpoint, BF16 Llama 3.1 8B on CoreWeave, the first word came after about 0.3 seconds for a short prompt or one of 2,000 tokens, mostly network, and 0.74 seconds for 16,000.

### 2:03 The cache rides along

Decode ran at 156 tokens a second, then 13 percent slower with the long prompt, because each step also rereads that prompt's 2.1 gigabytes of KV cache, 13 percent more bytes than the weights alone.

### 2:18 Two budgets

So treat them as two budgets: cut time to first token by trimming and caching the prompt, and buy tokens per second with bandwidth, quantisation and batching, not with faster arithmetic.

## Related terms

- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)
- [Input tokens](https://understandingdata.com/ai-coding-dictionary/input-tokens/)
- [Output tokens](https://understandingdata.com/ai-coding-dictionary/output-tokens/)
- [Parameters](https://understandingdata.com/ai-coding-dictionary/parameters/)
- [Context window](https://understandingdata.com/ai-coding-dictionary/context-window/)

## More visualisations

- [KV cache and paging](https://understandingdata.com/ai-engineering-visualised/kv-cache/)
- [Continuous batching](https://understandingdata.com/ai-engineering-visualised/continuous-batching/)
- [Speculative decoding](https://understandingdata.com/ai-engineering-visualised/speculative-decoding/)
- [Quantisation and outliers](https://understandingdata.com/ai-engineering-visualised/quantisation/)
- [Matrix multiplication on a GPU](https://understandingdata.com/ai-engineering-visualised/gpu-matmul/)
- [The transformer block, exploded](https://understandingdata.com/ai-engineering-visualised/transformer-block/)

---

Source: https://understandingdata.com/ai-engineering-visualised/prefill-decode/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.