# Token decoding

**AI Engineering Visualised** · Inference · Interactive explainer · Updated 2026-08-16

A language model outputs a probability distribution over the next token, not a token. Sampling settings such as temperature, top-k and top-p decide how that distribution is narrowed before one token is drawn.

> A model does not produce an answer, it produces a distribution over next tokens. Move temperature, top-k and top-p and watch the fan of possible continuations widen or collapse.

Ask why a model gave you a different answer the second time and the honest reply is that it never had one answer. At every step it produces a probability distribution over the whole vocabulary, and something has to turn that distribution into a single token. That something is the sampler, and it is configuration, not intelligence.

The left panel is one such distribution. The right panel is what happens when you sample it two hundred times and follow each continuation forward. Greedy decoding collapses the fan to a single line: the most likely token, every time, which is why it reads as competent and dull. Raising temperature flattens the distribution, so tokens that were nearly impossible become merely unlikely, and the fan spreads.

Top-p is the control that stops that becoming nonsense. It keeps only the smallest set of tokens whose probability sums to p, then renormalises. Watch the cut tokens on the left as you drag it: at 0.9 the absurd tail is gone, and the model can be varied without being unhinged. Set top-p back to 1.0 at temperature 2 to see the alternative.

The order matters and is easy to get backwards. Temperature is applied first, then top-k, then the nucleus is taken from the already-flattened distribution. That makes top-p a self-adjusting brake rather than a fixed one: raising temperature moves mass into the tail, so top-p admits more tokens, not fewer. The "tokens cut" counter drops as you heat things up, which is the opposite of what most descriptions imply, and it is why a temperature increase usually needs a tighter p to stay in the same place.

Top-k is the blunter sibling. It keeps a fixed number of candidates regardless of how confident the model is, which means it cuts too little when the distribution is flat and too much when the model is certain. That is why nucleus sampling largely replaced it, and why the two are applied in that order when both are on.

The logits here are hand-authored to make the tail obviously ridiculous rather than merely unlikely. The behaviour they demonstrate is not: any distribution with a heavy head and a long tail responds this way, which is why these two knobs appear in every inference API you will use.

## The maths

### Softmax with temperature

```latex
p_i \;=\; \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}
```

The temperature slider is T. Dividing the logits by a larger T pushes them closer together before exponentiating, so the distribution flattens. As T approaches zero the largest logit dominates completely and sampling becomes argmax.

### Nucleus (top-p) set

```latex
V_p \;=\; \arg\min_{S} \; |S| \quad \text{s.t.} \quad \sum_{i \in S} p_i \;\ge\; p
```

The smallest set of tokens whose probability sums to at least p. Because it is taken after temperature has been applied, raising T moves mass into the tail and the set gets larger, not smaller.

### Renormalisation after masking

```latex
p'_i \;=\; \frac{p_i \, \mathbb{1}[i \in V_p]}{\sum_{j \in V_p} p_j}
```

Cut tokens get probability zero and the survivors are rescaled to sum to one again. This is what the solid bars show; the faint bars behind them are the distribution before the cut.

### Entropy of the sampling distribution

```latex
H \;=\; -\sum_i p'_i \log_2 p'_i
```

The one-number summary of how undecided the model is, in bits. Zero means one token is certain; log2(n) means all n candidates are equally likely.

## Related terms

- [Next-token prediction](https://understandingdata.com/ai-coding-dictionary/next-token-prediction/)
- [Non-determinism](https://understandingdata.com/ai-coding-dictionary/non-determinism/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)
- [Token](https://understandingdata.com/ai-coding-dictionary/token/)

## More visualisations

- [Beam search](https://understandingdata.com/ai-engineering-visualised/beam-search/)
- [Embedding space](https://understandingdata.com/ai-engineering-visualised/embedding-space/)

---

Source: https://understandingdata.com/ai-engineering-visualised/token-decoding/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-08-16.