Ask why a model gave you a different answer the second time and the honest reply is that it never had one answer. At every step it produces a probability distribution over the whole vocabulary, and something has to turn that distribution into a single token. That something is the sampler, and it is configuration, not intelligence.
The left panel is one such distribution. The right panel is what happens when you sample it two hundred times and follow each continuation forward. Greedy decoding collapses the fan to a single line: the most likely token, every time, which is why it reads as competent and dull. Raising temperature flattens the distribution, so tokens that were nearly impossible become merely unlikely, and the fan spreads.
Top-p is the control that stops that becoming nonsense. It keeps only the smallest set of tokens whose probability sums to p, then renormalises. Watch the cut tokens on the left as you drag it: at 0.9 the absurd tail is gone, and the model can be varied without being unhinged. Set top-p back to 1.0 at temperature 2 to see the alternative.
The order matters and is easy to get backwards. Temperature is applied first, then top-k, then the nucleus is taken from the already-flattened distribution. That makes top-p a self-adjusting brake rather than a fixed one: raising temperature moves mass into the tail, so top-p admits more tokens, not fewer. The "tokens cut" counter drops as you heat things up, which is the opposite of what most descriptions imply, and it is why a temperature increase usually needs a tighter p to stay in the same place.
Top-k is the blunter sibling. It keeps a fixed number of candidates regardless of how confident the model is, which means it cuts too little when the distribution is flat and too much when the model is certain. That is why nucleus sampling largely replaced it, and why the two are applied in that order when both are on.
The logits here are hand-authored to make the tail obviously ridiculous rather than merely unlikely. The behaviour they demonstrate is not: any distribution with a heavy head and a long tail responds this way, which is why these two knobs appear in every inference API you will use.
The maths
- Softmax with temperature
The temperature slider is T. Dividing the logits by a larger T pushes them closer together before exponentiating, so the distribution flattens. As T approaches zero the largest logit dominates completely and sampling becomes argmax.
- Nucleus (top-p) set
The smallest set of tokens whose probability sums to at least p. Because it is taken after temperature has been applied, raising T moves mass into the tail and the set gets larger, not smaller.
- Renormalisation after masking
Cut tokens get probability zero and the survivors are rescaled to sum to one again. This is what the solid bars show; the faint bars behind them are the distribution before the cut.
- Entropy of the sampling distribution
The one-number summary of how undecided the model is, in bits. Zero means one token is certain; log2(n) means all n candidates are equally likely.
Related terms
- Next-token predictionNext-token prediction is the one job a language model does: given the text so far, predict the most likely next token, add it, and repeat. It is both the training objective and what runs at inference.
- Non-determinismNon-determinism is why the same prompt can give you different answers. At inference the model samples among likely next tokens with a controlled amount of randomness, so runs vary.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.