A language model does not emit a word. It emits a list of raw scores, one per token in its vocabulary, and those scores are called logits. They can be any real number, they do not add up to anything, and you cannot sample from them. Press play and the film opens on four of them, printed on cards in a rack: 4.0 for Paris, 2.0 for Lyon, 1.5 for France, minus 1.0 for the, the model's scores for the next token after "The capital of France is".
The softmax function is two moves, and the machine does them in order. First the bars ride into the glass chamber and each grows to the exponential of its score, with the new number printed above it: e to the 4 is about 55, e to the 2 about 7.4, and e to the minus 1 is 0.37, still positive. That is the step people skip over, and it is where the shape of the answer is decided. A gap of two logit points is not a gap of two of anything; it is a ratio of about seven to one, because the exponential turns differences into ratios.
Second, the grown bars pour into the share tube and are divided by their total. Now the segments add to exactly one and each is a probability: about 80% for Paris on the confident preset, 11% for Lyon, and so on down. The tube is the whole reason the exponential is there. Exponentials are always positive, so every token gets a share, and dividing by the sum makes the shares a distribution you can sample from.
Temperature is the dial on the side of the chamber. It divides every logit before the exponential. Below one the gaps between logits are stretched, the biggest exponential runs away from the rest and one segment takes over the tube; above one the gaps are squashed, the exponentials level out and the tube divides more evenly. The film sweeps the dial from cold to hot and back to your slider so the shares and the printed numbers can be watched changing together.
Cross-entropy is what training minimises, and against a single true label it is nothing more than minus the log of the share given to that label. The gauge on the right reads it in bits. In the film the true label's segment lights gold and the needle settles; then the label switches briefly to Lyon, the lit segment shrinks, and the needle jumps hard right. The loss never looks at the other segments. It only asks how much the model gave the right answer.
KL divergence is the one that most explanations leave vague. It compares two distributions, not a distribution and a label. The film stacks a second tube above the first, holding the heated shares, and draws a bar between them for each token: that token's share times the log of the ratio of the two shares. The bars add up to the divergence. Swap which tube does the believing and the bars and their total change, which is the whole point: KL is not a distance, and the direction matters.
Entropy is the one-number summary of how spread the tube is, in bits: zero when one segment has everything, two bits when four segments are equal. Perplexity is two to the power of that, and the last chapter redraws the tube as that many equal slices. A perplexity of 1.7 means the model is exactly as uncertain as a fair choice between 1.7 equally likely options. Four tokens is a simplification that costs nothing: a real vocabulary has around a hundred thousand, and the machine is the same, only the tube has more segments.
The maths
- The softmax function with temperature
The logits z are the printed scores on the cards; T is the dial on the chamber. Dividing by T before exponentiating stretches or squashes the gaps between logits. As T falls towards zero the largest exponential dwarfs the rest and one segment takes the whole tube; as T grows the exponentials level out and the tube divides evenly.
- Cross-entropy loss against the true label
With a one-hot target y this collapses to minus the log probability the model gave the correct token. It is the training loss for every language model. On screen it is the gauge: the needle reads the log of the lit segment in the tube, near zero when the model is confidently right, hard right when it is confidently wrong, off the scale if the label gets no share at all.
- KL divergence between two distributions
The extra bits you pay for believing q when the truth is p. Here p is the tube at temperature one and q the heated tube above it, and the bars between them are the per-token terms of the sum. Swap the direction and the terms and their total change: KL is not a distance, and which side does the believing matters.
- Entropy and perplexity
Entropy is how undecided the distribution is, in bits. Perplexity is the same number expressed as an effective count of options: a flat distribution over four tokens has perplexity four. Reported language-model perplexities are two (or e) to the power of the average cross-entropy over a test set, which is why a lower number is better.
Related terms
- Next-token predictionNext-token prediction is the one job a language model does: given the text so far, predict the most likely next token, add it, and repeat. It is both the training objective and what runs at inference.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.