# Softmax, temperature and cross-entropy **AI Engineering Visualised** · Foundations · Film, 1:38 · Updated 2026-09-11 Softmax turns a vector of raw scores into probabilities by exponentiating each one and dividing by the sum. Temperature scales the scores first, so a low temperature sharpens the distribution and a high one flattens it. > The softmax function turns raw logits into probabilities in two moves: exponentiate, then divide by the sum. Watch a machine do it with the real numbers printed at every step, then heat it with temperature and read the loss off a gauge. Watch the film: https://understandingdata.com/ai-engineering-visualised/softmax/ Video file: https://assets.understandingdata.com/viz/softmax/video/film.mp4?v=cd5858f4b2 A language model does not emit a word. It emits a list of raw scores, one per token in its vocabulary, and those scores are called logits. They can be any real number, they do not add up to anything, and you cannot sample from them. Press play and the film opens on four of them, printed on cards in a rack: 4.0 for Paris, 2.0 for Lyon, 1.5 for France, minus 1.0 for the, the model's scores for the next token after "The capital of France is". The softmax function is two moves, and the machine does them in order. First the bars ride into the glass chamber and each grows to the exponential of its score, with the new number printed above it: e to the 4 is about 55, e to the 2 about 7.4, and e to the minus 1 is 0.37, still positive. That is the step people skip over, and it is where the shape of the answer is decided. A gap of two logit points is not a gap of two of anything; it is a ratio of about seven to one, because the exponential turns differences into ratios. Second, the grown bars pour into the share tube and are divided by their total. Now the segments add to exactly one and each is a probability: about 80% for Paris on the confident preset, 11% for Lyon, and so on down. The tube is the whole reason the exponential is there. Exponentials are always positive, so every token gets a share, and dividing by the sum makes the shares a distribution you can sample from. Temperature is the dial on the side of the chamber. It divides every logit before the exponential. Below one the gaps between logits are stretched, the biggest exponential runs away from the rest and one segment takes over the tube; above one the gaps are squashed, the exponentials level out and the tube divides more evenly. The film sweeps the dial from cold to hot and back to your slider so the shares and the printed numbers can be watched changing together. Cross-entropy is what training minimises, and against a single true label it is nothing more than minus the log of the share given to that label. The gauge on the right reads it in bits. In the film the true label's segment lights gold and the needle settles; then the label switches briefly to Lyon, the lit segment shrinks, and the needle jumps hard right. The loss never looks at the other segments. It only asks how much the model gave the right answer. KL divergence is the one that most explanations leave vague. It compares two distributions, not a distribution and a label. The film stacks a second tube above the first, holding the heated shares, and draws a bar between them for each token: that token's share times the log of the ratio of the two shares. The bars add up to the divergence. Swap which tube does the believing and the bars and their total change, which is the whole point: KL is not a distance, and the direction matters. Entropy is the one-number summary of how spread the tube is, in bits: zero when one segment has everything, two bits when four segments are equal. Perplexity is two to the power of that, and the last chapter redraws the tube as that many equal slices. A perplexity of 1.7 means the model is exactly as uncertain as a fair choice between 1.7 equally likely options. Four tokens is a simplification that costs nothing: a real vocabulary has around a hundred thousand, and the machine is the same, only the tube has more segments. ## The maths ### The softmax function with temperature ```latex q_i \;=\; \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)} ``` The logits z are the printed scores on the cards; T is the dial on the chamber. Dividing by T before exponentiating stretches or squashes the gaps between logits. As T falls towards zero the largest exponential dwarfs the rest and one segment takes the whole tube; as T grows the exponentials level out and the tube divides evenly. ### Cross-entropy loss against the true label ```latex H(y, q) \;=\; -\sum_i y_i \log_2 q_i \;=\; -\log_2 q_{\text{label}} ``` With a one-hot target y this collapses to minus the log probability the model gave the correct token. It is the training loss for every language model. On screen it is the gauge: the needle reads the log of the lit segment in the tube, near zero when the model is confidently right, hard right when it is confidently wrong, off the scale if the label gets no share at all. ### KL divergence between two distributions ```latex D_{\mathrm{KL}}(p \,\|\, q) \;=\; \sum_i p_i \log_2 \frac{p_i}{q_i} \;=\; H(p, q) - H(p) ``` The extra bits you pay for believing q when the truth is p. Here p is the tube at temperature one and q the heated tube above it, and the bars between them are the per-token terms of the sum. Swap the direction and the terms and their total change: KL is not a distance, and which side does the believing matters. ### Entropy and perplexity ```latex H(q) \;=\; -\sum_i q_i \log_2 q_i, \qquad \mathrm{PPL} \;=\; 2^{H} ``` Entropy is how undecided the distribution is, in bits. Perplexity is the same number expressed as an effective count of options: a flat distribution over four tokens has perplexity four. Reported language-model perplexities are two (or e) to the power of the average cross-entropy over a test set, which is why a lower number is better. ## Transcript ### 0:00 Logits Every step ends with one raw score per candidate token, called a logit: here Paris scores four, Lyon two, France one and a half, and the word the minus one, and none of that is a probability yet. ### 0:17 Exponentiate So first each score goes through the exponential, which makes every value positive and turns Paris's two point lead over Lyon into 54.6 against 7.4, about seven times, and real code subtracts the largest score first so nothing overflows. ### 0:35 Normalise Then divide each one by their sum, 66.8, and the shares add to exactly one: Paris owns 82 percent. ### 0:45 Temperature The part people get backwards is the dial: temperature divides the logits before the exponential, so below one it sharpens the shares and above one it flattens them. ### 0:56 Cross-entropy Training scores only the share on the right answer, so being confident and wrong is what costs: make Lyon the answer, at 11 percent, and the gauge jumps past three bits. ### 1:08 KL both ways When one model is distilled into another, KL divergence compares their whole distributions: each bar is a token's share times its log ratio, and swapping which tube does the believing changes the total, because KL is not a distance. ### 1:25 Perplexity The takeaway is perplexity: this model is as unsure as a choice between 1.9 equal options, and that is what temperature really sets. ## Related terms - [Next-token prediction](https://understandingdata.com/ai-coding-dictionary/next-token-prediction/) - [Inference](https://understandingdata.com/ai-coding-dictionary/inference/) - [Token](https://understandingdata.com/ai-coding-dictionary/token/) ## More visualisations - [Token decoding](https://understandingdata.com/ai-engineering-visualised/token-decoding/) - [Beam search](https://understandingdata.com/ai-engineering-visualised/beam-search/) - [The loss landscape](https://understandingdata.com/ai-engineering-visualised/loss-landscape/) --- Source: https://understandingdata.com/ai-engineering-visualised/softmax/ From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.