Nearly every explanation of attention shows a single heatmap. That is a reasonable way to introduce the mechanism and a misleading way to leave it, because it implies the heads are interchangeable and that a layer computes one weighted average. They are not, and it does not.
Look along the grid. Some heads put almost all their weight one position back, which builds the local signal that later layers compose into longer-range structure. Some sit on the first token almost regardless of input: these are attention sinks, and they are not broken. Softmax forces every row to sum to one, so a head with nothing useful to contribute at this position still has to put its mass somewhere, and the first token is a convenient dump. Removing it turns out to hurt, which is why streaming implementations keep the first few tokens pinned in the cache forever.
The interesting one is induction. It looks for an earlier occurrence of the token currently being processed and attends to whatever came next. Select it and follow the second "The": it points at "cat", because that is what followed "The" last time. That circuit is the best-understood mechanism behind in-context learning, and it is a large part of why a model can pick up a pattern from two examples in your prompt without any weights changing.
One structural caveat about the grid. Induction heads are not standalone: the circuit is a composition across depth, in which a previous-token head in an earlier layer writes the preceding token's identity into the residual stream, and the induction head in a later layer reads it back out. The heads drawn side by side here therefore live at different depths in a real model. They are shown together because the taxonomy is the point, not because they could coexist in one layer.
The spread number under the picture is normalised entropy: near zero means the head looks at exactly one place, near one means it spreads evenly across everything available. It separates a sharp positional head from a broad averaging one without you having to squint.
These patterns are synthesised from behaviours documented in the interpretability literature rather than pulled from a specific checkpoint. The taxonomy is real and reproducible across models. The exact numbers here are illustrative.
The maths
- One head, one row
Each row of each heatmap is a softmax over the causal prefix, which is why every row sums to one and the upper triangle is empty. The sum-to-one constraint is also why a head with nothing useful to say still has to put its mass somewhere, which is what makes sinks exist.
- The spread statistic
Normalised entropy averaged over rows: each row’s entropy divided by the maximum possible for its prefix width. Zero means the head looks at exactly one place; one means it spreads evenly over everything available.
Related terms
- AttentionAttention is the mechanism a model uses to weigh how strongly each token in its context relates to the others when predicting the next one. It is the basis of how a model actually uses context.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.