Interpretability

Attention head fingerprints

Heads are not redundant copies of each other. Step through twelve fingerprints and watch a previous-token head, an induction head and an attention sink behave in visibly different ways.

Head
Highlight role
TWELVE HEADS, DIFFERENT ROLESH1 Previous tokenH2 InductionH3 Attention sinkH4 DelimiterH5 Self / identityH6 Previous tokenH7 Broad averageH8 InductionH9 Fixed offsetH10 Attention sinkH11 DelimiterH12 Broad averageHEAD 2: INDUCTION1.000.500.500.330.330.330.250.250.250.250.990.980.140.140.140.140.140.140.140.98Thecatsat.Thecatran.Thecatsat.Thecatran.rows attend to columns. upper triangle is causally masked.patterns synthesised from documented head behaviours, not extracted weights.
2 of 12
Head
Induction
Role
0.60mixed
Spread
7
Distinct roles

Finds an earlier occurrence of the current token and attends to whatever followed it. This is the best-understood mechanism behind in-context learning.

Nearly every explanation of attention shows a single heatmap. That is a reasonable way to introduce the mechanism and a misleading way to leave it, because it implies the heads are interchangeable and that a layer computes one weighted average. They are not, and it does not.

Look along the grid. Some heads put almost all their weight one position back, which builds the local signal that later layers compose into longer-range structure. Some sit on the first token almost regardless of input: these are attention sinks, and they are not broken. Softmax forces every row to sum to one, so a head with nothing useful to contribute at this position still has to put its mass somewhere, and the first token is a convenient dump. Removing it turns out to hurt, which is why streaming implementations keep the first few tokens pinned in the cache forever.

The interesting one is induction. It looks for an earlier occurrence of the token currently being processed and attends to whatever came next. Select it and follow the second "The": it points at "cat", because that is what followed "The" last time. That circuit is the best-understood mechanism behind in-context learning, and it is a large part of why a model can pick up a pattern from two examples in your prompt without any weights changing.

One structural caveat about the grid. Induction heads are not standalone: the circuit is a composition across depth, in which a previous-token head in an earlier layer writes the preceding token's identity into the residual stream, and the induction head in a later layer reads it back out. The heads drawn side by side here therefore live at different depths in a real model. They are shown together because the taxonomy is the point, not because they could coexist in one layer.

The spread number under the picture is normalised entropy: near zero means the head looks at exactly one place, near one means it spreads evenly across everything available. It separates a sharp positional head from a broad averaging one without you having to squint.

These patterns are synthesised from behaviours documented in the interpretability literature rather than pulled from a specific checkpoint. The taxonomy is real and reproducible across models. The exact numbers here are illustrative.

The maths

One head, one row
Aij  =  exp(sij)jiexp(sij),sij=qikjd,jiA_{ij} \;=\; \frac{\exp(s_{ij})}{\sum_{j' \le i} \exp(s_{ij'})}, \qquad s_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}}, \quad j \le i

Each row of each heatmap is a softmax over the causal prefix, which is why every row sums to one and the upper triangle is empty. The sum-to-one constraint is also why a head with nothing useful to say still has to put its mass somewhere, which is what makes sinks exist.

The spread statistic
Hˉ  =  1n1i1jAijlog2Aijlog2(i+1)\bar{H} \;=\; \frac{1}{n-1} \sum_{i \ge 1} \frac{-\sum_j A_{ij} \log_2 A_{ij}}{\log_2 (i+1)}

Normalised entropy averaged over rows: each row’s entropy divided by the maximum possible for its prefix width. Zero means the head looks at exactly one place; one means it spreads evenly over everything available.

Related terms

More visualisations

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help