# Attention head fingerprints

**AI Engineering Visualised** · Interpretability · Interactive explainer · Updated 2026-08-16

An attention head is one independent copy of the attention mechanism inside a layer, with its own query, key and value projections. Different heads learn different jobs, from copying the previous token to matching earlier patterns.

> Heads are not redundant copies of each other. Step through twelve fingerprints and watch a previous-token head, an induction head and an attention sink behave in visibly different ways.

Nearly every explanation of attention shows a single heatmap. That is a reasonable way to introduce the mechanism and a misleading way to leave it, because it implies the heads are interchangeable and that a layer computes one weighted average. They are not, and it does not.

Look along the grid. Some heads put almost all their weight one position back, which builds the local signal that later layers compose into longer-range structure. Some sit on the first token almost regardless of input: these are attention sinks, and they are not broken. Softmax forces every row to sum to one, so a head with nothing useful to contribute at this position still has to put its mass somewhere, and the first token is a convenient dump. Removing it turns out to hurt, which is why streaming implementations keep the first few tokens pinned in the cache forever.

The interesting one is induction. It looks for an earlier occurrence of the token currently being processed and attends to whatever came next. Select it and follow the second "The": it points at "cat", because that is what followed "The" last time. That circuit is the best-understood mechanism behind in-context learning, and it is a large part of why a model can pick up a pattern from two examples in your prompt without any weights changing.

One structural caveat about the grid. Induction heads are not standalone: the circuit is a composition across depth, in which a previous-token head in an earlier layer writes the preceding token's identity into the residual stream, and the induction head in a later layer reads it back out. The heads drawn side by side here therefore live at different depths in a real model. They are shown together because the taxonomy is the point, not because they could coexist in one layer.

The spread number under the picture is normalised entropy: near zero means the head looks at exactly one place, near one means it spreads evenly across everything available. It separates a sharp positional head from a broad averaging one without you having to squint.

These patterns are synthesised from behaviours documented in the interpretability literature rather than pulled from a specific checkpoint. The taxonomy is real and reproducible across models. The exact numbers here are illustrative.

## The maths

### One head, one row

```latex
A_{ij} \;=\; \frac{\exp(s_{ij})}{\sum_{j' \le i} \exp(s_{ij'})}, \qquad s_{ij} = \frac{\mathbf{q}_i \cdot \mathbf{k}_j}{\sqrt{d}}, \quad j \le i
```

Each row of each heatmap is a softmax over the causal prefix, which is why every row sums to one and the upper triangle is empty. The sum-to-one constraint is also why a head with nothing useful to say still has to put its mass somewhere, which is what makes sinks exist.

### The spread statistic

```latex
\bar{H} \;=\; \frac{1}{n-1} \sum_{i \ge 1} \frac{-\sum_j A_{ij} \log_2 A_{ij}}{\log_2 (i+1)}
```

Normalised entropy averaged over rows: each row’s entropy divided by the maximum possible for its prefix width. Zero means the head looks at exactly one place; one means it spreads evenly over everything available.

## Related terms

- [Attention](https://understandingdata.com/ai-coding-dictionary/attention/)
- [Token](https://understandingdata.com/ai-coding-dictionary/token/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)
- [Model](https://understandingdata.com/ai-coding-dictionary/model/)

## More visualisations

- [FlashAttention](https://understandingdata.com/ai-engineering-visualised/flash-attention/)
- [Rotary position embeddings](https://understandingdata.com/ai-engineering-visualised/rotary-position-embeddings/)
- [Lost in the middle](https://understandingdata.com/ai-engineering-visualised/lost-in-the-middle/)

---

Source: https://understandingdata.com/ai-engineering-visualised/attention-heads/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-08-16.