# Attention as geometry

**AI Engineering Visualised** · Foundations · Interactive explainer · Updated 2026-08-16

Attention scores a query against every key with a dot product, then turns the scores into weights with a softmax. It is geometry: aligned vectors win weight, and longer vectors sharpen the distribution.

> Attention is dot products and a softmax. Drag the query and key vectors with your hands and feel it: alignment wins weight, and vector length is secretly a temperature.

Every explanation of attention eventually shows the formula, and the formula is honest but mute. It does not tell you what changing anything feels like. This page makes the vectors physical objects: drag them and the weights respond at every frame.

Three things are worth discovering with your hands rather than reading. First, the obvious one: swing the query toward a key and that key wins weight, because the dot product rewards alignment. Second, the one people miss: drag the query further from the origin without changing its angle. Every dot product grows in proportion, the softmax sharpens, and attention concentrates. Query magnitude is an inverse temperature, and nothing in the usual diagram tells you that. Third, the asymmetric counterpart: lengthening a single key raises only that key's score, so a key can win attention by being long rather than by being aligned, which is one reason real networks need normalisation layers to keep magnitudes from quietly deciding everything.

The dashed output line is the part most diagrams omit. Attention does not choose a key, it blends values in proportion to the weights, and here the values are the keys themselves, so the output visibly swings between vectors as you drag. When the weights flatten, it retreats toward the average of everything, which is what attention degenerating into a blur actually means.

The divide-by-root-d in the formula exists precisely because of the magnitude effect you just felt. In high dimensions, dot products of random vectors grow with the square root of the dimension, so without the correction the softmax would start saturated. What you produce here by dragging the query outward is the exact disease that scaling factor treats.

The honest limit: this plane is two-dimensional and the values are the keys themselves, both chosen so the geometry stays visible. Real attention runs the same equations at d of 64 or 128 with separate value vectors, where nothing changes except that you can no longer watch it happen.

## The maths

### Scaled dot-product attention

```latex
w_i \;=\; \frac{\exp\!\big(\mathbf{q} \cdot \mathbf{k}_i / \sqrt{d}\big)}{\sum_j \exp\!\big(\mathbf{q} \cdot \mathbf{k}_j / \sqrt{d}\big)}, \qquad \mathbf{o} \;=\; \sum_i w_i \, \mathbf{v}_i
```

The bars are w, the dashed arrow is o. Here d = 2 and the values are the keys themselves, so the output is a visible blend of the arrows you are dragging.

### Magnitude is temperature

```latex
\mathbf{q} \to c\,\mathbf{q} \;\;\Rightarrow\;\; \mathbf{q} \cdot \mathbf{k}_i \to c\,(\mathbf{q} \cdot \mathbf{k}_i)
```

Scaling the query multiplies every score by the same c, which is identical to dividing softmax temperature by c. Dragging the query outward is cooling; dragging it toward the origin is heating.

### Why the root-d correction exists

```latex
\mathbb{E}\big[(\mathbf{q} \cdot \mathbf{k})^2\big] \;=\; d \quad \text{for independent zero-mean, unit-variance components}
```

Random high-dimensional vectors have dot products that grow like root d, so without dividing by it the softmax would begin life saturated at exactly the sharp extreme you can produce here by stretching the query.

## Related terms

- [Attention](https://understandingdata.com/ai-coding-dictionary/attention/)
- [Token](https://understandingdata.com/ai-coding-dictionary/token/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)

## More visualisations

- [Attention head fingerprints](https://understandingdata.com/ai-engineering-visualised/attention-heads/)
- [Embedding space](https://understandingdata.com/ai-engineering-visualised/embedding-space/)
- [Token decoding](https://understandingdata.com/ai-engineering-visualised/token-decoding/)

---

Source: https://understandingdata.com/ai-engineering-visualised/attention-geometry/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-08-16.