Attention as geometry

Every explanation of attention eventually shows the formula, and the formula is honest but mute. It does not tell you what changing anything feels like. This page makes the vectors physical objects: drag them and the weights respond at every frame.

Three things are worth discovering with your hands rather than reading. First, the obvious one: swing the query toward a key and that key wins weight, because the dot product rewards alignment. Second, the one people miss: drag the query further from the origin without changing its angle. Every dot product grows in proportion, the softmax sharpens, and attention concentrates. Query magnitude is an inverse temperature, and nothing in the usual diagram tells you that. Third, the asymmetric counterpart: lengthening a single key raises only that key's score, so a key can win attention by being long rather than by being aligned, which is one reason real networks need normalisation layers to keep magnitudes from quietly deciding everything.

The dashed output line is the part most diagrams omit. Attention does not choose a key, it blends values in proportion to the weights, and here the values are the keys themselves, so the output visibly swings between vectors as you drag. When the weights flatten, it retreats toward the average of everything, which is what attention degenerating into a blur actually means.

The divide-by-root-d in the formula exists precisely because of the magnitude effect you just felt. In high dimensions, dot products of random vectors grow with the square root of the dimension, so without the correction the softmax would start saturated. What you produce here by dragging the query outward is the exact disease that scaling factor treats.

The honest limit: this plane is two-dimensional and the values are the keys themselves, both chosen so the geometry stays visible. Real attention runs the same equations at d of 64 or 128 with separate value vectors, where nothing changes except that you can no longer watch it happen.