A transformer has no inherent notion of order. Attention is a sum over positions, and a sum does not care what order you write it in, so position has to be put in by hand. The obvious approaches add something to the embedding, which works but has a nagging problem: what the model learns about position five does not automatically transfer to position five thousand.
Rotary embeddings take a different route. Split the vector into two-dimensional slices and rotate each slice by an angle proportional to its position, with a different rate for each slice. Nothing is added. The vector is turned.
The payoff is what happens when a rotated query meets a rotated key. Rotating one vector by m and another by n and taking their dot product gives exactly the same answer as rotating one of them by m minus n and leaving the other alone. Absolute positions go in and only their difference comes out. Use the move-both buttons: the two score readouts stay identical however far along the sequence you slide, which is why the score curve at the bottom is a single curve rather than a family of them.
The frequency base decides how far this stays useful, though not for the reason it first appears. Each band is a hand on a clock, and at base ten thousand the slowest hand takes roughly fifty thousand tokens to complete a turn, far beyond the few thousand these models were originally trained on. Ambiguity from wrap-around is therefore not the binding problem.
The binding problem is the opposite. Because those slow bands have wavelengths longer than the training window, the model never observes a full period of them, so relative distances past what it was trained on produce phase values it has simply never encountered, and attention behaves badly on inputs it has no experience of. Raising the base stretches every wavelength, which keeps the fast bands recognisable while moving the slow ones further from the region that has never been seen. That is why long-context models raise the base to hundreds of thousands, and they do it alongside continued training on long documents rather than instead of it.
Base 100 shows the wrap-around failure directly, which is a real effect even though it is not the one that limits real models: every band cycles so quickly that distant positions produce the same set of angles.
The maths
- Band frequencies
Each two-dimensional slice of the vector turns at its own rate. The eight dials sample the bands of a 64-dimensional slice; the period stat is quoted at d = 128, where the slowest band at base 10,000 takes roughly 54,000 tokens per turn.
- The rotation
Position m rotates each slice by m times its band frequency. Nothing is added to the vector; it is turned, which preserves its length.
- The invariance
Rotations compose by adding angles and transposing negates one, so the two absolute positions collapse into their difference. This is the one-line proof behind the move-both buttons refusing to change the score.
Related terms
- AttentionAttention is the mechanism a model uses to weigh how strongly each token in its context relates to the others when predicting the next one. It is the basis of how a model actually uses context.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.