Embedding space

The maths

Cosine similarity

$$\cos(\mathbf{a}, \mathbf{b}) \;=\; \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}$$

The measure behind the nearest-neighbour readout. It compares direction only, ignoring length, which is why embeddings are usually compared this way rather than by Euclidean distance.

Analogy arithmetic

$$\mathbf{v} \;=\; \mathbf{a} - \mathbf{b} + \mathbf{c}, \qquad \text{answer} \;=\; \arg\max_{w \notin \{a,b,c\}} \cos(\mathbf{v}, \mathbf{w})$$

The parallelogram the presets draw. king minus man plus woman lands near queen because the man-to-woman offset and the king-to-queen offset are approximately the same vector.

Why a picture hides structure

$$\mathbf{x}_{\text{drawn}} \;=\; P\,\mathbf{x}, \qquad P \in \mathbb{R}^{3 \times d}, \quad d \gg 3$$

Any plot of an embedding is a projection P from d dimensions down to the three you can see, and whatever varies only in the dropped directions vanishes. The film squashes one axis here and Paris lands exactly on France, because the capital relation lived entirely in that axis.

Transcript

The narration, chapter by chapter. A timestamp opens the film at that moment.

0:00A space of meaning

To compare words, a model turns each one into a point, placed so that nearness means similar meaning: here are forty-two of them, in three dimensions.

0:11Clusters

Train that space well and the points gather by meaning: the animals sit together, the professions sit together, and the capitals hover just above their countries.

0:22Neighbours

Trace one word: ask for the nearest neighbours of king, and uncle, prince and father come back, each with a similarity above 0.98.

0:34Directions

The surprise is that directions carry meaning: man to woman is the same arrow as king to queen, so king minus man plus woman lands exactly on queen, and every country to its capital is one more parallel arrow.

0:52Angle, not gap

Similarity is the angle between arrows from the origin, not the gap between the points: king and queen are four units apart and forty-four degrees apart, a similarity of 0.72, while prince, at ten degrees, scores 0.98.

1:12What pictures hide

The catch: real embeddings have hundreds or thousands of dimensions, and every picture drops most of them, so flatten just one here and each capital lands on its country, and each queen on her king.

1:28The takeaway

So rank neighbours by cosine, in the full space, and use a picture like this only for the shape of the groups, never for how close two words really are.

Also available as captions (WebVTT), the video file and a Markdown copy of this page.

Related terms

• TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.
• InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
• ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
• ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.