Representations

Embedding space

Words become points in space, and the distance between them carries meaning. Explore the clusters in 2D, rotate them in 3D, and watch analogy arithmetic land on the word you expect.

Analogy
View
boygirlbrothersistermanwomanfathermotheruncleauntprinceprincesskingqueen
peoplecountriescapitalsanimalsprofessions

Closest to king: uncle (0.988), prince (0.984), father (0.982), brother (0.971)

An embedding turns a piece of text into a list of numbers. That sounds unremarkable until you notice what the numbers do: words that mean similar things end up close together, and the directions between words turn out to carry meaning of their own.

The clusters are the easy part to see. Animals sit near other animals, countries near other countries. The more surprising property is that relationships become directions. The step from "man" to "woman" is roughly the same step as the one from "king" to "queen", so you can do arithmetic on meaning: take king, subtract man, add woman, and you land next to queen.

Try the analogy presets below, then switch to 3D and drag to orbit the cloud, which is how you tell that the structure is genuinely spatial rather than an artefact of how it was drawn. The nearest-neighbour readout under the plot uses cosine similarity, direction rather than distance, because embedding length mostly tracks word frequency and direction is where the meaning lives.

A necessary caveat: the vectors here are illustrative, hand-built to show the geometry clearly. Real embeddings have hundreds or thousands of dimensions, and no one can look at those directly. These were authored in three so the geometry survives being drawn, which is the compromise every picture of an embedding space makes.

The maths

Cosine similarity
cos(a,b)  =  abab\cos(\mathbf{a}, \mathbf{b}) \;=\; \frac{\mathbf{a} \cdot \mathbf{b}}{\lVert \mathbf{a} \rVert \, \lVert \mathbf{b} \rVert}

The measure behind the nearest-neighbour readout. It compares direction only, ignoring length, which is why embeddings are usually compared this way rather than by Euclidean distance.

Analogy arithmetic
v  =  ab+c,answer  =  argmaxw{a,b,c}cos(v,w)\mathbf{v} \;=\; \mathbf{a} - \mathbf{b} + \mathbf{c}, \qquad \text{answer} \;=\; \arg\max_{w \notin \{a,b,c\}} \cos(\mathbf{v}, \mathbf{w})

The parallelogram the presets draw. king minus man plus woman lands near queen because the man-to-woman offset and the king-to-queen offset are approximately the same vector.

The 2D projection
u,w  =  top eigenvectors of    Σ  =  1ni(xixˉ)(xixˉ) ⁣\mathbf{u}, \mathbf{w} \;=\; \text{top eigenvectors of}\;\; \Sigma \;=\; \tfrac{1}{n}\textstyle\sum_i (\mathbf{x}_i - \bar{\mathbf{x}})(\mathbf{x}_i - \bar{\mathbf{x}})^{\!\top}

The flat view projects onto the two directions of greatest variance, which is principal component analysis. Both axes then share one scale on screen, so equal steps in the space are drawn as equal steps in the picture.

Related terms

More visualisations

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help