What are embeddings?

A language model never sees a word. The text is first cut into tokens, each token is replaced by a number from the vocabulary, and each number is then replaced by a list of numbers called an embedding. Everything the model does after that, every attention head and every layer, is arithmetic on those lists. This film follows one token through that last swap and shows what the numbers mean.

The embedding table is the cabinet. It has one row for every token the model knows, about fifty thousand for GPT-2 and well over a hundred thousand for recent models, and each row holds the same count of numbers. The token id is simply the row number, so the lookup is a drawer sliding out. Real models store hundreds or thousands of numbers per row; the slider lets you see eight, sixteen, twenty-four or thirty-two of them, and the film prints the values when there are few enough to read.

Nobody writes those numbers by hand. They start random and training nudges them, billions of times, towards whatever helps the model predict the next token. Words that appear in similar places end up with similar rows, because the model gains nothing by telling them apart. That is why king and queen share most of their pattern on the board, lit in gold, while an animal or a city looks nothing like either.

Plot every row as a point and those shared patterns become places. The scatter beside the bench squeezes each row down to three numbers, keeping the three directions along which the groups differ most, and the words fall into groups: royalty, animals, places, money and water. A picture like this is a map of the groups, not a ruler. Squeezing throws most of each row away, so two points that look close may not be, which is why similarity is always measured on the full rows. The scatter also draws each group spread to the same size so every label reads; where the groups sit relative to each other, and where bank sits between money and water, is the projection's.

Similarity has a precise meaning here. Each row is an arrow in a space with as many directions as there are numbers, and two words are alike when their arrows point the same way. The cosine of the angle between them is the standard measure: close to one for close relatives, close to zero for words with nothing in common, and negative when the arrows point apart. The dial draws the true angle, because it is drawn in the flat plane the two rows share.

Because meaning is stored as direction, you can do arithmetic with it. Take the row for king, subtract the row for man, add the row for woman, and the result lands nearer to queen than to any other word. The famous demonstration came from word2vec in 2013; the same kind of structure shows up in the input embeddings of large language models, though it is messier there than in a tidy example.

There is one thing the table cannot do. The row for bank is the same whether the sentence is about a river or about money, because a lookup has no idea what surrounds it. Telling the two apart is the job of the layers that come next, which read the whole sentence and blend neighbouring words into each position. The film shows the direction of that effect with a simple blend, first on the board and then in the scatter, where the two banks drift apart, one towards the water and one towards the money; in a real model it takes many layers of attention to get there.

The honest caveat: the vectors on this page are hand-built so the effects are clean. Each word is written as a mix of a few concepts and then rotated into unlabelled numbers, which is roughly how learned embeddings behave but far tidier. Real rows are longer, noisier, and no single number means anything on its own; meaning lives in directions spread across all of them. The token ids are illustrative GPT-2 style values.