Byte pair encoding

A model never sees your text. It sees token ids, and the thing that decides which ids is a tokeniser that was trained before the model was. Understanding it is unusually practical, because tokens are the unit you are billed in, the unit the context window is measured in, and the reason some perfectly ordinary strings behave strangely.

Byte pair encoding starts with individual characters and repeatedly merges the most frequent adjacent pair, recording each merge as a rule. Training produces an ordered list of those rules. Encoding replays them: find the highest-ranked rule that matches anywhere in the sequence, apply it, repeat until nothing matches. The brackets in the diagram are those merges, stacked in the order they fired, so you can read the history of how each token was assembled.

The greedy-by-rank detail matters. The algorithm does not scan left to right, it always applies the best available rule wherever it is, which is why the result depends on the whole merge table rather than on local context. It is also why token boundaries do not line up with what you think of as words: a boundary can fall mid-word, and the same substring can be split differently depending on what surrounds it. The mapping itself is exactly reversible, so decoding recovers your text precisely. It is the segmentation that is unintuitive, not the encoding.

Drag the vocabulary slider down. Rules disappear from the end of the table, common sequences stop collapsing, and the token count rises for identical text. That is the entire tradeoff a tokeniser makes: a larger vocabulary means fewer tokens per document and a larger embedding matrix to pay for it.

Try the rare word and the numbers samples. A long uncommon word fragments into pieces, which is why rare terms cost several times more than common ones of the same length. Numbers are worth watching too, though the story there has moved on: older tokenisers merged digits into arbitrary multi-digit chunks, so "1024" and "1025" could have completely unrelated segmentations, and recent models deliberately split numbers into fixed groups of digits instead. Consistent grouping was the fix, not fewer pieces.

One honest caveat. The algorithm here is the real one, but the merge table is forty-one hand-written rules rather than a trained vocabulary of a hundred thousand. That is why the characters-per-token figure comes out around two, where a production tokeniser averages closer to four on English prose. The behaviour is faithful; the compression is not.