A language model does arithmetic on numbers. It cannot do arithmetic on the letter c, so before your prompt reaches it a tokeniser cuts the text into pieces and replaces every piece with an integer, its id in a fixed vocabulary. Those pieces are tokens, and they are the only thing the model ever sees.
The machine in the film does exactly that to real sentences. The splits and the ids on the tiles are not illustrations: they come from the GPT-2 vocabulary, 50,257 pieces learned in 2019, and from the GPT-4o vocabulary of about 200,000 pieces. In ordinary English most common words are a single token, and the space in front of a word is part of the same token, which is why the tiles print "·cat" rather than "cat". The dot is the space.
Tokens are not words, though, and the gap is where the surprises are. A word the vocabulary never learned as a whole falls apart into fragments: flibbertigibbets becomes six. An emoji can fall all the way down to raw bytes. Numbers are chopped into chunks of up to three digits whether or not that respects place value, so 2026 becomes 202 and 6, which is one reason models make arithmetic slips a calculator never would.
The count also depends on which tokeniser you use. The same line of indented code is thirteen tokens for GPT-2, which spends one token on every leading space, and nine for GPT-4o, which learned a single piece for a run of spaces. The same prompt can cost a different amount, and fit differently, on two models.
That matters because tokens are the budget. The context window is a fixed number of tokens, every API prices input and output per token, and rate limits are counted in tokens per minute. The rack in the film holds 32 so the effect is visible: when it fills, the oldest tokens drop out. A real window holds hundreds of thousands, but applications that trim history keep the newest turns and cut the oldest in exactly this way, and whatever was in them is gone unless the application puts it back.
Finally, the model never sees the letters at all. It receives the row of ids, and what it returns is also an id, one token at a time, which the tokeniser turns back into text. Everything a model knows about " mat" starts from the number 2450, and the next step, turning that number into a vector the network can compute with, is its embedding.
The maths
- Tokens are what you count
Prices, rate limits and the context window are all measured in tokens, not words or characters. The tray prints the count for every sentence\; the price tag on the rack multiplies it out at an illustrative three dollars per million.
- A rough rule for English
The rule holds for ordinary English prose and breaks everywhere else. The numbers sentence is 25 characters but 11 tokens in GPT-4o, and indented code or rare words can cost two or three times what their word count suggests.
- The text is bytes first
Byte-level tokenisers work on the UTF-8 bytes, so they can encode anything, even text they have never seen. When no learned piece covers a character, it falls back to smaller pieces, down to single bytes: the giraffe emoji arrives as three tokens that are not whole characters, printed on the tiles as hex.
Transcript
The narration, chapter by chapter. A timestamp opens the film at that moment.
- 0:00Tokens
- A language model cannot read letters, only numbers, so a tokeniser first cuts your text into pieces called tokens and swaps each piece for an id.
- 0:11Words
- Trace one sentence: The cat sat on the mat becomes seven tokens, with ids like 976 for The and 9059 for cat, because each space rides on the front of the word after it.
- 0:27Rare words
- The part people miss is that tokens are not words, so a rare word like flibbertigibbets shatters into six fragments and the giraffe emoji is cut into raw bytes.
- 0:40Numbers
- Numbers fare worse: the price is chopped into chunks of three digits and even twenty twenty-six is split after its first three digits, one reason models slip on arithmetic.
- 0:53Two tokenisers
- Token counts hinge on the vocabulary: this code is thirteen tokens in GPT-2 but only nine in GPT-4o, whose bigger vocabulary trades a larger embedding table for shorter sequences.
- 1:07Context window
- That count is your budget, since you pay and wait for every token, and the failure mode is a full window: once it fills, the oldest tokens silently fall out.
- 1:20Ids in, ids out
- So budget in tokens, not words, because the model only sees this row of ids and answers with one id, which is where embeddings take over.
Also available as captions (WebVTT), the video file and a Markdown copy of this page.
Related terms
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.
- Input tokensInput tokens are the tokens you send in a request: the system prompt, the conversation history, loaded files, and tool definitions. You are billed for them, and they count against the context window.
- Output tokensOutput tokens are the tokens a model generates in its response, including any hidden reasoning. They are usually priced higher than input tokens, and turning up effort produces more of them.
More visualisations
Embed this film
Paste this into a post, a course page or a newsletter. The film plays in place, with the same controls as here, and credits the source.