A language model does arithmetic on numbers. It cannot do arithmetic on the letter c, so before your prompt reaches it a tokeniser cuts the text into pieces and replaces every piece with an integer, its id in a fixed vocabulary. Those pieces are tokens, and they are the only thing the model ever sees.
The machine in the film does exactly that to real sentences. The splits and the ids on the tiles are not illustrations: they come from the GPT-2 vocabulary, 50,257 pieces learned in 2019, and from the GPT-4o vocabulary of about 200,000 pieces. In ordinary English most common words are a single token, and the space in front of a word is part of the same token, which is why the tiles print "·cat" rather than "cat". The dot is the space.
Tokens are not words, though, and the gap is where the surprises are. A word the vocabulary never learned as a whole falls apart into fragments: flibbertigibbets becomes six. An emoji can fall all the way down to raw bytes. Numbers are chopped into chunks of up to three digits whether or not that respects place value, so 2026 becomes 202 and 6, which is one reason models make arithmetic slips a calculator never would.
The count also depends on which tokeniser you use. The same line of indented code is thirteen tokens for GPT-2, which spends one token on every leading space, and nine for GPT-4o, which learned a single piece for a run of spaces. The same prompt can cost a different amount, and fit differently, on two models.
That matters because tokens are the budget. The context window is a fixed number of tokens, every API prices input and output per token, and rate limits are counted in tokens per minute. The rack in the film holds 32 so the effect is visible: when it fills, the oldest tokens drop out. A real window holds hundreds of thousands, but applications that trim history keep the newest turns and cut the oldest in exactly this way, and whatever was in them is gone unless the application puts it back.
Finally, the model never sees the letters at all. It receives the row of ids, and what it returns is also an id, one token at a time, which the tokeniser turns back into text. Everything a model knows about " mat" starts from the number 2450, and the next step, turning that number into a vector the network can compute with, is its embedding.
The maths
- Tokens are what you count
Prices, rate limits and the context window are all measured in tokens, not words or characters. The tray prints the count for every sentence\; the price tag on the rack multiplies it out at an illustrative three dollars per million.
- A rough rule for English
The rule holds for ordinary English prose and breaks everywhere else. The numbers sentence is 25 characters but 11 tokens in GPT-4o, and indented code or rare words can cost two or three times what their word count suggests.
- The text is bytes first
Byte-level tokenisers work on the UTF-8 bytes, so they can encode anything, even text they have never seen. When no learned piece covers a character, it falls back to smaller pieces, down to single bytes: the giraffe emoji arrives as three tokens that are not whole characters, printed on the tiles as hex.
Related terms
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.
- Input tokensInput tokens are the tokens you send in a request: the system prompt, the conversation history, loaded files, and tool definitions. You are billed for them, and they count against the context window.
- Output tokensOutput tokens are the tokens a model generates in its response, including any hidden reasoning. They are usually priced higher than input tokens, and turning up effort produces more of them.