WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/what-are-tokens/

1
00:00:00.070 --> 00:00:10.834
A language model cannot read letters, only numbers, so a tokeniser first cuts your text into pieces called tokens and swaps each piece for an id.

2
00:00:10.834 --> 00:00:27.066
Trace one sentence: The cat sat on the mat becomes seven tokens, with ids like 976 for The and 9059 for cat, because each space rides on the front of the word after it.

3
00:00:27.066 --> 00:00:40.101
The part people miss is that tokens are not words, so a rare word like flibbertigibbets shatters into six fragments and the giraffe emoji is cut into raw bytes.

4
00:00:40.101 --> 00:00:52.752
Numbers fare worse: the price is chopped into chunks of three digits and even twenty twenty-six is split after its first three digits, one reason models slip on arithmetic.

5
00:00:52.752 --> 00:01:07.360
Token counts hinge on the vocabulary: this code is thirteen tokens in GPT-2 but only nine in GPT-4o, whose bigger vocabulary trades a larger embedding table for shorter sequences.

6
00:01:07.360 --> 00:01:20.010
That count is your budget, since you pay and wait for every token, and the failure mode is a full window: once it fills, the oldest tokens silently fall out.

7
00:01:20.010 --> 00:01:31.204
So budget in tokens, not words, because the model only sees this row of ids and answers with one id, which is where embeddings take over.