WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/what-are-embeddings/

1
00:00:00.070 --> 00:00:11.914
A model cannot do arithmetic on words, so, before it does anything else, every single token is swapped for a list of numbers, hundreds or thousands long, called an embedding.

2
00:00:11.914 --> 00:00:28.614
Take king, token 5822: that number opens row 5822 of a table holding one row per token, and the numbers in that drawer, starting minus 0.56 and minus 0.42, are all the model knows about the word.

3
00:00:28.614 --> 00:00:42.785
Training arranges these numbers so that words used in similar ways end up with similar rows: king and queen, lit in gold, share most of their pattern, while cat shares almost none.

4
00:00:42.785 --> 00:00:59.054
Squeeze every row down to three numbers and plot it as a point, and the words fall into groups: royalty together, animals together, places, money and water apart, with bank caught between the money and the water.

5
00:00:59.054 --> 00:01:14.391
To compare two words, measure the angle between their rows: king and queen sit about forty degrees apart, a similarity near 0.8, while dog swings out to ninety degrees, a similarity near zero.

6
00:01:14.391 --> 00:01:27.301
The surprising part is that meaning lives in these directions, so sums work: king minus man plus woman lands on queen, with a similarity above 0.9.

7
00:01:27.301 --> 00:01:45.859
The catch: a table row never changes, so bank beside a river and bank on a high street begin as identical numbers, one point, and only the later layers, reading the whole sentence, pull one towards the water and one towards the money.

8
00:01:45.859 --> 00:02:04.353
So the practical lesson is simple: for search, run whole passages through a model trained for retrieval and pool its outputs into one fixed-length vector an index can store, never the raw table rows, because only context can tell which bank you meant.