WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/lora/

1
00:00:00.072 --> 00:00:11.745
A full fine-tune can change every weight, but LoRA keeps the pretrained matrix and learns two thin factors, so this four-by-four example exposes eight trainable numbers instead of sixteen.

2
00:00:11.745 --> 00:00:26.063
The input takes two routes: the frozen weights produce four, one, one and minus one, while A compresses that same input to one, and B expands it into a correction that is added coordinate by coordinate.

3
00:00:26.063 --> 00:00:38.717
B starts at zero, so the first prediction matches the base exactly, but A needs nonzero values: setting both factors to zero makes every adapter gradient vanish and training cannot start.

4
00:00:38.717 --> 00:00:52.477
Now one squared-error training step changes B to a half, minus a half, one and zero, leaves A unchanged on this first step, and cuts the toy loss from three to zero point seven five.

5
00:00:52.477 --> 00:01:04.929
Those eight trainable numbers do not buy sixteen independent edits: each column of the update is a multiple of B, so two independent corrections cannot both fit through this rank-one path.

6
00:01:04.929 --> 00:01:17.073
Adding a second direction fits both corrections, but doubles the adapter budget, and on our tiny four-by-four matrix those sixteen factor entries already cost as much as a dense update.

7
00:01:17.073 --> 00:01:38.027
On a 4,096 by 4,096 projection, rank eight trains 65,536 parameters, one part in 256, while all 16.8 million base weights remain, so this is a trainable-parameter saving, not a matching reduction in memory.

8
00:01:38.027 --> 00:01:54.224
For a fixed adapter, merging its scaled product into the base gives the same output with one matrix multiply, while preserving the base size, so choose rank and target layers with held-out evaluations rather than assuming fewer parameters guarantee quality.