Fine-tuning a large model the obvious way means updating every weight, which means storing an optimiser state as big as the model and shipping a full copy per task. LoRA's bet is that the UPDATE a fine-tune needs is approximately low rank: not that the weights are simple, but that the change to them is.
This page makes rank tangible. The left panel is a weight matrix with structure in it. The middle panel rebuilds it from only the top r singular directions, each of which is one column vector times one row vector: the cheapest possible matrix. The right panel is what the rebuild misses. Drag the slider from zero and watch the order in which reality arrives: the broad strokes in the first two or three ranks, refinements over the next few, and then a long tail that only sharpens noise.
The spectrum underneath is why this works when it works. The singular values fall off steeply and then flatten into a knee; everything left of the knee is structure and everything right of it is texture. A rank at the knee captures nearly all the energy for a small fraction of the parameters, and the counter above the slider prices it: storing two thin matrices costs 2nr numbers against n squared for the full update.
Honest limits. This matrix was built with planted low-rank structure plus noise, so the knee is clean; real weight updates have messier spectra, and how low-rank they truly are varies by task and layer, which is why LoRA rank is a tuning knob rather than a constant. And LoRA does not compress the base model, only the update: the pretrained weights stay full rank underneath, frozen, with the thin correction added on top.
The maths
- The LoRA update
The pretrained W is frozen; only the thin factors A and B are trained. Storage per adapted layer falls from n squared to 2nr, which is the counter above the slider. In practice the update is scaled by alpha over r so changing rank does not change its magnitude, and B starts at zero so the adapted model begins identical to the base.
- Singular value decomposition
Any matrix is a sum of rank-one pieces, ordered by their singular values. The middle heatmap is this sum truncated at r terms, computed by an actual Jacobi SVD in the page.
- Why truncating there is optimal
The Eckart-Young theorem: no rank-r matrix beats the truncated SVD, and the error is exactly the energy in the discarded tail. The "energy captured" stat is one minus that discarded tail as a fraction of the total.
Related terms
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.