LoRA: learn a correction, keep the base

The maths

A frozen path and a trainable path

$$y = W_0 x + \frac{\alpha}{r} B(Ax), \quad A \in \mathbb{R}^{r \times d_{in}}, \quad B \in \mathbb{R}^{d_{out} \times r}$$

Only A and B are trained. The original LoRA scale is alpha/r; changing rank while holding alpha fixed does not guarantee a constant update magnitude. The toy uses alpha = r = 1.

Rank and parameter count

$$\operatorname{rank}(BA) \le r, \qquad P_{adapter} = r(d_{in}+d_{out}), \qquad P_{base}=d_{in}d_{out}$$

These counts apply to one selected linear layer, excluding biases. Each column of BA lies in the span of B, which explains the rank-one counterexample and the cost of adding a second direction.

The worked training step

$$L=\tfrac12\sum_i(y_i-t_i)^2, \qquad A^{+}=A-\eta\nabla_A L, \quad B^{+}=B-\eta\nabla_B L$$

Both gradients use the old factors. With B initially zero, the gradient for A is zero on this first step. At learning rate 0.5, the chosen toy loss moves from 3 to 0.75.

Merge a fixed adapter

$$W_{merged}=W_0+\frac{\alpha}{r}BA, \qquad y=W_{merged}x$$

Merging removes the additional adapter branch for a fixed linear transformation. It keeps the base matrix dimensions and does not establish the adapter’s task quality.

Transcript

The narration, chapter by chapter. A timestamp opens the film at that moment.

0:00Freeze the base

A full fine-tune can change every weight, but LoRA keeps the pretrained matrix and learns two thin factors, so this four-by-four example exposes eight trainable numbers instead of sixteen.

0:12Two routes

The input takes two routes: the frozen weights produce four, one, one and minus one, while A compresses that same input to one, and B expands it into a correction that is added coordinate by coordinate.

0:26Start at the base

B starts at zero, so the first prediction matches the base exactly, but A needs nonzero values: setting both factors to zero makes every adapter gradient vanish and training cannot start.

0:39One training step

Now one squared-error training step changes B to a half, minus a half, one and zero, leaves A unchanged on this first step, and cuts the toy loss from three to zero point seven five.

0:52One direction

Those eight trainable numbers do not buy sixteen independent edits: each column of the update is a multiple of B, so two independent corrections cannot both fit through this rank-one path.

1:05Add a direction

Adding a second direction fits both corrections, but doubles the adapter budget, and on our tiny four-by-four matrix those sixteen factor entries already cost as much as a dense update.

1:17Price the adapter

On a 4,096 by 4,096 projection, rank eight trains 65,536 parameters, one part in 256, while all 16.8 million base weights remain, so this is a trainable-parameter saving, not a matching reduction in memory.

1:38Merge and evaluate

For a fixed adapter, merging its scaled product into the base gives the same output with one matrix multiply, while preserving the base size, so choose rank and target layers with held-out evaluations rather than assuming fewer parameters guarantee quality.

Also available as captions (WebVTT), the video file and a Markdown copy of this page.

Sources and model assumptions

Follow the original mechanism behind this explainer. The interactive examples identify their toy data and simplifying assumptions above.

• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, section 4.1
• Microsoft LoRA reference implementation: linear-layer initialization and merge

Related terms

• ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
• TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
• ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.