# LoRA: learn a correction, keep the base **AI Engineering Visualised** · Training · Film, 1:58 · Updated 2026-09-11 LoRA, low-rank adaptation, fine-tunes a model by freezing its weights and training two small matrices whose product is added to each weight as a correction. It trains a fraction of the parameters and the correction can be merged back in. > Follow a frozen weight matrix and two trainable factors through a forward pass, one real training step, a rank limit, an exact parameter budget and a merge. Watch the film: https://understandingdata.com/ai-engineering-visualised/lora/ Video file: https://assets.understandingdata.com/viz/lora/video/film.mp4?v=f182b86e90 LoRA adapts a pretrained model by freezing selected weight matrices and training a pair of thin factors beside each one. The same input passes through the base and the adapter, and their outputs add. The method learns a low-rank change to the weights; it does not take an SVD of the base or compress the base model itself. The film works through one four-by-four linear layer. Its hand-chosen input produces [4, 1, 1, -1] through the frozen base. A reduces that input to one coordinate, and B expands it back to four. Starting B at zero makes the initial correction zero. A is already nonzero, so the chosen training example can update B; setting both factors to zero would leave all adapter gradients at zero. Real linear-layer implementations randomize A, and their exact initialization distributions can differ. One simultaneous SGD step on a squared-error toy changes B to [0.5, -0.5, 1, 0], keeps A unchanged on that first step, and moves the prediction to [4.5, 0.5, 2, -1]. The loss falls from 3 to 0.75. Every number comes from the pure matrix model behind the film, with gradients checked against finite differences. This is a transparent arithmetic demonstration, not a language-model benchmark or a promise that later optimization steps improve. Rank limits the independent directions available to the update. A separate counterexample asks for two independent diagonal corrections: rank one cannot fit both, while rank two can. Extra rank also costs parameters. On a four-by-four matrix, a rank-two adapter already contains as many entries as a dense update. On one 4,096-by-4,096 projection, rank eight trains 65,536 parameters, compared with 16,777,216 for a dense update, while the original base weights remain. Use the calculator to explore that exact count; it is not a VRAM estimate. For a fixed adapter, the scaled product BA can be merged into the base matrix. The two routes and the merged matrix produce the same output in exact arithmetic, with possible rounding differences in floating-point or quantized implementations. The base stays the same size. Choose rank and target layers using held-out evaluations at the task you actually need. ## The maths ### A frozen path and a trainable path ```latex y = W_0 x + \frac{\alpha}{r} B(Ax), \quad A \in \mathbb{R}^{r \times d_{in}}, \quad B \in \mathbb{R}^{d_{out} \times r} ``` Only A and B are trained. The original LoRA scale is alpha/r; changing rank while holding alpha fixed does not guarantee a constant update magnitude. The toy uses alpha = r = 1. ### Rank and parameter count ```latex \operatorname{rank}(BA) \le r, \qquad P_{adapter} = r(d_{in}+d_{out}), \qquad P_{base}=d_{in}d_{out} ``` These counts apply to one selected linear layer, excluding biases. Each column of BA lies in the span of B, which explains the rank-one counterexample and the cost of adding a second direction. ### The worked training step ```latex L=\tfrac12\sum_i(y_i-t_i)^2, \qquad A^{+}=A-\eta\nabla_A L, \quad B^{+}=B-\eta\nabla_B L ``` Both gradients use the old factors. With B initially zero, the gradient for A is zero on this first step. At learning rate 0.5, the chosen toy loss moves from 3 to 0.75. ### Merge a fixed adapter ```latex W_{merged}=W_0+\frac{\alpha}{r}BA, \qquad y=W_{merged}x ``` Merging removes the additional adapter branch for a fixed linear transformation. It keeps the base matrix dimensions and does not establish the adapter’s task quality. ## Transcript ### 0:00 Freeze the base A full fine-tune can change every weight, but LoRA keeps the pretrained matrix and learns two thin factors, so this four-by-four example exposes eight trainable numbers instead of sixteen. ### 0:12 Two routes The input takes two routes: the frozen weights produce four, one, one and minus one, while A compresses that same input to one, and B expands it into a correction that is added coordinate by coordinate. ### 0:26 Start at the base B starts at zero, so the first prediction matches the base exactly, but A needs nonzero values: setting both factors to zero makes every adapter gradient vanish and training cannot start. ### 0:39 One training step Now one squared-error training step changes B to a half, minus a half, one and zero, leaves A unchanged on this first step, and cuts the toy loss from three to zero point seven five. ### 0:52 One direction Those eight trainable numbers do not buy sixteen independent edits: each column of the update is a multiple of B, so two independent corrections cannot both fit through this rank-one path. ### 1:05 Add a direction Adding a second direction fits both corrections, but doubles the adapter budget, and on our tiny four-by-four matrix those sixteen factor entries already cost as much as a dense update. ### 1:17 Price the adapter On a 4,096 by 4,096 projection, rank eight trains 65,536 parameters, one part in 256, while all 16.8 million base weights remain, so this is a trainable-parameter saving, not a matching reduction in memory. ### 1:38 Merge and evaluate For a fixed adapter, merging its scaled product into the base gives the same output with one matrix multiply, while preserving the base size, so choose rank and target layers with held-out evaluations rather than assuming fewer parameters guarantee quality. ## Sources - [Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, section 4.1](https://arxiv.org/html/2106.09685v2#S4.SS1) - [Microsoft LoRA reference implementation: linear-layer initialization and merge](https://github.com/microsoft/LoRA/blob/main/loralib/layers.py) ## Related terms - [Parameters](https://understandingdata.com/ai-coding-dictionary/parameters/) - [Training](https://understandingdata.com/ai-coding-dictionary/training/) - [Model](https://understandingdata.com/ai-coding-dictionary/model/) ## More visualisations - [Quantisation and outliers](https://understandingdata.com/ai-engineering-visualised/quantisation/) - [The loss landscape](https://understandingdata.com/ai-engineering-visualised/loss-landscape/) - [The transformer block, exploded](https://understandingdata.com/ai-engineering-visualised/transformer-block/) --- Source: https://understandingdata.com/ai-engineering-visualised/lora/ From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.