# Gradient accumulation: count every example **AI Engineering Visualised** · Training · Film, 1:33 · Updated 2026-09-16 Gradient accumulation adds gradients from several smaller microbatches before one optimizer step. Correct weighting divides the total gradient sum by the total number of examples, so unequal microbatch sizes do not give smaller batches extra influence. > Trace a tiny network, add gradients across microbatches of different sizes, expose the mean-of-means trap and make one correctly weighted update. Watch the film: https://understandingdata.com/ai-engineering-visualised/gradient-accumulation/ Video file: https://assets.understandingdata.com/viz/gradient-accumulation/video/film.mp4?v=cb607689bf Gradient accumulation lets several smaller microbatches contribute gradients before one optimizer update. If the intended loss is the mean over examples, each example must receive the same weight. Summing per-example gradients and dividing by the total number of examples preserves that objective even when the microbatches have different sizes. This film uses a scalar linear network with two weights. The input is multiplied by w1 = 2 and then w2 = -1. For input 1 and target 0, the prediction is -2 and half the squared error is 2. The chain rule gives gradients (2, -4). Two more examples, with input and target pairs (2, -3) and (-1, 1), give gradients (2, -4) and (1, -2), evaluated at the same starting weights. Put the first example in a microbatch of one and the other two in a microbatch of two. Their gradient sums are (2, -4) and (3, -6). The total is (5, -10), so the correct mean is (5/3, -10/3). Averaging the two microbatch means equally would instead give (1.75, -3.5): the lone example would get half of the influence, while each other example gets a quarter. Regrouping the same data as two examples followed by one changes that wrong result to (1.5, -3). The explorer changes the split and normalization rule while keeping the examples and starting weights fixed. Correct example weighting produces the same gradient and update for every split. Equal microbatch means coincide with the correct answer when the microbatches have equal sizes, which can hide this bug in regular batches and reveal it in a smaller final batch. At learning rate 0.1, one plain SGD update changes the weights to (11/6, -2/3), about (1.833, -0.667). The underlying calculation retains full floating-point precision. This is an arithmetic demonstration with independent examples, a fixed parameter snapshot and no stateful or stochastic layers. Batch-dependent normalization, random operations, mixed-precision scaling and distributed reductions need their own treatment; accumulating several optimizer steps is a different procedure. ## The maths ### A small forward and backward pass ```latex h=w_1x,\quad y=w_2h,\quad L=\tfrac12(y-t)^2,\quad \frac{\partial L}{\partial w_1}=(y-t)w_2x,\quad \frac{\partial L}{\partial w_2}=(y-t)h ``` Both derivatives use the same old weights. The film traces every factor on the first example. ### Weight examples equally ```latex \bar g=\frac{\sum_b\sum_{i\in B_b}g_i}{\sum_b |B_b|}=\frac{\sum_b |B_b|\bar g_b}{N} ``` If each microbatch already reports a mean, multiply that mean by its example count before the final division by N. ### Why a mean of means can be wrong ```latex \tilde g=\frac1M\sum_{b=1}^{M}\bar g_b,\qquad \text{example weight in batch }b=\frac{1}{M|B_b|} ``` With microbatch sizes one and two, the weights become one half, one quarter and one quarter instead of one third each. ### One optimizer step ```latex w^+=w-\eta\bar g,\qquad (2,-1)-0.1(5/3,-10/3)=(11/6,-2/3) ``` The toy uses plain SGD. Keep the parameter snapshot fixed while accumulating this effective batch, then update once. ## Transcript ### 0:00 Every example counts Three examples can arrive as one microbatch and then two. If every example should count equally, the split must not decide how far we move the weights. Let us test that. ### 0:11 A tiny forward pass Our tiny network multiplies an input by two, then by minus one. With input one and target zero, its prediction is minus two, and half the squared error is two. ### 0:22 Trace the gradient Backpropagation carries the error through both multiplies. This example contributes gradients two and minus four. Compute the other examples at those same weights, so their gradients can be added before any update. ### 0:35 Add microbatch sums The first microbatch has one example. The second has two, whose gradient sum is three and minus six. Across all three examples, the accumulated sum is five and minus ten. ### 0:47 The mean-of-means trap Here is the trap: averaging the two microbatch means gives the single example half the influence. It should receive one third. Changing the grouping can now change the update, even though the data is identical. ### 1:01 Divide by examples Instead, divide the total gradient sum by three examples. The correct mean is five thirds and minus ten thirds. With learning rate zero point one, make exactly one optimizer step after that division. ### 1:16 One update, same result The new weights are about one point eight three three and minus zero point six six seven. Try another split below: the correct result stays fixed. This scalar example uses plain gradient descent and excludes stateful batch layers. ## Sources - [PyTorch: gradient accumulation and effective-batch optimizer steps](https://docs.pytorch.org/docs/2.14/notes/amp_examples.html#gradient-accumulation) - [PyTorch: automatic differentiation and the chain rule](https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html) ## Related terms - [Training](https://understandingdata.com/ai-coding-dictionary/training/) - [Parameters](https://understandingdata.com/ai-coding-dictionary/parameters/) - [Model](https://understandingdata.com/ai-coding-dictionary/model/) ## More visualisations - [LoRA: learn a correction, keep the base](https://understandingdata.com/ai-engineering-visualised/lora/) - [The loss landscape](https://understandingdata.com/ai-engineering-visualised/loss-landscape/) - [The transformer block, exploded](https://understandingdata.com/ai-engineering-visualised/transformer-block/) --- Source: https://understandingdata.com/ai-engineering-visualised/gradient-accumulation/ From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-16.