WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/gradient-accumulation/

1
00:00:00.127 --> 00:00:04.339
Three examples can arrive as one microbatch and then two.

2
00:00:04.339 --> 00:00:09.659
If every example should count equally, the split must not decide how far we move the weights.

3
00:00:09.659 --> 00:00:10.986
Let us test that.

4
00:00:11.039 --> 00:00:15.420
Our tiny network multiplies an input by two, then by minus one.

5
00:00:15.420 --> 00:00:22.013
With input one and target zero, its prediction is minus two, and half the squared error is two.

6
00:00:22.020 --> 00:00:25.420
Backpropagation carries the error through both multiplies.

7
00:00:25.420 --> 00:00:29.059
This example contributes gradients two and minus four.

8
00:00:29.059 --> 00:00:34.780
Compute the other examples at those same weights, so their gradients can be added before any update.

9
00:00:34.939 --> 00:00:37.399
The first microbatch has one example.

10
00:00:37.399 --> 00:00:42.159
The second has two, whose gradient sum is three and minus six.

11
00:00:42.159 --> 00:00:47.271
Across all three examples, the accumulated sum is five and minus ten.

12
00:00:47.399 --> 00:00:54.199
Here is the trap: averaging the two microbatch means gives the single example half the influence.

13
00:00:54.199 --> 00:00:56.439
It should receive one third.

14
00:00:56.439 --> 00:01:01.596
Changing the grouping can now change the update, even though the data is identical.

15
00:01:01.596 --> 00:01:05.879
Instead, divide the total gradient sum by three examples.

16
00:01:05.879 --> 00:01:09.900
The correct mean is five thirds and minus ten thirds.

17
00:01:09.900 --> 00:01:15.634
With learning rate zero point one, make exactly one optimizer step after that division.

18
00:01:15.739 --> 00:01:21.500
The new weights are about one point eight three three and minus zero point six six seven.

19
00:01:21.500 --> 00:01:25.479
Try another split below: the correct result stays fixed.

20
00:01:25.479 --> 00:01:30.581
This scalar example uses plain gradient descent and excludes stateful batch layers.