WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/backpropagation/

1
00:00:00.065 --> 00:00:13.664
The network's answer is off, and training has to decide how much each of its 3,281 parameters is to blame, so backpropagation runs the error backwards through the same pipes the numbers came in by.

2
00:00:13.664 --> 00:00:34.578
Take one neuron: inputs of 2.0 and 1.0, weights of 0.4 and minus 0.6 and a bias of minus 0.2 sum to exactly zero, the sigmoid turns that into 0.5, and against a target of 1 the loss is 0.125.

3
00:00:34.578 --> 00:00:53.898
Now the error runs back, multiplied at every junction by the local slope: minus 0.5 from the loss, times the sigmoid's 0.25, is minus 0.125, and times the input of 2.0, the first weight's gradient is minus 0.25.

4
00:00:53.898 --> 00:01:10.543
The catch is that 0.25 was the sigmoid at its very steepest: its slope can never pass back more than a quarter, so six sigmoid slopes in a row multiply to 0.00024 at best.

5
00:01:10.543 --> 00:01:28.444
Stack twelve sigmoid layers and measure it: six layers down, the signal is already 0.00013 of what the last layer gets, and layer one gets 18 million times less, so the first layers barely learn.

6
00:01:28.444 --> 00:01:42.445
Swap in ReLU, whose slope is exactly 1 wherever it is on, with He initialisation to match, and the light stays lit all the way down: layer one gets 1.7 times the last layer's signal.

7
00:01:42.445 --> 00:02:01.113
But the slope is only half of it: double the starting weights and the signal grows at every junction instead, reaching layer one 3,700 times stronger, and one small step sends the loss from 5.8 million to 10 to the power 97.

8
00:02:01.113 --> 00:02:17.549
So use a ReLU-style activation, scale the starting weights to the width of the layer, and add residual connections, which give the error a straight pipe round every layer: even twelve sigmoid layers then deliver 1.6 times the signal to layer one.