Backpropagation and vanishing gradients

A network makes a prediction, compares it with the answer, and gets a single number back: the loss. Training then has to share that one number out among every weight and bias in the network, deciding how much each one contributed and which way to nudge it. Backpropagation is how it decides. It runs the error backwards through the same computation that produced the prediction, and at every junction multiplies it by that junction's local slope. Those products are the gradients, and gradient descent moves each parameter a small step against its own.

The first half of the film traces one neuron you can check by hand. Inputs of 2.0 and 1.0, weights of 0.4 and minus 0.6 and a bias of minus 0.2 add up to exactly zero; the sigmoid turns zero into 0.5; against a target of 1, a squared-error loss with the usual half in front comes to 0.125. On the way back the loss contributes minus 0.5, the sigmoid multiplies that by its slope of 0.25 to give minus 0.125, and the weighted sum hands each weight that amount times its own input: minus 0.25 for the first weight, minus 0.125 for the second and for the bias. Nothing else is going on in backpropagation. A framework such as PyTorch builds this chain for every operation automatically, which is what automatic differentiation and a computational graph mean, but each link is exactly this multiplication.

The neuron was chosen to sit at the sigmoid's steepest point, and that is the uncomfortable part. The sigmoid's slope is its output times one minus its output, which peaks at 0.25 when the output is a half and is smaller everywhere else. Every sigmoid layer the error passes through therefore shrinks it by at least a factor of four from the slope alone. Six in a row leave 0.00024 at best.

The second half measures it. The run is a real network, twelve layers of sixteen units with 3,281 parameters, initialised from a fixed seed and fed a batch of 32 random examples, with a forward and a backward pass written out by hand. The gauges show the size of the error signal reaching each layer, divided by what reaches the last one. With sigmoid activations and standard initialisation it is 0.00013 six layers down and about 18 million times smaller at layer one, so the first layers' weights get almost no gradient and barely learn. That is the vanishing gradient problem, and it is why deep networks were close to untrainable before 2010. Tanh, whose slope reaches 1, fades by about a factor of five over the same twelve layers.

The fixes are what every modern network is built from. ReLU passes a slope of exactly 1 wherever it is switched on, and He initialisation scales the starting weights to the width of the layer to make up for the half of the units it switches off: the same run then delivers 1.7 times the last layer's signal to layer one. Initialisation matters as much as the activation. Double the starting weights and the signal grows at every junction instead, arriving at layer one 3,700 times stronger, and a single gradient step at a learning rate of 0.01 sends the loss from 5.8 million to about 10 to the 97, which is the exploding gradient problem. Residual connections are the third fix: adding each layer's input to its output gives the error a path that skips the layer, and even twelve sigmoid layers then deliver 1.6 times the signal to layer one. Every transformer block is built around two of them.

The simplifications, honestly. The network is sixteen units wide rather than thousands, and the numbers come from one seed and one batch, so the exact figures would move a little with another draw, although the orders of magnitude would not. The gauges show the error signal, the backpropagated delta, rather than the weight gradients themselves; the two tell the same story here (the weight gradient ratio for the sigmoid run is also below one in ten million), but in the exploding run the weight gradients are huge at every layer rather than growing towards layer one, because the forward activations blow up too. The residual run shows only that the signal gets through: without the normalisation real residual networks pair with it, its activations grow with depth and its loss starts high, so it is not something you would train as it stands.