Backpropagation and vanishing gradients

The maths

The chain rule, one neuron

$$\frac{\partial L}{\partial w_1} \;=\; \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial w_1} \;=\; (a - y)\cdot a(1-a)\cdot x_1 \;=\; (-0.5)(0.25)(2.0) = -0.25$$

The traced neuron, junction by junction. Each factor is a local slope printed at one junction of the pipes: the loss against the output, the sigmoid against its input, the weighted sum against the weight. Backpropagation is nothing more than multiplying them in order, from the loss back.

The sigmoid never passes back more than a quarter

$$\sigma'(z) \;=\; \sigma(z)\,\big(1-\sigma(z)\big) \;\le\; \tfrac14, \qquad \big(\tfrac14\big)^6 \approx 0.00024$$

The slope is largest at z = 0, where the output is 0.5 and the slope exactly 0.25, which is where the traced neuron happens to sit. Anywhere else it is smaller. Six sigmoid slopes in a row can therefore shrink the signal to 0.00024 at best, before the weights have had any say.

The signal one layer down

$$\delta^{(l)} \;=\; f'\!\big(z^{(l)}\big) \,\odot\, \big(W^{(l+1)}\big)^{\!\top} \delta^{(l+1)}, \qquad \frac{\partial L}{\partial W^{(l)}} = \delta^{(l)}\, \big(h^{(l-1)}\big)^{\!\top}$$

The light in the run. Each layer multiplies the error arriving from above by its weights and by the activation slope at every unit. The gauges show the size of delta at each layer, relative to the last layer, for sixteen units per layer and a batch of 32 seeded examples; the junction labels are the measured multiplier from one layer to the next.

Initialisation and the residual pipe

$$W_{ij} \sim \mathcal{N}\!\Big(0, \tfrac{1}{n}\Big)\ \text{(LeCun)}, \quad \mathcal{N}\!\Big(0, \tfrac{2}{n}\Big)\ \text{(He)}, \qquad h^{(l)} = h^{(l-1)} + f\big(W^{(l)} h^{(l-1)}\big) \;\Rightarrow\; \frac{\partial h^{(l)}}{\partial h^{(l-1)}} = I + \cdots$$

He initialisation doubles the variance to make up for ReLU switching half its units off, so the signal neither shrinks nor grows on average. Doubling the standard deviation again multiplies it by roughly two per layer, which is the exploding run. A residual connection adds the identity to every layer's Jacobian: the straight pipe round each station.

Transcript

The narration, chapter by chapter. A timestamp opens the film at that moment.

0:00Who is to blame

The network's answer is off, and training has to decide how much each of its 3,281 parameters is to blame, so backpropagation runs the error backwards through the same pipes the numbers came in by.

0:14One neuron, forward

Take one neuron: inputs of 2.0 and 1.0, weights of 0.4 and minus 0.6 and a bias of minus 0.2 sum to exactly zero, the sigmoid turns that into 0.5, and against a target of 1 the loss is 0.125.

0:35The chain rule

Now the error runs back, multiplied at every junction by the local slope: minus 0.5 from the loss, times the sigmoid's 0.25, is minus 0.125, and times the input of 2.0, the first weight's gradient is minus 0.25.

0:54At most a quarter

The catch is that 0.25 was the sigmoid at its very steepest: its slope can never pass back more than a quarter, so six sigmoid slopes in a row multiply to 0.00024 at best.

1:10Vanishing

Stack twelve sigmoid layers and measure it: six layers down, the signal is already 0.00013 of what the last layer gets, and layer one gets 18 million times less, so the first layers barely learn.

1:28ReLU and He init

Swap in ReLU, whose slope is exactly 1 wherever it is on, with He initialisation to match, and the light stays lit all the way down: layer one gets 1.7 times the last layer's signal.

1:42Exploding

But the slope is only half of it: double the starting weights and the signal grows at every junction instead, reaching layer one 3,700 times stronger, and one small step sends the loss from 5.8 million to 10 to the power 97.

2:01What keeps it alive

So use a ReLU-style activation, scale the starting weights to the width of the layer, and add residual connections, which give the error a straight pipe round every layer: even twelve sigmoid layers then deliver 1.6 times the signal to layer one.

Also available as captions (WebVTT), the video file and a Markdown copy of this page.

Related terms

• TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
• ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
• ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.