# Backpropagation and vanishing gradients

**AI Engineering Visualised** · Training · Film, 2:21 · Updated 2026-09-11

Backpropagation is the algorithm that computes how much each weight contributed to the error, by applying the chain rule backwards through the network one layer at a time. Vanishing gradients happen when those per-layer factors multiply to almost nothing.

> Press play and watch backpropagation run the error back through one neuron by the chain rule, then through twelve layers, where the sigmoid lets it fade to 18 million times weaker and ReLU, careful initialisation and residual connections keep it alive.

Watch the film: https://understandingdata.com/ai-engineering-visualised/backpropagation/
Video file: https://assets.understandingdata.com/viz/backpropagation/video/film.mp4?v=1db12ff036

A network makes a prediction, compares it with the answer, and gets a single number back: the loss. Training then has to share that one number out among every weight and bias in the network, deciding how much each one contributed and which way to nudge it. Backpropagation is how it decides. It runs the error backwards through the same computation that produced the prediction, and at every junction multiplies it by that junction's local slope. Those products are the gradients, and gradient descent moves each parameter a small step against its own.

The first half of the film traces one neuron you can check by hand. Inputs of 2.0 and 1.0, weights of 0.4 and minus 0.6 and a bias of minus 0.2 add up to exactly zero; the sigmoid turns zero into 0.5; against a target of 1, a squared-error loss with the usual half in front comes to 0.125. On the way back the loss contributes minus 0.5, the sigmoid multiplies that by its slope of 0.25 to give minus 0.125, and the weighted sum hands each weight that amount times its own input: minus 0.25 for the first weight, minus 0.125 for the second and for the bias. Nothing else is going on in backpropagation. A framework such as PyTorch builds this chain for every operation automatically, which is what automatic differentiation and a computational graph mean, but each link is exactly this multiplication.

The neuron was chosen to sit at the sigmoid's steepest point, and that is the uncomfortable part. The sigmoid's slope is its output times one minus its output, which peaks at 0.25 when the output is a half and is smaller everywhere else. Every sigmoid layer the error passes through therefore shrinks it by at least a factor of four from the slope alone. Six in a row leave 0.00024 at best.

The second half measures it. The run is a real network, twelve layers of sixteen units with 3,281 parameters, initialised from a fixed seed and fed a batch of 32 random examples, with a forward and a backward pass written out by hand. The gauges show the size of the error signal reaching each layer, divided by what reaches the last one. With sigmoid activations and standard initialisation it is 0.00013 six layers down and about 18 million times smaller at layer one, so the first layers' weights get almost no gradient and barely learn. That is the vanishing gradient problem, and it is why deep networks were close to untrainable before 2010. Tanh, whose slope reaches 1, fades by about a factor of five over the same twelve layers.

The fixes are what every modern network is built from. ReLU passes a slope of exactly 1 wherever it is switched on, and He initialisation scales the starting weights to the width of the layer to make up for the half of the units it switches off: the same run then delivers 1.7 times the last layer's signal to layer one. Initialisation matters as much as the activation. Double the starting weights and the signal grows at every junction instead, arriving at layer one 3,700 times stronger, and a single gradient step at a learning rate of 0.01 sends the loss from 5.8 million to about 10 to the 97, which is the exploding gradient problem. Residual connections are the third fix: adding each layer's input to its output gives the error a path that skips the layer, and even twelve sigmoid layers then deliver 1.6 times the signal to layer one. Every transformer block is built around two of them.

The simplifications, honestly. The network is sixteen units wide rather than thousands, and the numbers come from one seed and one batch, so the exact figures would move a little with another draw, although the orders of magnitude would not. The gauges show the error signal, the backpropagated delta, rather than the weight gradients themselves; the two tell the same story here (the weight gradient ratio for the sigmoid run is also below one in ten million), but in the exploding run the weight gradients are huge at every layer rather than growing towards layer one, because the forward activations blow up too. The residual run shows only that the signal gets through: without the normalisation real residual networks pair with it, its activations grow with depth and its loss starts high, so it is not something you would train as it stands.

## The maths

### The chain rule, one neuron

```latex
\frac{\partial L}{\partial w_1} \;=\; \frac{\partial L}{\partial a}\cdot\frac{\partial a}{\partial z}\cdot\frac{\partial z}{\partial w_1} \;=\; (a - y)\cdot a(1-a)\cdot x_1 \;=\; (-0.5)(0.25)(2.0) = -0.25
```

The traced neuron, junction by junction. Each factor is a local slope printed at one junction of the pipes: the loss against the output, the sigmoid against its input, the weighted sum against the weight. Backpropagation is nothing more than multiplying them in order, from the loss back.

### The sigmoid never passes back more than a quarter

```latex
\sigma'(z) \;=\; \sigma(z)\,\big(1-\sigma(z)\big) \;\le\; \tfrac14, \qquad \big(\tfrac14\big)^6 \approx 0.00024
```

The slope is largest at z = 0, where the output is 0.5 and the slope exactly 0.25, which is where the traced neuron happens to sit. Anywhere else it is smaller. Six sigmoid slopes in a row can therefore shrink the signal to 0.00024 at best, before the weights have had any say.

### The signal one layer down

```latex
\delta^{(l)} \;=\; f'\!\big(z^{(l)}\big) \,\odot\, \big(W^{(l+1)}\big)^{\!\top} \delta^{(l+1)}, \qquad \frac{\partial L}{\partial W^{(l)}} = \delta^{(l)}\, \big(h^{(l-1)}\big)^{\!\top}
```

The light in the run. Each layer multiplies the error arriving from above by its weights and by the activation slope at every unit. The gauges show the size of delta at each layer, relative to the last layer, for sixteen units per layer and a batch of 32 seeded examples; the junction labels are the measured multiplier from one layer to the next.

### Initialisation and the residual pipe

```latex
W_{ij} \sim \mathcal{N}\!\Big(0, \tfrac{1}{n}\Big)\ \text{(LeCun)}, \quad \mathcal{N}\!\Big(0, \tfrac{2}{n}\Big)\ \text{(He)}, \qquad h^{(l)} = h^{(l-1)} + f\big(W^{(l)} h^{(l-1)}\big) \;\Rightarrow\; \frac{\partial h^{(l)}}{\partial h^{(l-1)}} = I + \cdots
```

He initialisation doubles the variance to make up for ReLU switching half its units off, so the signal neither shrinks nor grows on average. Doubling the standard deviation again multiplies it by roughly two per layer, which is the exploding run. A residual connection adds the identity to every layer's Jacobian: the straight pipe round each station.

## Transcript

### 0:00 Who is to blame

The network's answer is off, and training has to decide how much each of its 3,281 parameters is to blame, so backpropagation runs the error backwards through the same pipes the numbers came in by.

### 0:14 One neuron, forward

Take one neuron: inputs of 2.0 and 1.0, weights of 0.4 and minus 0.6 and a bias of minus 0.2 sum to exactly zero, the sigmoid turns that into 0.5, and against a target of 1 the loss is 0.125.

### 0:35 The chain rule

Now the error runs back, multiplied at every junction by the local slope: minus 0.5 from the loss, times the sigmoid's 0.25, is minus 0.125, and times the input of 2.0, the first weight's gradient is minus 0.25.

### 0:54 At most a quarter

The catch is that 0.25 was the sigmoid at its very steepest: its slope can never pass back more than a quarter, so six sigmoid slopes in a row multiply to 0.00024 at best.

### 1:10 Vanishing

Stack twelve sigmoid layers and measure it: six layers down, the signal is already 0.00013 of what the last layer gets, and layer one gets 18 million times less, so the first layers barely learn.

### 1:28 ReLU and He init

Swap in ReLU, whose slope is exactly 1 wherever it is on, with He initialisation to match, and the light stays lit all the way down: layer one gets 1.7 times the last layer's signal.

### 1:42 Exploding

But the slope is only half of it: double the starting weights and the signal grows at every junction instead, reaching layer one 3,700 times stronger, and one small step sends the loss from 5.8 million to 10 to the power 97.

### 2:01 What keeps it alive

So use a ReLU-style activation, scale the starting weights to the width of the layer, and add residual connections, which give the error a straight pipe round every layer: even twelve sigmoid layers then deliver 1.6 times the signal to layer one.

## Related terms

- [Training](https://understandingdata.com/ai-coding-dictionary/training/)
- [Parameters](https://understandingdata.com/ai-coding-dictionary/parameters/)
- [Model](https://understandingdata.com/ai-coding-dictionary/model/)

## More visualisations

- [Activation functions and normalisation](https://understandingdata.com/ai-engineering-visualised/activations/)
- [The loss landscape](https://understandingdata.com/ai-engineering-visualised/loss-landscape/)
- [The optimiser race](https://understandingdata.com/ai-engineering-visualised/optimiser-race/)
- [The transformer block, exploded](https://understandingdata.com/ai-engineering-visualised/transformer-block/)

---

Source: https://understandingdata.com/ai-engineering-visualised/backpropagation/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.