WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/activations/

1
00:00:00.267 --> 00:00:13.646
Stack linear layers with nothing between them and they collapse into one linear map, so every block needs a nonlinearity: here are sixty-four raw inputs, from minus 2.3 to plus 5.1.

2
00:00:13.646 --> 00:00:28.434
ReLU keeps the positives and zeroes the rest, so plus 0.86 passes straight through while minus 0.26 becomes zero, and 23 of these 64 units go dark with no gradient.

3
00:00:28.434 --> 00:00:43.854
The part people miss is that GELU is not just a smoother ReLU: minus 0.26 still comes out at minus 0.1, and its gradient is 0.3 rather than zero, so the unit keeps learning.

4
00:00:43.854 --> 00:00:54.842
SwiGLU adds a third matrix as a valve on each unit, and models pay for it by shrinking the hidden layer by a third, because the learned gate is worth more.

5
00:00:54.842 --> 00:01:06.685
Then layer norm pulls the outputs back to mean zero and variance one, here from 0.89 and 1.26, which stops the scale drifting as layers stack up.

6
00:01:06.685 --> 00:01:16.155
Dropout guards against units that only work in pairs: training switches off 16 of the 64 at random and scales the rest up by a quarter.

7
00:01:16.155 --> 00:01:25.706
The takeaway is the residual stream: each block only adds its output to what came in, so a hundred blocks in a row still train.