A neural network without activation functions is a single matrix multiplication wearing a trench coat: stack as many linear layers as you like and the result is still linear. The activation is the one non-linear step between them, and its shape decides two things at once: what the layer can represent, and how much gradient makes it back through during training. This page treats each activation as a physical surface and rolls a ball along it, because the slope under the ball is the gradient, and the gradient is the whole story.
ReLU is the hard ramp. Positive inputs pass unchanged, negative inputs become exactly zero, and the crease at the origin is where the trouble lives. A unit whose input is negative has a slope of zero, so it receives no gradient, so it cannot learn its way back to positive. Watch the dead fraction as you move the ball: on this sample about a third of the units are dead, which is normal, and a unit that stays there for the whole of training is a parameter you paid for and never used.
GELU and SiLU are the same ramp sanded smooth. Slightly negative inputs leak through as small negative outputs, and the slope never reaches exactly zero, so no unit is ever entirely dead. GELU is what BERT and GPT-2 used, SiLU is nearly the same shape and slightly cheaper to compute, and both replaced ReLU in transformers for the same reason: the gradient survives the crease. The older sigmoid and tanh are there for contrast. They squash everything into a bounded range, and at either end the slope collapses, which is the vanishing gradient problem that made deep networks impractical before ReLU.
SwiGLU is not a curve but a valve. Every modern decoder (Llama, Mistral, Gemma, DeepSeek) uses it in its feed-forward block, and the reason is visible once you drag the gate. A second projection of the same hidden state passes through SiLU and multiplies the first, so the activation becomes a straight line whose slope the gate controls. The unit can be fully open, fully closed, or anywhere between, and the decision is itself learned. It costs a third more parameters in the feed-forward block and is worth every one of them.
The cloud on the right is the other half of what happens between two matrix multiplications. Sixty-four pre-activations for a single token, off-centre and skewed the way real ones are. LayerNorm drags the cloud back so its mean is zero and its spread is one; RMSNorm only does the second half. Neither has any parameters of its own (the learned gain and bias come afterwards), and the effect is that every layer sees inputs at the same scale regardless of what the layer before it did. That is what makes a hundred-layer network trainable with a single learning rate. Batch normalisation did the same job across the batch instead of across the units, which works for images and fails for variable-length sequences, and is why you will not find it in a language model.
Dropout is the floor's lights going out. At rate p, each unit is switched off for this forward pass, and the survivors are scaled up by one over one minus p so the layer's expected output is unchanged. The network cannot rely on any one unit being present, so it spreads the work out, which is a regulariser. Weight decay is the other common one and is not drawn here: it shrinks every parameter a little on every step, which keeps the ramp from becoming a cliff. Modern large language models mostly drop dropout and keep weight decay, because with enough data the model is not the thing that overfits.
The simplification to be honest about: the sixty-four values are one token's worth of one layer, held still. In a real forward pass the pre-activations depend on the previous layer's output, which depends on the normalisation before it, which depends on the residual stream, and the whole thing moves at once. The shapes on this page are exactly right; what they leave out is that the input distribution is itself a moving target, which is precisely the thing normalisation exists to pin down.
The maths
- GELU, tanh approximation
The curve drawn when GELU is selected. The exact form is x times the Gaussian CDF; this approximation is what GPT-2 shipped and what most frameworks still default to. The dip below zero around x = -0.75 is real, not an artefact: slightly negative inputs pass a small negative value and, more importantly, a non-zero gradient.
- SwiGLU
Two projections of the same hidden state. One goes through SiLU and becomes the gate; the other is multiplied by it. The gate slider is the value of xW for one unit: the curve on screen is SiLU(gate) times x, a straight line whose slope the gate sets. Close the gate and the unit passes nothing regardless of x.
- LayerNorm and RMSNorm
Applied across the sixty-four units of one token, not across the batch. LayerNorm subtracts the mean then divides by the standard deviation; RMSNorm skips the subtraction. The learned gain and bias that sit on top are left out here so the picture shows the normalisation itself, which has no parameters at all.
- Inverted dropout
Each unit survives with probability 1 - p, and the survivors are scaled up by 1 / (1 - p) so the expected value of the layer is unchanged. That scaling is why nothing has to be adjusted at inference time, when dropout is simply switched off. The greyed-out units in the third strip are the m_i = 0 cases.
Related terms
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.