The transformer block, exploded

A transformer is a stack of identical blocks, and one block is small enough to hold in your head if you can see it. This is one decoder block, the kind every GPT and Llama model is made of, drawn as an exploded view so each part sits apart from the next. Drag the explode slider to open it up. In the 3D view you can turn it in your hands.

Start at the bottom. Each token is looked up in the embedding table and becomes a vector of d numbers, 768 of them for GPT-2 small. That vector enters the residual stream, the vertical pipe on the left, and the stream is the one thing that runs the full height of the block. Every other part reads from it and writes back into it. Nothing in the block ever replaces the stream; the sub-layers only add to it. That single design choice, from the 2017 paper Attention Is All You Need, is why a hundred of these blocks can be stacked without the gradient vanishing on the way down.

The first sub-layer is attention. A normalised copy of the stream goes in, and each token gets to look at the tokens before it. The threads are that looking: one colour per head, thickness for how much weight the head puts on each earlier word. Pick a token to follow and its threads brighten. Heads specialise, which is why the threads differ so much from one colour to the next. One head watches the previous word, another parks on the first token, another looks at itself. Attention is the only place in the block where tokens exchange information; everything else happens to each token on its own.

The second sub-layer is the MLP, and it is where most of the parameters live. It takes each token's vector, widens it four times (three and a half in Llama, with a gate), applies a nonlinearity, and narrows it back. The gates in the drawing are the hidden units, mostly dark for any given token because the activation is sparse. For GPT-2 small the MLP holds about two thirds of the weights in a block, and in a gated Llama block more than three quarters. When people say the knowledge is in the feed-forward layers, this is what they are pointing at.

At the top, after the last block, the stream is normalised once more and multiplied by the unembedding matrix, which turns the vector into one score per vocabulary entry. Those are the logits, drawn as the histogram, and a softmax turns them into the distribution the token decoding page samples from. GPT-2 reuses the embedding matrix for this step; Llama learns a separate one, which is why its parameter count jumps by half a billion between the two ends of the model.

Switch presets and watch the stats. The four models here span three orders of magnitude, yet the block is the same shape in every one. What changes is the width d, the number of blocks, and a handful of choices that matter more than they look: grouped-query attention shrinks the key and value projections so the KV cache fits in memory, gated MLPs trade a third matrix for a better nonlinearity, RMSNorm drops the bias, and rotary positions replace the learned table. Encoder-decoder transformers, the original design for translation, add a second attention step that reads the encoder's output; the decoder-only stack drawn here is what language models settled on.

What the drawing hides is scale. Eight tokens are drawn where a real context holds thousands; twelve MLP units stand in for three thousand; four heads for twelve, and the threads are illustrative rather than a real model's weights. The parameter counts, however, are exact. Check GPT-2 small against its paper and Llama 3 8B against its model card and the formulas below give the same numbers.