Training a language model is a memory problem before it is a compute problem. Every parameter needs its weight, its gradient and its optimiser state resident at the same time, and with Adam in mixed precision that is 16 bytes each. A 7B model is 112 GB of state before you store a single activation, and a single H100 has 80. Full fine-tuning the model everyone calls "small" does not fit on the card everyone calls "big". Distributed training is the set of answers to that arithmetic, and the four strategies here are the ones every framework combines.
The rack shows the model as a loaf of layers and the strategy as the way it is sliced. Data parallelism does not slice at all: every card holds a full copy and trains on its own slice of the batch, then all cards average their gradients in a ring. It is the simplest and the least memory-efficient, and the memory bars do not move as you add cards. What it buys is throughput, and what it costs is one all-reduce of the full gradient buffer per step, which the "Comm / step" stat shows as the largest number on the page.
Tensor parallelism slices every weight matrix down the middle along its hidden dimension, so each card holds half (or an eighth) of every layer. Memory falls as one over the card count, but the two halves of a layer have to exchange partial results before the next layer can start, four times per layer, forward and backward. That traffic is small per exchange and unforgiving in latency, which is why tensor parallelism lives inside the NVLink domain of a single node and almost never crosses InfiniBand.
Pipeline parallelism slices across layers instead: card zero owns the first ten, card one the next ten, and activations flow forward like work along an assembly line. Traffic drops to almost nothing, one activation tensor per stage boundary per micro-batch. The cost is idle time. The last stage cannot start until the first has finished the first micro-batch, and the first cannot start the backward pass until the last has finished the forward. The timeline draws that bubble as empty slots, and the micro-batch slider is the lever that shrinks it.
FSDP (fully sharded data parallel, ZeRO stage 3) keeps the shape of data parallelism and fixes its memory. Weights, gradients and optimiser state are sharded across the cards, and each layer's full weights are gathered onto every card just before they are needed and freed just after. The FSDP chapter of the film shows one layer assembling from its shards and the gauges falling as it does. The bars fall as one over the card count, like tensor parallelism, but the communication is 1.5x data parallel rather than a latency-bound exchange per layer, so it scales across nodes. This is how the 7B that did not fit on one card fits comfortably on eight.
Real runs combine them, which the 3D pill shows in one common shape: tensor parallel across pairs of cards on NVLink, pipeline stages across the rest, and data-parallel replicas across nodes when the rack is bigger than this one. Add gradient checkpointing (recompute activations during the backward pass instead of storing them) and the activation bar collapses to almost nothing at the cost of a third more compute. Drop precision to fp8 and the weights halve again, though the fp32 master copy and the optimiser moments stay exactly where they were.
The numbers here are the first-order estimates an engineer writes on a whiteboard, not a profiler trace. Activation memory uses the Megatron estimate of 34 bytes per token per hidden unit per layer, which assumes FlashAttention has already removed the sequence-squared term. Communication is counted as bytes and converted to time on a single link with no overlap, where real frameworks overlap most of it with compute. And the CUDA context, fragmentation, and the memory the framework itself needs are not modelled, so "fits" here means "fits with a few gigabytes to spare on paper". The shape of the trade-offs survives all of that; the exact gigabyte does not.
The maths
- Bytes per parameter in training
This is the number behind the "Per card" stat under data parallel: the 7B model is 112 GB before a single activation is stored. Switch precision from bf16 to fp32 and the weights and gradients double while the optimiser state shrinks, and the total stays at 16. Mixed precision does not shrink the parameter footprint; it shrinks activations and compute.
- Memory per card under sharding
P is the parameter count, t the tensor-parallel degree, p the pipeline stages and s the FSDP shard count (only one of these is above 1 at a time on the strategy pills, two under 3D). A is activation memory, which depends on how many layers this card holds and how much of the batch is in flight. This is why the bars fall as you add cards under everything except data parallel.
- Ring all-reduce traffic
Each of n cards sends and receives nearly twice the buffer B: once for the reduce-scatter, once for the all-gather. Data parallel does this on the gradient buffer once per step. FSDP does an all-gather forward, another backward, and a reduce-scatter, so the "Comm / step" stat reads 1.5x data parallel. Tensor parallel does it four times per layer on an activation-sized buffer, which is small, but it happens in the critical path of every layer.
- The pipeline bubble
With p stages and m micro-batches, the pipeline spends p minus 1 slots filling and the same draining. Drag the micro-batch slider and watch the dead space in the timeline shrink; the price is that each micro-batch is smaller, so the matrix multiplications get less efficient and the activations for min(m, p) micro-batches sit in memory at once.
Related terms
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.