A dense model runs every parameter for every token. A mixture-of-experts model replaces the feed-forward block with many parallel experts and a small router that picks a couple of them per token. Total capacity goes up, and the compute per token does not, because only the chosen experts run.
That is the headline and it is true. The interesting content is what has to be arranged for it to hold.
The router is learned, and a learned router left to itself collapses. Early in training a few experts are marginally better, so they get more tokens, so they train faster, so they get more tokens. Turn off the balancing loss here and watch it happen: the busiest expert takes several times the average load while others sit almost idle, which wastes most of the capacity you added the experts for. Every MoE therefore has some balancing mechanism, whether an auxiliary loss or, in more recent designs, a per-expert bias adjusted during training specifically because the auxiliary loss itself costs quality.
Then there is capacity. When experts run as fixed-size batched matrix multiplications, each has a maximum number of tokens it can accept per batch, and tokens routed to an expert that is already full are dropped. The capacity factor is the buffer, and it is a direct trade, because raising it means padding every expert with wasted slots. This is a consequence of fixed-shape batching rather than of the idea: dropless implementations avoid it using variable-size grouped matrix multiplications, and most inference stacks now do.
Dropping sounds fatal and usually is not, which is worth understanding. With top-2 routing a dropped token still gets its other expert, and the residual stream carries it forward regardless, so the system degrades rather than breaks. At top-1 there is no other expert, which is why top-1 routing is far more sensitive to capacity. Push the capacity factor below one and watch the difference.
One number to read carefully: the share of parameters this saves you is the share of EXPERT parameters. Attention, embeddings and norms run for every token no matter what the router decides, so a model that activates two of eight experts is not running a quarter of itself. Mixtral is top-2 of 8 and runs about 28% of its parameters per token.
The maths
- The gate
The router scores every expert, keeps the top k, and softmaxes over only those. The output is the gate-weighted sum of the chosen experts, which is why the link opacity in the diagram follows the gate weight.
- Capacity
With T tokens, E experts and capacity factor phi. Perfect balance gives each expert Tk/E tokens; phi is the buffer over that. Any routing that arrives at a full expert is dropped, which is the red line in each bar.
- Load imbalance
Measured on demand, not on post-capacity load: clipping caps the busiest expert at C, so measuring afterwards makes a collapsed router look healthy. The theoretical worst case is E/k, and at the default eight experts with top-2 the collapsed router here reaches it exactly.
Related terms
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.