# Mixture-of-experts routing

**AI Engineering Visualised** · Architecture · Interactive explainer · Updated 2026-08-16

A mixture-of-experts model splits each feed-forward layer into many expert networks and uses a router to send each token to only a few of them. The model can hold far more parameters than it uses for any one token.

> How a model can have a trillion parameters and run like a much smaller one. Route tokens to experts, watch the router collapse onto favourites, and see tokens dropped at capacity.

A dense model runs every parameter for every token. A mixture-of-experts model replaces the feed-forward block with many parallel experts and a small router that picks a couple of them per token. Total capacity goes up, and the compute per token does not, because only the chosen experts run.

That is the headline and it is true. The interesting content is what has to be arranged for it to hold.

The router is learned, and a learned router left to itself collapses. Early in training a few experts are marginally better, so they get more tokens, so they train faster, so they get more tokens. Turn off the balancing loss here and watch it happen: the busiest expert takes several times the average load while others sit almost idle, which wastes most of the capacity you added the experts for. Every MoE therefore has some balancing mechanism, whether an auxiliary loss or, in more recent designs, a per-expert bias adjusted during training specifically because the auxiliary loss itself costs quality.

Then there is capacity. When experts run as fixed-size batched matrix multiplications, each has a maximum number of tokens it can accept per batch, and tokens routed to an expert that is already full are dropped. The capacity factor is the buffer, and it is a direct trade, because raising it means padding every expert with wasted slots. This is a consequence of fixed-shape batching rather than of the idea: dropless implementations avoid it using variable-size grouped matrix multiplications, and most inference stacks now do.

Dropping sounds fatal and usually is not, which is worth understanding. With top-2 routing a dropped token still gets its other expert, and the residual stream carries it forward regardless, so the system degrades rather than breaks. At top-1 there is no other expert, which is why top-1 routing is far more sensitive to capacity. Push the capacity factor below one and watch the difference.

One number to read carefully: the share of parameters this saves you is the share of EXPERT parameters. Attention, embeddings and norms run for every token no matter what the router decides, so a model that activates two of eight experts is not running a quarter of itself. Mixtral is top-2 of 8 and runs about 28% of its parameters per token.

## The maths

### The gate

```latex
g_i \;=\; \frac{\exp(r_i)}{\sum_{j \in \text{top-}k} \exp(r_j)}, \qquad \mathbf{y} \;=\; \sum_{i \in \text{top-}k} g_i \, E_i(\mathbf{x})
```

The router scores every expert, keeps the top k, and softmaxes over only those. The output is the gate-weighted sum of the chosen experts, which is why the link opacity in the diagram follows the gate weight.

### Capacity

```latex
C \;=\; \Big\lceil \phi \cdot \frac{T \cdot k}{E} \Big\rceil
```

With T tokens, E experts and capacity factor phi. Perfect balance gives each expert Tk/E tokens; phi is the buffer over that. Any routing that arrives at a full expert is dropped, which is the red line in each bar.

### Load imbalance

```latex
I \;=\; \frac{\max_e D_e}{\bar{D}}, \qquad D_e = \text{demand before capacity}
```

Measured on demand, not on post-capacity load: clipping caps the busiest expert at C, so measuring afterwards makes a collapsed router look healthy. The theoretical worst case is E/k, and at the default eight experts with top-2 the collapsed router here reaches it exactly.

## Related terms

- [Parameters](https://understandingdata.com/ai-coding-dictionary/parameters/)
- [Model](https://understandingdata.com/ai-coding-dictionary/model/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)
- [Token](https://understandingdata.com/ai-coding-dictionary/token/)

## More visualisations

- [Quantisation and outliers](https://understandingdata.com/ai-engineering-visualised/quantisation/)
- [KV cache and paging](https://understandingdata.com/ai-engineering-visualised/kv-cache/)

---

Source: https://understandingdata.com/ai-engineering-visualised/mixture-of-experts/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-08-16.