Given a fixed amount of compute, you face a genuine trade: a bigger model seen less data, or a smaller model seen more. For years the field leaned hard toward bigger, and GPT-3 is the monument to that instinct: 175 billion parameters trained on 300 billion tokens.
The Chinchilla result reframed the question as calculus. Fit a simple law for loss as a function of parameters N and tokens D, add the constraint that compute is roughly six times N times D, and minimise. The answer was uncomfortable: at Gopher's budget you should have trained a model a quarter the size on nearly five times the data. The 70-billion-parameter Chinchilla, trained on 1.4 trillion tokens for the same compute as the 280-billion-parameter Gopher, beat Gopher, GPT-3 and MT-NLG across the board.
The map shows the whole argument at once. Grey curves are iso-loss contours: everywhere on one curve is an equally good model. The straight diagonal is your budget, every point on it costing the same FLOPs. The best model your budget buys is where the diagonal touches the lowest contour, and the green dashed ridge collects those touching points across all budgets. Slide the budget and read the tokens-per-parameter stat: it runs from about nineteen at the smallest budgets to about fourteen at the largest, passing sixteen at Chinchilla's own budget. The paper's rule of thumb says twenty; the refit says the truth is a little lower and drifts gently with budget, which is what a rule of thumb compresses away.
Then look at Llama 3 8B, sitting far off the ridge on the data-heavy side, and resist calling it a mistake. Chinchilla optimises loss per unit of TRAINING compute and says nothing about inference. A small model overtrained well past its optimum is worse per training FLOP but far cheaper to serve forever after, and once deployment costs dominate, deliberately overtraining is the rational choice. The frontier tells you where optimal is; economics decides whether you want to stand on it.
A footnote worth knowing: the constants here are not the ones printed in the Chinchilla paper. A 2024 replication showed the published parametric fit contradicts the paper's own headline model, putting the optimum at its budget near ninety tokens per parameter rather than twenty; this page uses the replication's refit, under which the ridge genuinely passes through the model the paper trained. Empirical laws are like that: data quality, architecture and objective all shift the constants, which is why labs refit these curves rather than quoting them.
The maths
- The fitted loss law
E = 1.82, A = 482, B = 2085, alpha = 0.35, beta = 0.37: the 2024 replication refit of the Chinchilla law, used because the originally published constants contradict the paper’s own headline model. E is the irreducible loss no scale removes; the power laws are the model-size and data-size deficits.
- The budget constraint
A transformer costs about six FLOPs per parameter per token (two for the forward pass, four for the backward). In log-log space this constraint is the straight diagonal the slider moves.
- The compute-optimal split
Substitute the constraint into the loss and set the derivative to zero. Both exponents are near a half, so doubling the budget scales parameters and tokens up together. They are not exactly equal, so the optimal tokens-per-parameter ratio drifts gently with budget rather than being a mathematical constant; under this fit it stays in the mid-teens across the slider, a little below the headline twenty.
Related terms
- TrainingTraining is the process that produces a model: showing it enormous amounts of text and adjusting its parameters until it gets good at predicting what comes next. It happens once, before you ever use the model.
- ParametersParameters are the learned numbers (weights) inside a model that hold everything it appears to know. The count of them is what people mean by model size, and they are fixed once training ends.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.