Generating one token from a large dense model requires reading every weight in it. That is the whole cost, and it barely changes whether you are scoring one token or eight. Autoregressive decoding is therefore memory-bound at low batch size, and it leaves most of the arithmetic capacity of the accelerator idle.
Speculative decoding spends that idle capacity. A small draft model proposes several tokens cheaply and sequentially, then the large model scores all of them in one batched pass. Every token the target agrees with is close to free, because verifying eight tokens costs roughly the same single pass as verifying one would have. That equivalence holds while you are memory-bound and stops holding at high batch sizes, where verification becomes compute-bound and the extra tokens are genuinely extra work, which is why serving stacks often disable speculation under load.
Rejection is the interesting case. When the target disagrees at position three, positions four onward are discarded even if the draft would have got them right, because they were conditioned on a token that no longer exists. That is why the accepted region is always a prefix, and why the diagram shows the tail visibly dying rather than scoring each position independently.
The round still produces a token when everything is rejected, because the target contributes one from its own distribution. That is why the expected yield is one more than you might guess, and it means a round can never come away empty.
The property that makes this worth doing at all is that it is free. In the real algorithm the accept test and the correction token are constructed so that the tokens coming out are drawn from exactly the target model's distribution, not an approximation of it. Speculative decoding is a latency optimisation with no quality term to trade against, which is why it can be switched on without an eval. The simulation here abstracts that acceptance machinery into a single rate, but the guarantee is the point.
It does not, however, mean the technique always wins. The round still cost you the draft model's time, so speculative decoding pays off only once expected yield exceeds one plus depth times draft cost. Drag acceptance down to 10% with an expensive draft and the readout will tell you honestly that you have built a slowdown.
The curve on the right is the part to sit with. Speedup rises with draft depth and eventually turns over, because rejected drafts still cost draft-model time. Where it turns depends on how good your draft model is and how much it costs: with a cheap, accurate draft the peak can sit past any depth you would realistically run, and with a weak one it can be at a depth of one. A depth tuned for one pairing is wrong for another.
The maths
- Expected tokens per round
With acceptance probability alpha and gamma drafted tokens. Acceptance is a prefix property, so the chance of getting at least k tokens is alpha to the k. The sum starts at zero rather than one because the target always contributes a correction token, which is why a round can never produce nothing.
- Speedup over plain decoding
The denominator is the cost of a round in units of one target forward pass: one batched verification regardless of gamma, plus gamma sequential draft calls at relative cost c. Plain decoding produces one token per pass, so this ratio is the speedup directly.
- When a maximum exists
Setting the derivative of S with respect to gamma to zero reduces to a strictly decreasing function, so there is at most one turning point, and it exists only when this holds. Above the threshold the curve falls from the start and no draft depth helps.
Related terms
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- ModelA model is the trained artifact at the centre of every AI coding tool: a large file of numbers (parameters) that, given some text, produces the most likely continuation. When people say "which model are you using," this is the thing they mean.
- Output tokensOutput tokens are the tokens a model generates in its response, including any hidden reasoning. They are usually priced higher than input tokens, and turning up effort produces more of them.
- Non-determinismNon-determinism is why the same prompt can give you different answers. At inference the model samples among likely next tokens with a controlled amount of randomness, so runs vary.