Continuous batching

Generation lengths are wildly unequal: one user asks for a word, another for an essay, and nothing about the prompt reliably tells you which. That variance is the central scheduling problem of serving.

Classic static batching handles it badly in an instructive way. Grab a batch of requests, run them together, and the batch is only finished when its LONGEST member is. Every request that finishes early leaves its slot idle, hatched red in the upper panel, until the straggler completes. With one 26-step request batched beside a 4-step one, most of that window is a GPU multiplying zeros.

Continuous batching, sometimes called in-flight batching, is the fix every serious serving stack converged on: admission happens per step, not per batch. The moment a slot frees, the next queued request takes it. Nothing about the arithmetic gets faster, and the lower panel does exactly the same work; it simply refuses to let a finished request hold capacity hostage. Watch the completed counter: the gap opens the first time a short request ends inside a long batch, and does not close until the static schedule finally drains, long after the continuous panel has finished.

Two things make this practical for transformers specifically. Each decode step is a batch-wide forward pass anyway, so swapping one sequence for another between steps is cheap. And paged KV caches remove the memory obstacle: a newly admitted request can take non-contiguous cache blocks, so admission does not wait for a tidy hole. Continuous batching came first, from Orca, and paged attention arrived in vLLM a year later, but they belong together because each one removes the other's obstacle.

The simplification to be honest about: this model charges every request one slot for its whole length, ignoring the prefill step being heavier than decode steps and ignoring per-token batching effects on kernel efficiency. Those change the constants, not the shape: idle slots are pure loss under any accounting.