Every token a model has already processed leaves behind a key and a value at every layer, and they have to stay resident so the next token can attend to them. That is the KV cache. For a 70B model with grouped-query attention it is roughly 300 KB per token, so a single 8000-token conversation is around 2.5 GB, and you are serving many of them at once.
The size is unavoidable. The waste is not. Attention kernels want to stride through the cache in a straight line, so the simple implementation reserves each request's declared maximum length up front as one contiguous run. A request that says it might produce 4096 tokens and actually produces 200 holds all 4096 tokens of memory for its entire life, and it holds them whether or not anything is ever written there.
Paged attention borrows the fix from operating systems. Cut the cache into small fixed-size blocks, give each sequence a table mapping logical position to physical block, and hand out blocks only as the sequence actually grows. Contiguity stops being a requirement because the kernel consults the table. The only waste left is the unused tail of each sequence's final block, so it is bounded by one block per request instead of by the difference between what was declared and what was used.
Drag the declared maximum up and watch the top half of the diagram fill with dead space while the bottom half barely changes. That gap is why the same hardware serves several times more concurrent requests, and it is one of the two or three highest-leverage changes in an inference stack.
The simulation is honest about mechanism, small about scale: four requests with fixed lengths stand in for the hundreds a real server juggles, so read the fragmentation pattern rather than the absolute numbers.
The maths
- Cache size per token
Two tensors (keys and values), at every layer, for every KV head, times the head dimension, times bytes per element. Grouped-query attention shrinks h_kv, which is why the 8B preset costs a quarter of the no-GQA 7B despite similar size.
- Contiguous waste
Each request reserves its declared ceiling and uses only what it generates. The gap is dead memory for the whole life of the request, and it is unbounded: declare a bigger maximum and the waste grows with it.
- Paged waste
With block size B, only the tail of each sequence’s final block can be empty, so waste is bounded by one block per request regardless of what anyone declared. That bound is the entire argument for paging.
Related terms
- Context windowThe context window is the maximum amount of text, measured in tokens, that a model can consider for a single request. It is a hard ceiling, and it is the main resource you manage when working with an agent.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.
- Cache tokensCache tokens are input tokens served from the prefix cache at a reduced rate. They are how prompt caching shows up as a separate line in your usage numbers.
- TokenA token is the unit of text a model reads and writes: a chunk that is usually part of a word, not a whole word or a single character. Everything is measured in tokens, including your context window and your bill.