Systems

KV cache and paging

Serving a language model is a memory problem. See why reserving the declared context length for every request wastes most of your accelerator, and what paging the cache recovers.

Model
Declared max 4k
Page size 16 tok
Length spread wide
CONTIGUOUS: RESERVE THE DECLARED MAXIMUMreq 11180 / 4096 tokreq 2460 / 4096 tokreq 3190 / 4096 tokreq 470 / 4096 tokPAGED: BLOCKS ON DEMAND, LIKE VIRTUAL MEMORYreq 174 blocksreq 229 blocksreq 312 blocksreq 45 blocksCONCURRENT REQUESTS IN 28.00 GB OF KV BUDGETcontiguous52paged445
131 KB
Per token
1.90 GB
Contiguous waste
2.6 MB
Paged waste
52 → 445
Requests that fit
99% paged
Utilisation

Reserving 536.9 MB for every request wastes 88% of the cache, because almost no request reaches its declared ceiling. Paging in 16-token blocks reserves only what is actually used plus one partial block, which is why the same accelerator serves 9x more concurrent requests.

Every token a model has already processed leaves behind a key and a value at every layer, and they have to stay resident so the next token can attend to them. That is the KV cache. For a 70B model with grouped-query attention it is roughly 300 KB per token, so a single 8000-token conversation is around 2.5 GB, and you are serving many of them at once.

The size is unavoidable. The waste is not. Attention kernels want to stride through the cache in a straight line, so the simple implementation reserves each request's declared maximum length up front as one contiguous run. A request that says it might produce 4096 tokens and actually produces 200 holds all 4096 tokens of memory for its entire life, and it holds them whether or not anything is ever written there.

Paged attention borrows the fix from operating systems. Cut the cache into small fixed-size blocks, give each sequence a table mapping logical position to physical block, and hand out blocks only as the sequence actually grows. Contiguity stops being a requirement because the kernel consults the table. The only waste left is the unused tail of each sequence's final block, so it is bounded by one block per request instead of by the difference between what was declared and what was used.

Drag the declared maximum up and watch the top half of the diagram fill with dead space while the bottom half barely changes. That gap is why the same hardware serves several times more concurrent requests, and it is one of the two or three highest-leverage changes in an inference stack.

The simulation is honest about mechanism, small about scale: four requests with fixed lengths stand in for the hundreds a real server juggles, so read the fragmentation pattern rather than the absolute numbers.

The maths

Cache size per token
bytes/token  =  2×L×hkv×dhead×b\text{bytes/token} \;=\; 2 \times L \times h_{kv} \times d_{head} \times b

Two tensors (keys and values), at every layer, for every KV head, times the head dimension, times bytes per element. Grouped-query attention shrinks h_kv, which is why the 8B preset costs a quarter of the no-GQA 7B despite similar size.

Contiguous waste
Wcontig  =  i(LmaxLi)×bytes/tokenW_{contig} \;=\; \sum_i (L_{max} - L_i) \times \text{bytes/token}

Each request reserves its declared ceiling and uses only what it generates. The gap is dead memory for the whole life of the request, and it is unbounded: declare a bigger maximum and the waste grows with it.

Paged waste
Wpaged    n×(B1)×bytes/tokenW_{paged} \;\le\; n \times (B - 1) \times \text{bytes/token}

With block size B, only the tail of each sequence’s final block can be empty, so waste is bounded by one block per request regardless of what anyone declared. That bound is the entire argument for paging.

Related terms

More visualisations

Building with language models?

These explainers come out of the work. If you want the same thinking applied to your own system, that is what I do.

See how I can help