Keep the stable context at the beginning and put the changing question afterwards. In this small experiment, the first sixteen words describe Northbank’s assistant and its policy rules. Changing the question preserves that opening. Changing the very first word breaks reuse from the start.
The model compares exact token sequences and caches complete blocks. Each block is identified by its preceding prefix as well as its own content. Repeated words further down a different branch therefore cannot produce a hit. This illustrates the block-prefix mechanism documented by vLLM.
Try four experiments: change the opening, increase block size, advance the clock beyond ten minutes, and switch the team namespace. The orange blocks expose a different reason for recomputation in each case. Selecting a block reveals its exact words and positions. A block size of five exposes a final partial block that cannot enter the complete-block cache.
This lab uses visible word tokens and an illustrative fixed expiry. It is not a provider pricing or performance calculator. Real serving systems have different tokenisers, memory pressure, retention rules and additional cache-key inputs. A full KV prefix match does not mean zero model work: a runtime may recompute the final prompt token or block to obtain logits. Reusing prompt state also does not mean reusing a previously generated answer.
The maths
- Reuse only complete prefix blocks
L is the longest exact opening sequence in a live cache entry from the same namespace. B is the block size, N is the incoming token count, R counts tokens in reusable KV blocks and C counts tokens outside those blocks. These counts exclude runtime-specific recomputation to obtain logits.
Sources and model assumptions
Follow the original mechanism behind this explainer. The interactive examples identify their toy data and simplifying assumptions above.
Put it to work
Related terms
- Prefix cacheA prefix cache lets a provider reuse the unchanged front of your request instead of reprocessing it, so repeated prefixes are cheaper and faster. It is the main reason keeping the start of your prompt stable pays off.
- Cache tokensCache tokens are input tokens served from the prefix cache at a reduced rate. They are how prompt caching shows up as a separate line in your usage numbers.
- Input tokensInput tokens are the tokens you send in a request: the system prompt, the conversation history, loaded files, and tool definitions. You are billed for them, and they count against the context window.
- InferenceInference is the act of running a trained model to get an answer: text goes in, a prediction comes out. Every message you send to a coding agent is an inference. It is the opposite end of the lifecycle from training.