Prompt caching: reuse the stable prefix

Most production prompts are mostly the same. A support bot sends its instructions, its tool definitions, a few worked examples and a reference document on every call, and only the user's question changes at the end. The model is stateless, so without help it reprocesses that whole prefix for every request, and you pay for all of it. Prompt caching keeps the processed prefix on the provider's side for a few minutes, so a later request that starts with the same tokens can read it back instead.

The film records it for real. The prompt is a support assistant for a fictional bike-share: 1,411 tokens of system instructions, 1,573 of tool definitions, 996 of few-shot examples and 1,917 of a fare and station handbook, 5,897 tokens in all, followed by a rider's question of about 18 tokens. Every call went to `anthropic/claude-sonnet-5` through OpenRouter on 11 September 2026, with the provider pinned to Anthropic, temperature 0 and streaming on, and each block ended with a cache_control breakpoint. The whole sequence was recorded three times, and the numbers on the board are the usage fields the API returned, OpenRouter's billed cost and the measured time to the first streamed token, as medians.

The machine is a tree. The trunk is the stable prefix, stacked in prompt order from the root, each block as tall as its token count, and the branches are the questions that change on every call. Each request is a bead that climbs the trunk and leaves along a branch. Beside the trunk, the cache vault has a drawer per block. On the first call the bead processes every block and writes each one into its drawer, which costs a quarter more than a plain read: 1.5 cents. On the next call, with a different question, every block lights gold as it is read back, only the question's 18 tokens are processed fresh, and the call costs 0.17 cents. That held for all 24 different questions.

The part that is easy to miss is that the cache matches from the very first token. It is not a lookup of blocks by content: it is a match on the exact prefix. So order matters. Put the question in front of the instructions and nothing after it can ever match, because every call now starts differently: 18 calls out of 18 read nothing. Stable blocks first and the changing input last is not a style preference, it is what makes reuse possible at all.

The failure mode follows from the same rule. Add a timestamp, a user's name or a request id to the top of the system prompt, which is the most natural place to put it, and the prefix changes on every call. Nothing after the change matches, and all 5,915 tokens are written again at the write price, every time. Move the same edit lower and you lose only what comes after it: a stamp at the top of the tools kept the 1,411 tokens of instructions, one in the examples kept 2,984, and one in the handbook kept 3,980. The API's cache reads land exactly on the block boundaries, which is how the block sizes here were measured rather than estimated.

The saving is money more than speed, at least here. The first streamed token arrived in about a second whether the prefix was cached or not, because network and queueing dominate at this size. With the handbook grown to a 49,947-token prefix, a hit's median time to first token was 1.23 s against 1.44 s for a miss, about a fifth of a second, and the ranges overlapped. The price gap was the same tenfold as before: 1.0 cent against 12.4 cents per call.

The practical rules are short. Order a prompt from most stable to least: instructions, then tools, then examples and documents, then the conversation, then the new input. Keep anything that changes per request, such as dates, names, ids and retrieved snippets, below the last cached block. Watch the cache fields in the usage report, because a prompt that silently stops hitting costs nearly ten times as much and nothing else will tell you.

The honest caveats. This is one model, one provider and one prompt, with explicit breakpoints. OpenAI and Gemini cache automatically, with their own minimum sizes and granularity, but they match prefixes in the same way. The cache here lives for five minutes after its last use, so a quiet endpoint pays the write price again. Providers also refuse to cache a prefix below a minimum length, typically a thousand tokens or more, so a short system prompt may never cache at all; the shortest prefix cached here was the 1,411-token system block. The times were measured from a laptop over the public internet, so they include network variation that a server in the same region would not see, and a much longer prefix would show a larger prefill saving than the one recorded here.