# Prompt caching: reuse the stable prefix

**AI Engineering Visualised** · Inference · Film, 1:44 · Updated 2026-09-11

Prompt caching lets an API reuse the computed state of a prompt prefix across calls, so a long system prompt is processed once and read back cheaply. The cache matches from the first token, which is why anything that changes at the top of the prompt discards all of it.

> A support bot resends the same 5,897-token prefix on every call. Watch real Claude Sonnet 5 calls read it back from a prompt cache for a tenth of the price, see a question-first prompt reuse nothing, and see one timestamp at the top of the system prompt throw the whole cache away.

Watch the film: https://understandingdata.com/ai-engineering-visualised/prompt-caching-prefix/
Video file: https://assets.understandingdata.com/viz/prompt-caching-prefix/video/film.mp4?v=080c368319

Most production prompts are mostly the same. A support bot sends its instructions, its tool definitions, a few worked examples and a reference document on every call, and only the user's question changes at the end. The model is stateless, so without help it reprocesses that whole prefix for every request, and you pay for all of it. Prompt caching keeps the processed prefix on the provider's side for a few minutes, so a later request that starts with the same tokens can read it back instead.

The film records it for real. The prompt is a support assistant for a fictional bike-share: 1,411 tokens of system instructions, 1,573 of tool definitions, 996 of few-shot examples and 1,917 of a fare and station handbook, 5,897 tokens in all, followed by a rider's question of about 18 tokens. Every call went to `anthropic/claude-sonnet-5` through OpenRouter on 11 September 2026, with the provider pinned to Anthropic, temperature 0 and streaming on, and each block ended with a cache_control breakpoint. The whole sequence was recorded three times, and the numbers on the board are the usage fields the API returned, OpenRouter's billed cost and the measured time to the first streamed token, as medians.

The machine is a tree. The trunk is the stable prefix, stacked in prompt order from the root, each block as tall as its token count, and the branches are the questions that change on every call. Each request is a bead that climbs the trunk and leaves along a branch. Beside the trunk, the cache vault has a drawer per block. On the first call the bead processes every block and writes each one into its drawer, which costs a quarter more than a plain read: 1.5 cents. On the next call, with a different question, every block lights gold as it is read back, only the question's 18 tokens are processed fresh, and the call costs 0.17 cents. That held for all 24 different questions.

The part that is easy to miss is that the cache matches from the very first token. It is not a lookup of blocks by content: it is a match on the exact prefix. So order matters. Put the question in front of the instructions and nothing after it can ever match, because every call now starts differently: 18 calls out of 18 read nothing. Stable blocks first and the changing input last is not a style preference, it is what makes reuse possible at all.

The failure mode follows from the same rule. Add a timestamp, a user's name or a request id to the top of the system prompt, which is the most natural place to put it, and the prefix changes on every call. Nothing after the change matches, and all 5,915 tokens are written again at the write price, every time. Move the same edit lower and you lose only what comes after it: a stamp at the top of the tools kept the 1,411 tokens of instructions, one in the examples kept 2,984, and one in the handbook kept 3,980. The API's cache reads land exactly on the block boundaries, which is how the block sizes here were measured rather than estimated.

The saving is money more than speed, at least here. The first streamed token arrived in about a second whether the prefix was cached or not, because network and queueing dominate at this size. With the handbook grown to a 49,947-token prefix, a hit's median time to first token was 1.23 s against 1.44 s for a miss, about a fifth of a second, and the ranges overlapped. The price gap was the same tenfold as before: 1.0 cent against 12.4 cents per call.

The practical rules are short. Order a prompt from most stable to least: instructions, then tools, then examples and documents, then the conversation, then the new input. Keep anything that changes per request, such as dates, names, ids and retrieved snippets, below the last cached block. Watch the cache fields in the usage report, because a prompt that silently stops hitting costs nearly ten times as much and nothing else will tell you.

The honest caveats. This is one model, one provider and one prompt, with explicit breakpoints. OpenAI and Gemini cache automatically, with their own minimum sizes and granularity, but they match prefixes in the same way. The cache here lives for five minutes after its last use, so a quiet endpoint pays the write price again. Providers also refuse to cache a prefix below a minimum length, typically a thousand tokens or more, so a short system prompt may never cache at all; the shortest prefix cached here was the 1,411-token system block. The times were measured from a laptop over the public internet, so they include network variation that a server in the same region would not see, and a much longer prefix would show a larger prefill saving than the one recorded here.

## The maths

### The cache matches an exact prefix

```latex
T_{\text{cached}} \;=\; \sum_{j < k} T_j, \qquad k = \text{the first block that changed}
```

The prompt is four blocks of 1,411, 1,573, 996 and 1,917 tokens. A change at the start of block k keeps every block before it and invalidates the rest, which is exactly what the API reported: 0 tokens cached with a stamp in the system prompt, 1,411 with one in the tools, 2,984 in the examples and 3,980 in the handbook.

### What a call costs

```latex
C \;=\; \frac{0.2\,R \;+\; 2.5\,W \;+\; 2\,U \;+\; 10\,O}{10^6}\ \text{dollars}
```

R tokens read from the cache, W written to it, U input tokens neither read nor written (the question), and O output tokens, at the list prices for Claude Sonnet 5 on OpenRouter in dollars per million. The formula predicts every recorded prompt cost within two percent: 1.5 cents for the first call, 0.17 cents for a hit.

### The write premium pays back on the first hit

```latex
1.25\,P + 0.1\,P\,(n-1) \;<\; n\,P \iff n > 1.28
```

Writing a prefix of P dollars costs a quarter more than sending it plain, and every later read costs a tenth. Two calls inside the cache lifetime (five minutes here) already cost less than two uncached calls, and every call after that saves nine tenths of the prefix.

### Time to first token

```latex
t_{\text{first}} \;\approx\; t_{\text{network}} + t_{\text{queue}} + t_{\text{prefill}}(U + W)
```

Caching removes only the prefill of the cached tokens. Through OpenRouter the other terms dominated: about a second either way at 5,897 tokens, and at 49,947 tokens a median of 1.23 s on a hit against 1.44 s on a miss, with the two ranges overlapping.

## Transcript

### 0:00 The same prefix

Every call to this support bot starts with the same 5,897 tokens of instructions, tools, examples and a handbook. Only the question changes, yet you pay for all of it every time.

### 0:17 The first call

So cache it. The first call processes every block and writes it to the cache, at a quarter over the normal price: 1.5 cents.

### 0:26 A cache hit

The next question reads the whole prefix back from the cache. Only 18 tokens are new, so the call costs 0.17 cents.

### 0:35 One trunk, many branches

That held for all 24 questions we sent. The trunk is shared; only the branches change.

### 0:41 Order matters

Here is the part that surprises people. The cache matches from the very first token, so order matters. Put the question first, and none of 18 calls reused anything.

### 0:53 One early edit

And here is the failure mode. Add a timestamp to the top of the system prompt, and nothing after it matches. All 5,915 tokens are written again.

### 1:05 Only what follows is lost

Edit lower down, and you lose only what comes after the edit. Change the tools and the instructions survive; change the handbook and 3,980 tokens still come from the cache.

### 1:18 Money, not speed

The saving here is money, not speed. First tokens took about a second either way, and even a 50,000-token prefix saved only a fifth of a second.

### 1:28 Stable first

So build prompts like a tree: stable blocks first, the changing input last, and no timestamps or names in the prefix. And keep it warm: this cache expires five minutes after its last use.

## Related terms

- [Prefix cache](https://understandingdata.com/ai-coding-dictionary/prefix-cache/)
- [Cache tokens](https://understandingdata.com/ai-coding-dictionary/cache-tokens/)
- [Input tokens](https://understandingdata.com/ai-coding-dictionary/input-tokens/)
- [Output tokens](https://understandingdata.com/ai-coding-dictionary/output-tokens/)
- [Context window](https://understandingdata.com/ai-coding-dictionary/context-window/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)

## More visualisations

- [KV cache and paging](https://understandingdata.com/ai-engineering-visualised/kv-cache/)
- [The anatomy of a good prompt](https://understandingdata.com/ai-engineering-visualised/anatomy-of-a-prompt/)
- [Conversation history versus persistent memory](https://understandingdata.com/ai-engineering-visualised/history-versus-memory/)
- [What are tokens?](https://understandingdata.com/ai-engineering-visualised/what-are-tokens/)

---

Source: https://understandingdata.com/ai-engineering-visualised/prompt-caching-prefix/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-11.