# Prompt Caching Visualised

**AI Engineering Visualised** · Inference systems · Interactive explainer · Updated 2026-09-08

Prompt caching stores the processed state of a shared prompt prefix so later calls can skip recomputing it. Which cached blocks survive an edit depends on where in the prompt the edit lands.

> Change a prompt and trace exactly which cached blocks survive. Explore shared prefixes, block size, expiry and cache isolation in an interactive 2D and 3D prefix tree.

Keep the stable context at the beginning and put the changing question afterwards. In this small experiment, the first sixteen words describe Northbank’s assistant and its policy rules. Changing the question preserves that opening. Changing the very first word breaks reuse from the start.

The model compares exact token sequences and caches complete blocks. Each block is identified by its preceding prefix as well as its own content. Repeated words further down a different branch therefore cannot produce a hit. This illustrates the block-prefix mechanism documented by vLLM.

Try four experiments: change the opening, increase block size, advance the clock beyond ten minutes, and switch the team namespace. The orange blocks expose a different reason for recomputation in each case. Selecting a block reveals its exact words and positions. A block size of five exposes a final partial block that cannot enter the complete-block cache.

This lab uses visible word tokens and an illustrative fixed expiry. It is not a provider pricing or performance calculator. Real serving systems have different tokenisers, memory pressure, retention rules and additional cache-key inputs. A full KV prefix match does not mean zero model work: a runtime may recompute the final prompt token or block to obtain logits. Reusing prompt state also does not mean reusing a previously generated answer.

## The maths

### Reuse only complete prefix blocks

```latex
R = B \left\lfloor \frac{L}{B} \right\rfloor, \qquad C = N - R
```

L is the longest exact opening sequence in a live cache entry from the same namespace. B is the block size, N is the incoming token count, R counts tokens in reusable KV blocks and C counts tokens outside those blocks. These counts exclude runtime-specific recomputation to obtain logits.

## Sources

- [vLLM: final-token recomputation in the KV cache manager](https://github.com/vllm-project/vllm/blob/main/vllm/v1/core/kv_cache_manager.py)
- [vLLM: Automatic Prefix Caching design](https://docs.vllm.ai/en/latest/design/prefix_caching/)

## Related terms

- [Prefix cache](https://understandingdata.com/ai-coding-dictionary/prefix-cache/)
- [Cache tokens](https://understandingdata.com/ai-coding-dictionary/cache-tokens/)
- [Input tokens](https://understandingdata.com/ai-coding-dictionary/input-tokens/)
- [Inference](https://understandingdata.com/ai-coding-dictionary/inference/)

## More visualisations

- [KV cache and paging](https://understandingdata.com/ai-engineering-visualised/kv-cache/)
- [Byte pair encoding](https://understandingdata.com/ai-engineering-visualised/tokenisation/)
- [MCP Architecture Visualised](https://understandingdata.com/ai-engineering-visualised/mcp-architecture/)

---

Source: https://understandingdata.com/ai-engineering-visualised/prompt-caching-prefix/
From AI Engineering Visualised by James Phoenix, Understanding Data. Published 2026-09-08.