WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/prompt-caching-prefix/

1
00:00:00.047 --> 00:00:10.631
Every call to this support bot starts with the same 5,897 tokens of instructions, tools, examples and a handbook.

2
00:00:10.631 --> 00:00:16.626
Only the question changes, yet you pay for all of it every time.

3
00:00:16.626 --> 00:00:17.538
So cache it.

4
00:00:17.538 --> 00:00:25.979
The first call processes every block and writes it to the cache, at a quarter over the normal price: 1.5 cents.

5
00:00:25.979 --> 00:00:30.689
The next question reads the whole prefix back from the cache.

6
00:00:30.689 --> 00:00:34.781
Only 18 tokens are new, so the call costs 0.17 cents.

7
00:00:34.781 --> 00:00:37.761
That held for all 24 questions we sent.

8
00:00:37.761 --> 00:00:41.275
The trunk is shared; only the branches change.

9
00:00:41.275 --> 00:00:44.217
Here is the part that surprises people.

10
00:00:44.217 --> 00:00:48.893
The cache matches from the very first token, so order matters.

11
00:00:48.893 --> 00:00:53.494
Put the question first, and none of 18 calls reused anything.

12
00:00:53.494 --> 00:00:55.866
And here is the failure mode.

13
00:00:55.866 --> 00:01:02.248
Add a timestamp to the top of the system prompt, and nothing after it matches.

14
00:01:02.248 --> 00:01:05.111
All 5,915 tokens are written again.

15
00:01:05.111 --> 00:01:09.772
Edit lower down, and you lose only what comes after the edit.

16
00:01:09.772 --> 00:01:18.177
Change the tools and the instructions survive; change the handbook and 3,980 tokens still come from the cache.

17
00:01:18.177 --> 00:01:20.756
The saving here is money, not speed.

18
00:01:20.756 --> 00:01:28.422
First tokens took about a second either way, and even a 50,000-token prefix saved only a fifth of a second.

19
00:01:28.422 --> 00:01:36.510
So build prompts like a tree: stable blocks first, the changing input last, and no timestamps or names in the prefix.

20
00:01:36.510 --> 00:01:41.280
And keep it warm: this cache expires five minutes after its last use.