WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/prompt-optimisation-overfits/

1
00:00:00.035 --> 00:00:12.371
A prompt optimiser rewrites your prompt until it scores well on a few examples: here, eight support tickets, while twenty-four more wait behind a curtain it never sees.

2
00:00:12.371 --> 00:00:24.626
The seed prompt only names the four queues, and gets 16 of 40 runs right, because the house rules are not obvious: adding a colleague is billing, since every seat is paid.

3
00:00:24.626 --> 00:00:35.535
So the optimiser takes one failure at a time and writes one sentence to fix it, and in five steps the dev score climbs from 40 per cent to 100.

4
00:00:35.535 --> 00:00:46.448
Here is the catch: every edit was kept because this score rose, so 100 per cent is now a training score: it fits these eight tickets, not the next one.

5
00:00:46.448 --> 00:01:00.168
Lift the curtain: on twenty-four unseen tickets, ten runs each, the 100 per cent prompt falls to 88 per cent, beaten by the two-sentence prompt from step two, at 92.5.

6
00:01:00.168 --> 00:01:13.364
Read what it learned: two sentences are dev tickets reworded, stuck invoices and two-factor errors, and after the second went in, the SSO ticket went to product in 8 runs of 10.

7
00:01:13.364 --> 00:01:21.293
So keep a test set the optimiser never sees, choose the prompt on that, and read every sentence it added before shipping.