Prompt optimisation can overfit

The scenario is a support inbox for Tallyboard, a made-up project board app, where every message goes to one of four queues: billing, account, bug or product. The house rules are not the obvious ones. Every member is a paid seat, so adding a colleague is billing. A refund owed because the app was down goes to the incident team in bug. The development set is eight tickets, labelled by those rules. Twenty-four more, six per queue, sit behind the curtain as the test set. Everything on screen was recorded with `google/gemini-3.8-flash` at temperature 0 on 11 September 2026.

The seed prompt only names the queues, and it gets 16 of 40 runs right on the development tickets: five runs each, because answers vary even at temperature 0. The optimiser is a small loop in the style of ProTeGi and OPRO, the approaches behind tools such as DSPy's instruction optimisers. Each step it takes the first ticket the current prompt still gets wrong, shows the model that ticket, the right queue and the wrong answer, and asks for the smallest edit that fixes it. The edit is kept only if the dev score rises. Five steps later every run on every development ticket is right.

Then the curtain lifts, and every candidate is scored on the twenty-four held-out tickets, ten runs each. The seed scores 85.8 per cent. The first two sentences, adding users is billing and refunds for downtime are bug, are real house rules, and they carry over: the test score rises to 92.5 per cent at step 2. The next three sentences each fix one more ticket, and two of them repeat their ticket's own words: invoices stuck or failing to download, and two-factor errors. They win dev runs and nothing on the tickets behind the curtain, and after the last one went in, the SSO ticket was sent to product in 8 of 10 runs. The winner on the dev set ends at 88.3 per cent, below every edited prompt before it.

The part people get wrong is what the dev score means once you optimise against it. Every row on the leaderboard was kept because that column rose, so the column is no longer an estimate of how the prompt will do on the next ticket. It is a training score: a measure of how well the prompt fits those eight tickets, luck and quirks included. Nothing inside the loop can see the harm, because the harm is on tickets the loop never looks at.

Pick any candidate when the film is paused to see its lamps on both racks. A green lamp is a ticket every run got right, amber is some of the runs, and red is none, with the queue the prompt chose instead. Watch the free-guest ticket on the stage: the very first sentence, adding users to a workspace is billing, was written for a new hire and sends a free guest to billing as well. One example taught a rule wider than the house rule it was meant to capture.

The practical rules follow. Keep a test set the optimiser never sees, never feed its failures back into the loop, and choose the final prompt by it. Read the diff before you ship: a sentence that echoes a single example's words is a memorised example, not a policy. Prefer the shorter prompt when scores are close, and run each example several times, because a single run hides the tickets that flip.

An honest caveat: the overfit in this film needed the right development set. I ran the same loop on four development sets against the same twenty-four test tickets. On the first three (ordinary tickets, edge cases, and tickets from an outage week) the optimiser mostly learned the house rules: the dev winner scored 110, 115 and 110 of 120 test runs, against a best of 110, 115 and 113 and a seed of 102 or 103. The set in the film is deliberately odd: every label follows the written policy, but most of its tickets are right for an unusual reason, which is what a handful of examples picked from whatever was to hand often looks like. The test set is small too, so the gap in the film is one ticket that flipped, not a measured rate.