Draft, critique, revise, check

Asking a model to "make it better" and then asking again tends to make a draft different rather than better. Each pass fixes something the last pass noticed and quietly moves something else. The film shows a loop that keeps iteration honest: write down the criteria, aim each edit at one of them, and re-score every criterion after every edit.

The task is a customer-facing status update for a payments incident, written from an internal log that invites trouble on purpose: it names the on-call engineer, a paging mistake, and the vendor whose key rotation caused the outage. The rubric has five checks. The update must say who is affected and what they cannot do, say when it began (14:02 UTC), say when the next update comes (15:00 UTC), stay under 80 words, and blame no person, team or vendor. Start time, next update and length are checked in code. Impact and blame are checked by a model grader, three votes, majority wins.

Every text, stamp and score in the film is real. Each version was written by `google/gemini-3.8-flash` through OpenRouter at temperature 0 on 11 September 2026. The critique station is a separate call to the same model, given the rubric and the version, which lists what fails with a note; the stamps on the card are those notes, cut to a few words. The revise step is handed the criterion it targets and, when the rubric is in view, the whole rubric; it never sees the critic's prose. The check station is the code checks plus the grader.

The first draft ran to 136 words and never said when the outage began: its only 14:02 was about which orders might show as payment pending. The critic stamped it on start time and length. The revise step took one target, the start time, with the whole rubric in the prompt, and the model rewrote the update in 43 words. That one edit fixed length as well, and the check re-scored all five criteria and passed every one.

Then the failure mode, which we set up on purpose and want to be plain about. A first recording, where each lap simply fixed the next failing criterion with the rubric in view, produced no regression at all. So the third version comes from the kind of request that often arrives mid-review: one narrow stakeholder ask, "make the impact clearer: tell customers exactly what went wrong and why", sent with the update and the incident log but without the rubric. The model did what was asked. Version three is clearer about impact, and it grew to 103 words and explained that the outage "was caused by an unannounced security key update from our third-party payment vendor". The word count failed in code, and the grader voted three to nothing that it blamed the vendor. Two checks that version two passed turned red, and neither was what the edit aimed at.

The fix was the same aim with the rubric back in the prompt, plus the last version that passed every check, and an instruction to lose nothing that had passed. Version four keeps the new detail about pending orders and duplicate charges, drops the blame, and fits in 58 words. It passes all five.

Two honest notes on the checks themselves. The first version of the start-time check was a regular expression for 14:02 anywhere, and it passed the first draft because of the pending-orders sentence; the critic caught what the code missed, and the check now needs 14:02 to be stated as a start. The critic, meanwhile, said version three had 99 words when it has 103, which is why length is counted in code. Deterministic checks are fast and repeatable but only as good as their patterns, and model checks handle judgement but should be voted and kept away from anything a line of code can count.

What to keep: change one thing per lap, put the full rubric in the prompt with every edit, re-check every criterion after it, and keep the last version that passed so a bad lap can be undone.