Summary
When you are improving a large surface area (200 components, 200 assets, 500 articles) the bottleneck stops being the agent’s ability to fix things and becomes its ability to choose what to look at. Unweighted sampling re-tests the popular item forty times while a quarter of the catalogue is never touched. The fix is a small persistent file that records, per item, how many times it has been selected, how many clean passes it survived, how many findings it was named in, and every other item it has been combined with. Selection then becomes a weighted round robin over that file. The ledger is what turns a swarm of sub-agents from brute force into search.
Two things you have to know before you can search
Improving a big surface area needs two pieces of state, and neither of them lives in the code:
- What may already have been searched and fixed. Otherwise every run re-derives the same three findings and you pay for them again.
- What has never been touched at all. This is the expensive half. Untouched code is where the defect density is highest, and it is invisible in a diff, a test report, or a git log.
Neither question is answerable from the repository. A green test suite tells you the tests passed, not that a component was ever placed on screen next to another one. So the state has to be written down deliberately.
The setup
I have an animation engine in a worktree: roughly 190 React primitives in a 2D catalogue, about 200 Blender scripts producing 324 .glb assets, all driven through Remotion. Three checking layers sit on it. Mutation testing on the pure core (2,713 mutants over the clock, geometry, anchors and layout functions; 90.2% killed) proves the unit tests actually check the maths. Deterministic pixel checks catch illegible text, low contrast, clipping, overlap, blank frames and ink outside the safe area. A vision model reads a contact sheet for what the pixel checks cannot phrase.
The interesting bugs are not in any single primitive. Components that each pass their own tests break each other when combined, which means the real search space is not 190 items but the 17,955 pairs they can form. After 34 runs I have seen 3,434 of those pairs. Under a fifth. No amount of blind sampling is going to close that.
The loop
The loop is deliberately cheap to run and produces nothing you keep:
Generate a film plan from a seed. Render it to PNG frames in one headless Chromium with every text run and sample box measured. Run the pixel checks, then the vision pass. Write findings that name the seed, the composition, the exact frame and the defect. Throw the video away. Update the ledger. Patch the animation library. Run again.
The output of a run is not a video. It is a diff to the ledger plus a list of findings, each with a reproduction command that replays one recorded plan byte for byte.
The algorithm, which is a weighted round robin
Selection happens at two levels, and both read the same ledger.
Inside a composition, each candidate gets a weight that is three factors multiplied:
weight = room * neglect * freshness
room = min(2, fit / minScale) // capped; fit is already a hard gate
neglect = 1 / (1 + timesSelected) // the ledger term
freshness = 1 + 0.8 * (unseenPairs / partners)
neglect is the whole point. An item nobody has drawn is twenty times likelier to be picked than one drawn twenty times. Without it the generator keeps re-testing whatever the catalogue happens to list first. freshness asks whether this candidate has ever shared a scene with what is already placed, because composition bugs live in the pairs and a pair never tried is worth more than a third pass over a known one.
Across compositions, a cursor walks forward through seed space (the round robin part) and --target least-tested scores a window of up to 1,200 candidate plans by summing 1/(1+selected) over their items, plus 0.6 for every unseen pair, and takes the best N. The cursor advances past the whole window either way, so a later run never re-walks ground a previous one scored.
The consequence worth stating: a plan is a pure function of (seed, ledger), not of the seed alone. Both are recorded. Every report carries the ledger revision and every rendered composition keeps its resolved plan next to its frames, because once the ledger moves, a seed no longer means what it meant.
What the ledger actually holds, per item: selected, clean, findings, lastSeed, and the list of every partner it has appeared with. clean is the field that separates “selected” from “exercised”. An item can be picked and still tell you nothing if the run found something unrelated.
What this buys, in numbers
The distribution before weighting was the expected power law. Today bit-row sits at 126 selections while hash-ring, cache-hierarchy and outbox-timeline sit at 1 each, and those three are only at 1 because the weighting reached them at all: 4,765 selections over 185 items, 3,811 findings, 7,455 clean passes. Coverage moved from 159 of 190 items exercised to 185 of 190 over the run history, and pair coverage from 640 to 3,434.
The real payoff was not more findings. It was a diagnosis. The ledger showed reach saturating: 50 seeds reached 36 of 190 items, 600 seeds reached 70, and more seeds never helped. Those items were unreachable, not unlucky. They drew a fixed picture at fixed coordinates, so the only way to put one in a smaller slot was to scale the finished picture, type and all, which pushed every label under the legibility floor. Fixing the architecture (hand a component a box, never a scale) took reach from 36 to 60 with no catalogue change, then 72, then 89. A saturating coverage curve is a design finding. You cannot see it without the ledger.
Where adversarial agents come in
Once “least seen” is a number rather than a hunch, you can aim an expensive agent at it. A critic pass costs real money: my vision calibration run was 390 OpenRouter calls for $4.53 over ten films. Spending that uniformly is waste, because most of it lands on code that has already survived eighty clean passes.
So the ledger becomes the targeting function for the adversarial layer. The reviewer goes at the items with the lowest clean count, the pairs never combined, and the items whose findings-to-clean ratio is worst. The expensive reviewer should read the code the cheap loop has seen least. That inverts the usual instinct, which is to review what changed.
The failure mode
The ledger is a measurement, and measurements get gamed. Three rules keep it honest:
- A check that is wrong should be corrected and explained. A check that is inconvenient should not be weakened.
- Genuinely acceptable defects go on an allowlist with a written reason, never by relaxing a floor.
- Legibility and reach floors are a ratchet. Raise them when a migration lands, never lower one to make a run go green.
Without those, coverage numbers climb while quality does not, and you have built a very expensive way to feel productive. That is the antipattern this pattern degrades into.
When it is not worth it
Under roughly fifty items, the ledger costs more than exhaustive sweeps. If items are genuinely independent (no combination effects), you want per-item coverage, not pair coverage, and the freshness term is dead weight. And if you cannot check a result automatically, weighting the search does nothing: you will just reach more places you cannot evaluate.
Related
- Monte Carlo Quality Assurance - Brute-force stochastic hardening. This is the same loop with the sampling weighted.
- Property-Based Testing - Generative inputs against invariants. The ledger adds memory across runs.
- Building the Harness - The generate, render, check, report machinery this pattern needs underneath it.
- Put Browser Verification at the End of the Agent Loop - Why the pixel pass sits where it does.
- Agent Sprawl - What you get when you scale sub-agents without a search policy.
- Institutional Memory Learning Files - The same idea applied to knowledge rather than coverage.
