Evaluator-optimizer

Some things are hard to write and easy to check. A translation that drops a clause, a summary that misses the main point, code that ignores a stated constraint: spotting the problem is a smaller task than avoiding it in the first place.

Evaluator-optimizer exploits that asymmetry. One call generates, a second call critiques against explicit criteria, and the critique goes back for revision.

let draft = await generate(task)

for (let round = 0; round < 3; round++) {
  // separate call, own context, no memory of writing it
  const critique = await evaluate(draft, criteria)
  if (critique.acceptable) break
  draft = await revise(draft, critique.feedback)
}

Why the evaluator has to be separate

Asking one call to produce and then judge its own work gets you a defence of what it already wrote. A fresh call with only the output and the criteria, and no memory of having produced it, criticises far more honestly. The isolation is the mechanism, not an implementation detail.

This is LLM-as-judge placed inside the production loop rather than in an offline evaluation, and it wants the same thing: a rubric with defined levels, not "is this good?"

When it pays

• Clear criteria exist. "Every claim cited", "under 200 words", "no passive voice". The vaguer the criterion, the less the loop converges.
• Judging is genuinely cheaper than producing. If the evaluator has to redo the work to check it, you have doubled cost for nothing.
• A first draft is usefully close. The loop refines; it does not rescue.

When it does not

If quality is as hard to assess as to produce, the evaluator is just a second opinion of equal unreliability, and you have paid twice for it. Creative and open-ended work is often like this: the critique becomes generic advice that makes each round blander.

Bound the iterations at two or three. Beyond that, output usually oscillates rather than improves.