Pairwise comparison

Absolute scoring is hard for a reason: "is this a 7 or an 8" has no stable answer, for a model or a person. "Is A better than B" is a different kind of question, and both models and humans answer it far more consistently.

That makes pairwise comparison the natural tool for the question you usually actually have: did my change make things better?

The loop

for (const testCase of evalSet) {
  const a = await runOldPrompt(testCase.input)
  const b = await runNewPrompt(testCase.input)

  // randomise order: judges favour whichever they see first
  const [first, second] = Math.random() < 0.5 ? [a, b] : [b, a]
  const winner = await judge(testCase.input, first, second)
  record(testCase.id, winner)
}

Aggregate to a win rate. "The new prompt wins 31 of 40, loses 5, ties 4" is a claim you can act on, and it survives being re-run.

The bias you must control

Judges have position bias: they systematically favour the first option, or sometimes the last. If you always present the new version second, you will measure position as much as quality.

Two defences, and you want at least one:

• Randomise the order per comparison, as above.
• Run both orders and count a win only when the judge picks the same output twice. Doubles the cost and turns position bias into a detectable disagreement instead of a silent skew.

Watch for length bias too. Judges tend to prefer longer answers, so if your change mostly made outputs longer, check whether the win rate survives a conciseness criterion in the rubric.

What it cannot tell you

Pairwise comparison ranks; it does not measure. A 90% win rate says the new version is better, not that either version is good. Both could be poor. Keep some absolute scoring, or task-specific checks, so you know where you stand and not merely which way you moved.