Eval set

Without an eval set, every change to a prompt or a retrieval pipeline is judged by trying a couple of inputs and forming an impression. That impression is unreliable, does not survive the week, and cannot tell a real improvement from noise in a non-deterministic system.

An eval set fixes the inputs so the only thing varying is your system.

What one looks like

Not much:

export const evalSet = [
  {
    id: 'refund-outside-window',
    input: 'I bought this 45 days ago and want a refund',
    expects: { policy: 'refund-window', mentionsDays: 30, escalates: false },
  },
  {
    id: 'refund-damaged',
    input: 'Arrived cracked, bought yesterday',
    expects: { policy: 'damaged-goods', escalates: true },
  },
]

Note what expects holds: checkable properties, not an exact expected string. Demanding an exact match against generated prose fails on harmless rewording and teaches you nothing. Assert the things that must be true.

Where the cases come from

The best eval sets are grown, not authored. Every case should come from somewhere real:

• Production failures. The single richest source. Something went wrong, so it becomes case number 41 and never regresses silently again.
• Support tickets and user reports, which is where the inputs you did not imagine live.
• Edge cases you deliberately reasoned about, clearly marked as synthetic so you know which parts of the set reflect reality.

Twenty cases drawn from real failures beat two hundred you invented at a desk.

Keeping it honest

• Freeze it. An eval set you edit whenever it disagrees with you measures nothing. Add cases; resist rewriting them to pass.
• Hold some back. If you tune against every case, you have fitted to the set. Keep a slice you look at rarely.
• Record the score, not just the pass. "31 of 40" tracks progress; "it works" does not.