Rubric

"Rate this answer's quality from 1 to 10" is a bad instruction. The model has no idea what separates a 6 from a 7, so it will not separate them consistently, and neither would two humans. The scores look like measurement and are closer to mood.

A rubric removes the guesswork by naming the criteria and defining each level.

Named criteria beat one score

Split quality into things that can be judged independently, and score each:

const rubric = {
  grounded: {
    2: 'Every claim is supported by the provided sources.',
    1: 'Mostly supported, one unsupported claim.',
    0: 'Contains claims absent from the sources.',
  },
  answersQuestion: {
    2: 'Directly answers what was asked.',
    1: 'Related but sidesteps part of the question.',
    0: 'Does not answer it.',
  },
  concise: {
    2: 'No filler or restatement of the question.',
    1: 'Some padding.',
    0: 'Substantially padded.',
  },
}

Three small judgements with defined levels are far more stable than one vague ten-point scale, and they tell you *what* got worse rather than just that something did.

Why levels matter more than criteria

Most rubrics name criteria and then leave the scale undefined, which recovers the original problem one layer down. The work is in the level descriptions: a 0, 1, 2 scale where each number has a sentence beats a 1 to 10 scale where none of them do. Narrow scales are easier to apply consistently, and consistency is the entire point.

Practical notes

• Ask for the reason before the score. Requiring a short justification first makes the score follow from something, and gives you a trail when a judgement looks wrong.
• Return it as structured output, so scores are parseable and aggregatable rather than buried in prose.
• Spot-check the judge against yourself. Grade fifteen outputs by hand and compare. If you disagree with the judge often, the rubric is underspecified, not the model.