Goodharting

Goodhart's law: when a measure becomes a target, it ceases to be a good measure. In LLM systems this arrives faster than people expect, because iteration is cheap and the metric is usually a proxy sitting some distance from what you care about.

What it looks like in practice

• Your judge rewards citing sources, so the system cites sources for everything, including claims the sources do not support. Groundedness score up. Groundedness down.
• Judges mildly prefer longer answers. You tune toward the score and ship answers that are padded rather than better.
• You add every failure to the eval set and tune until all 40 pass. The system now handles those 40 inputs and generalises worse than before.
• The rubric has a "confident tone" criterion, so hedging disappears, including on the answers that genuinely should hedge.

In every case the number improved and the product got worse. That is the failure mode: the metric stops being evidence and starts being camouflage.

Why LLM systems are unusually exposed

A prompt can be reshaped in seconds, so hundreds of small adjustments can be made against a fixed score in an afternoon. Each is individually reasonable. Together they fit the system to the scorer. When the scorer is itself a model, you are also fitting to *its* quirks, which are not your users' preferences.

Defences

• Hold out cases you do not tune against. The single most effective measure. If held-out performance diverges from tuned performance, you are goodharting, and you can see it.
• Keep more than one metric, ideally ones that trade off. Conciseness and completeness pulling against each other makes gaming visible, because you cannot inflate both.
• Read outputs. Regularly, by hand. Scores tell you something moved; only reading tells you whether it got better. A number nobody sanity-checks is where this failure hides.
• Change the eval set over time, adding genuinely new cases from production rather than variations on ones you already pass.
• Prefer relative judgements. Pairwise comparison against the previous version is harder to game than an absolute score with a fixed ceiling.