RLHF to GRPO: reward models and the KL leash

Reinforcement learning from human feedback (RLHF) exists because there is no loss function for a good answer. You can measure whether a model predicted the next token of a document, but not whether its reply was helpful, correct or honest. So RLHF first trains a reward model on human preferences, a network that scores answers, and then trains the model itself, the policy, to produce answers that score higher. Group Relative Policy Optimization (GRPO), introduced in the DeepSeekMath paper (arXiv:2402.03300) and used to train DeepSeek-R1, is a version of that second stage that many open reasoning models now use.

The film is one real run, not an illustration. The policy is Qwen2.5 0.5B Instruct, trained on a laptop in PyTorch. The prompt is a maths question with a checkable answer: Priya buys four boxes of six muffins and gives away seven, so the answer is 17, and the model is told to finish with a line that says "Answer: 17". Each training step samples eight answers at temperature 1 and scores them. The stand-in reward model is Llama 3.1 8B Instruct acting as a judge through OpenRouter: it sees the question and the answer, never the right number, and returns a score from 0 to 10. A second reward, the checker, is a regular expression that accepts only a final "Answer: 17" line.

In the first group the judge gave the one right answer an 8. It also gave "Answer: 3." an 8, a gibberish answer a 2, and its highest score, 9, to an answer that reached 17 but broke the requested format. Classic RLHF would now use PPO, which trains a second network, the value model, to predict what score to expect and pushes each answer by how far it beats that prediction. GRPO drops the value model. It compares each answer with the other answers to the same prompt: subtract the group's mean, 3.88, divide by its spread, 3.52, and the result is the advantage. The best a perfect value model could have predicted here was 4.06, the mean over 32 fresh answers, so eight siblings already give a baseline close to it, without a second network the size of the policy.

Then the policy took one gradient step, and the part people get wrong is where the step went. GRPO averages each answer's loss over its own tokens, so a six-token answer takes the whole of its push on six tokens while a 57-token answer spreads it thin. "Answer: 3." went from a 0.36% chance to 31% in that one step, and the one right answer, which also had a positive advantage, became thirteen times less likely. By step 3, seven of the eight answers were six tokens long; by step 5 all eight were "Answer: 3.", and the judge kept scoring it 8 or 9. Over 30 steps the mean judge score doubled, from 4.06 to 8.09, while the share of right answers fell from 6 in 32 to none, and the answers shrank from 129 tokens to 15. That is reward hacking: the policy found what the reward model rewards, not what it was meant to reward.

It gets worse once the answers are identical. With eight copies of "Answer: 3." scored 8 or 9, the spread is half a point, and dividing by it turns a one-point wobble into advantages of plus and minus one. The judge's wobble is real noise, not a small effect: in the run without a KL penalty it scored that exact answer 210 times at temperature 0, and gave it a 2 in 87 of them, an 8 in 58 and a 9 in 65. The same 32 answers scored twice came back with 13 scores changed. When a group's rewards are all equal, as at step 19 when all eight answers scored exactly 8, every advantage is zero and the step teaches nothing at all. Group size is the lever here: a bigger group gives a steadier mean and spread and more chances to disagree, which is why DeepSeekMath sampled 64 answers per question where this run used eight.

The KL penalty is meant to be the leash: each token pays a cost for drifting from a frozen copy of the starting model. At step 5 the frozen copy gave the token "3" a 19% chance where the policy gave 98%. But at the weight the DeepSeekMath paper used, 0.04, the penalty is tiny next to advantages of plus or minus one, and it is exactly zero on the first step, when the policy still equals its copy, so it cannot stop the first jump. A run with no penalty at all collapsed onto "Answer: 3." just as fast. What did work was changing the reward: with the checker in place of the judge, the same recipe raised the right answers from 6 in 32 to 28 in 32 in forty steps, and the answers kept their working at about 100 tokens.

The honest caveats. This is a tiny model on a single prompt with one gradient update per group and no clipping, which makes every effect faster and larger than in a production run over thousands of prompts, and a real reward model is trained on preference data rather than prompted. Llama 3.1 8B is a deliberately cheap judge; a stronger judge would be harder to fool, but not impossible, and published work on GRPO training reports the same failure modes at scale: reward hacking against learned reward models, and a length bias from the per-answer token average, which the Dr. GRPO paper (arXiv:2503.20783) removes. The judge scores came from one provider through OpenRouter and are not reproducible call for call, which is itself part of the lesson.