WEBVTT

NOTE AI Engineering Visualised: https://understandingdata.com/ai-engineering-visualised/rlhf-grpo/

1
00:00:00.204 --> 00:00:04.731
You cannot write down what a good answer is as a loss function.

2
00:00:04.731 --> 00:00:11.844
So RLHF trains a reward model to score answers, then pushes the model towards whatever scores high.

3
00:00:11.844 --> 00:00:14.106
Here is one real step of that.

4
00:00:14.106 --> 00:00:23.454
A small model answers one muffin question eight times, and only one answer ends the way we asked, with the right number: 17.

5
00:00:23.454 --> 00:00:29.517
A judge model stands in for the reward model and stamps a score on each.

6
00:00:29.517 --> 00:00:31.791
The right answer gets an 8.

7
00:00:31.791 --> 00:00:33.475
So does "Answer: 3."

8
00:00:33.475 --> 00:00:43.449
Classic RLHF uses PPO, which trains a second network, a value model, to predict the score to expect: about 4 here, at best.

9
00:00:43.449 --> 00:00:47.747
Each answer is pushed by how far it beats that guess.

10
00:00:47.747 --> 00:00:54.358
GRPO throws the value model away, and a second network's worth of memory with it.

11
00:00:54.358 --> 00:01:04.070
It compares each answer with its seven siblings instead: subtract the group's mean, 3.9, and divide by its spread, 3.5.

12
00:01:04.070 --> 00:01:06.110
The two 8s get plus 1.17.

13
00:01:06.110 --> 00:01:08.112
Then one gradient step.

14
00:01:08.112 --> 00:01:20.562
"Answer: 3." jumps from a 0.4 percent chance to 31 percent, because GRPO shares each push across an answer's tokens, and this one has only six.

15
00:01:20.562 --> 00:01:25.438
The one right answer becomes thirteen times less likely.

16
00:01:25.438 --> 00:01:32.297
Four steps later, all eight answers say 3, and the judge still gives them 8s and 9s.

17
00:01:32.297 --> 00:01:37.359
The reward has more than doubled, and not one answer is right.

18
00:01:37.359 --> 00:01:39.237
That is reward hacking.

19
00:01:39.237 --> 00:01:48.220
Worse, identical answers still get plus and minus one, because dividing by a spread of half a point turns the judge's noise into the signal.

20
00:01:48.220 --> 00:01:57.436
The KL penalty is the leash against this: token by token, it compares the policy with a frozen copy of itself.

21
00:01:57.436 --> 00:02:02.965
The copy gave "3" a 19 percent chance; the policy now gives it 98.

22
00:02:02.965 --> 00:02:09.835
Yet at DeepSeek's weight, 0.04, a run with no leash at all collapsed just as fast.

23
00:02:09.835 --> 00:02:22.213
Swap the judge for a checker that only accepts 17, and the same training climbs from 19 percent right to 88 percent in forty steps, and the answers keep their working.

24
00:02:22.213 --> 00:02:33.637
So reward what you can verify, watch KL and answer length rather than reward alone, and keep each group hard enough to disagree, because eight equal scores teach nothing.