WEBVTT

1
00:00:00.700 --> 00:00:07.565
The first appendix isolates two GRPO evaluation choices that matter after the main mechanism is understood.

2
00:00:12.059 --> 00:00:22.337
Length normalization changes gradient weight. A fourteen-hundred-token correct answer and a one-hundred-eighty-token correct answer may receive the same reward but contribute very different numbers of token terms.

3
00:00:26.831 --> 00:00:34.831
Sequence averaging, token averaging, overlong penalties, and group normalization therefore change the effective objective.

4
00:00:39.326 --> 00:00:48.329
Pass-at-one measures reliability from one sample. Pass-at-k asks whether any of k samples succeeds and exposes remaining search diversity.

5
00:00:52.824 --> 00:01:00.100
A model can improve pass-at-one while collapsing onto one strategy, so report both metrics with the sampling configuration.

6
00:01:01.500 --> 00:01:09.590
The systems problem is storage. A large vocabulary creates one teacher logit for every token at every trajectory position.

7
00:01:13.011 --> 00:01:22.055
Top-k compression stores the most likely teacher logits and approximates the remaining tail. This reduces memory and bandwidth but changes the target distribution.

8
00:01:25.476 --> 00:01:32.200
Measure divergence from full logits on a validation sample, and count teacher-forward cost when comparing efficiency.

9
00:01:33.600 --> 00:01:40.143
The reported graphs belong in an appendix because they come from different models, datasets, and experimental budgets.

10
00:01:42.001 --> 00:01:47.822
The OPSD study reports its largest displayed aggregate gain at the smallest Qwen3 model.

11
00:01:49.680 --> 00:02:02.137
At one point seven billion parameters, the displayed base score is thirty-seven point one, GRPO is thirty-seven point seven, and OPSD is forty-three point four. The gaps are smaller at four and eight billion parameters.

12
00:02:03.994 --> 00:02:09.133
That is a hypothesis for replication, not a universal model-size law.

13
00:02:10.991 --> 00:02:21.720
The SDFT study emphasizes knowledge acquisition and retention. Its displayed strict aggregate moves from eighty to eighty-nine, while its out-of-distribution aggregate moves from eighty to ninety-eight.

14
00:02:23.578 --> 00:02:35.713
The SDPO study reports forty-one point two for GRPO and forty-eight point eight for SDPO on its code benchmark. Their bars answer different questions and should not be read as a controlled head-to-head comparison.

15
00:02:37.571 --> 00:02:50.208
SRPO means Sample-Routed Policy Optimization. It sends verified successes toward a GRPO-style outcome update and failures with explanations toward an SDPO-style corrective target.

16
00:02:52.066 --> 00:03:03.839
The useful principle is signal routing. A successful rollout already shows what worked and can support sparse selection. A failed rollout with diagnostics may reveal how to recover and can support dense correction.

17
00:03:05.697 --> 00:03:11.067
The router preserves that asymmetry instead of forcing every sample through one uniform loss.