研究对比 on-policy 蒸馏与 GRPO 的算力效率
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
这篇论文回答了一个很实际的问题：蒸馏到底比强化学习省多少算力。结论有点反直觉——教师必须选对，自己的 GRPO checkpoint 当教师时能省 50 倍 FLOPs，做训练的人值得看看细节。
一项系统研究比较了 on-policy distillation（OPD）与 GRPO 在训练 FLOPs 与 held-out 准确率之间的表现。结果显示，使用外部后训练教师时，OPD 在同设置下通常不如 GRPO，还常出现回复变长的问题。但当教师换成学生自己的 GRPO checkpoint 时，在数学任务上只需约 1/50 的蒸馏 FLOPs 就能恢复同等准确率。多教师 OPD 能把各领域 GRPO 专家合并进单个学生模型，蒸馏成本仅为专家训练的 5.2% 到 8.0%，两领域混合下总 FLOPs 比 joint GRPO 少 42.0%。
Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO
On-policy distillation (OPD) trains a student on its own rollouts under a frozen teacher's per-token supervision. Does this dense signal reduce the training FLOPs needed to reach a target accuracy compared with reinforcement learning? We systematically compare vanilla reverse-KL OPD with Group Relative Policy Optimization (GRPO) across task difficulty, student and teacher size, domain, and optimization choices, measuring held-out accuracy against training FLOPs. With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation. Yet a different teacher changes the picture: when the teacher is the student's own GRPO checkpoint, OPD recovers that checkpoint's accuracy on math with $50\times$ fewer distillation FLOPs than GRPO needs to reach the same accuracy. Following the same teacher setting, we consistently observe this high efficiency with a larger student, on code, and in a second model family. On a single domain, however, the GRPO checkpoint is already the desired model, so distilling it adds to total compute. Across domains, separate GRPO checkpoints hold different capabilities, and no single checkpoint serves every domain. Multi-teacher OPD (DeepSeek-AI, 2026) consolidates per-domain GRPO experts into a single student that matches each expert's accuracy, with the distillation stage costing only 5.2% (on 2 domains) and 8.0% (on 4 domains) of the specialist training FLOPs. Counting specialist training and distillation together, this pipeline reaches joint GRPO's performance on the 2-domain mix with 42.0% fewer total FLOPs.
