Salesforce 研究：从外部大模型蒸馏比 RL 更耗算力，合并领域专家可省 42% FLOPs
Salesforce 这篇论文算了笔账：蒸馏外部大模型的每个 token 都要跑一遍 teacher，反而不如直接 RL，先训领域专家再合并能省 42% 算力。
Salesforce 的新论文对比了 on-policy 蒸馏与 GRPO，以每 FLOP 的 held-out 准确率作为指标。结果显示，使用外部 post-trained teacher 时 GRPO 通常胜出，蒸馏还常导致回复长度膨胀。原因是蒸馏的每个训练 token 都需要一次 teacher 前向传播。替代方案是先用 GRPO 分别训练数学、代码等领域的专家，再蒸馏合并到一个学生模型，总 FLOPs 减少 42.0%，效果与联合 GRPO 相当。
New Salesforce paper finds that distilling from a big external teacher usually costs more compute than plain RL, while merging RL-trained domain experts saves 42%.
On-policy distillation lets a student learn from a teacher's per-token feedback on its own outputs, and many expect this dense signal to beat RL. But every training token also needs a teacher forward pass.
They compared distillation and GRPO by held-out accuracy per training FLOP, varying task difficulty, model sizes, domains, and training settings.
With an external post-trained teacher, GRPO generally won, and distillation often inflated response length. Distilling per-domain GRPO experts into 1 student matched joint GRPO on math plus code with 42.0% fewer total FLOPs.
To save compute, train per-domain experts with RL and distill them into 1 model, rather than distilling from a big external teacher.
