10 种自蒸馏智能体强化学习方法实测：无一稳定超越 GRPO
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
浙大团队烧了 1.2 万 GPU 小时把 10 种智能体训练方法拉到同一环境里跑了一遍，结论是 GRPO 依然难被打败，做 Agent RL 的值得看看.
这项研究将 10 种代表性自蒸馏方法与 GRPO 在 ALFWorld、ScienceWorld、Search-QA 和 WebShop 四个环境上直接对比，覆盖 3 种模型规模。整个研究共进行 132 次训练，记录 H800 GPU 时间达 12,124.4 小时，并对 11 个训练方法用 8 个 rollout 种子重复评估。结果显示没有任何方法在所有设置下稳定超过 GRPO，部分方法最终成绩好但训练过程不稳定，训练成本增加也不必然带来性能提升。
More Methods, Better Agents? A Reality Check on Self-Distillation in Agentic Reinforcement Learning
Self-distillation in agentic reinforcement learning has produced a growing number of methods. However, existing evaluations often reuse published baseline scores under differing experimental conditions, provide few direct comparisons among contemporaneous methods, and give limited attention to training stability and computational cost. These limitations obscure the source of reported gains and the practical strengths and weaknesses of competing methods. Costly agent training further motivates a common basis for comparison. To address these gaps, we compare ten representative methods against GRPO on ALFWorld, ScienceWorld, Search-QA, and WebShop across three model sizes. Our study comprises 132 training runs totaling 12,124.4 recorded H800 GPU-hours, with repeated evaluations over eight rollout seeds for the pretrained backbones and all eleven trained methods. Beyond final task performance, we examine training progress, stability, and measured computational cost. Our evaluation yields three key findings. First, gains over GRPO remain limited in scope, with no method consistently outperforming it across settings. Second, strong final performance can mask unstable training. Third, additional training cost does not consistently translate into better performance. We establish this benchmark to support fair, comprehensive comparison and guide future research toward reliable and cost-effective improvements in agent training. Project page: https://aim-uofa.github.io/Self-Distillation-AgentRL/