Agentic RL 研究提出 targeted guidance 自蒸馏方法稳定训练过程
Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning
做 RL 训练的朋友可以看看这篇：用判分模型的文本反馈做自蒸馏，专门救那些普通 RL 优化不动的奖励，还讲了怎么防止训练震荡。
这篇 arXiv 论文提出 targeted guidance 框架，通过 on-policy 自蒸馏目标从逐轮文本反馈中学习，用于优化难以直接优化的奖励。方法将 targeted 自蒸馏与 per-criteria RL 奖励结合，并把反馈限制在策略失败的具体轮次。实验在 agentic 多轮软件工程环境中进行，结果显示训练全程保持稳定的熵值并减少振荡行为。论文还分析了实现稳定训练的关键设计选择。
Stabilizing On-Policy Targeted Guidance in Agentic Reinforcement Learning
Naive on-policy RL is often bottlenecked by the capabilities of the base model --- insufficient coverage can result in an inability to learn from certain rewards. At the same time, rewards in modern post-training systems often come from rich textual sources, e.g. judge models with a rubric of criteria. Can we utilize such structures to provide a rich directional signal that enables improving on otherwise difficult to optimize rewards? In this work, we examine a framework we refer to as targeted guidance that learns from per-turn textual feedback via an on-policy self-distillation objective. Our focus is not only on directly improving hard-to-optimize behaviors, but on how to do so throughout the training process without degrading baseline RL performance. Over a range of settings across an agentic multi-turn software engineering environment, we find success by combining targeted self-distillation with a per-criteria RL reward, and by limiting feedback to turns where the policy specifically fails. We further examine the design choices that lead to stable entropy and reduce oscillatory behavior throughout training.
