R2S：先修复教师打分器，再做智能体在线蒸馏
Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
训练 Agent 蒸馏的同学看过来，这篇指出教师打分短视才是瓶颈，R2S 修复后 12 个组合全胜，平均多拿 12.8 个成功点。
论文提出 Agentic on-policy distillation 的失败根源并非复合误差，而是 next-token 打分的短视：教师给每个 token 打分时看不到回合结局，导致拖延性安全动作未被惩罚。作者将其命名为 rollout-scorer gap。R2S 方法先从未训练学生中挖掘高分歧状态，运行教师接管并用共识或 LLM judge 等无奖励检查筛选状态，再微调教师打分器，全程无需任务奖励。在 3 个环境、2 个教师、4 个学生的 12 个组合中，R2S 全部胜过 3 个 SOTA 基线，平均领先其中最佳者 12.8 个成功点。
Capable Teacher, Myopic Scorer: Reward-Free Teacher Repair for Agentic On-Policy Distillation
Agentic on-policy distillation (OPD) trains a student agent on its own trajectories while a teacher scores every token. A natural explanation of where it goes wrong is compounding error: the student's mistakes carry its trajectories beyond the region where the teacher is reliable, so recent methods keep training inside that region, through episode-length curricula, teacher-guided turns, prefix replay, or teacher intervention where the two disagree. We show this explanation is incomplete. From the states where the student departs most from the teacher, the teacher's own takeovers still finish the task, yet the student trained on its token scores keeps running out of turns. We attribute this to the myopia of next-token scoring. The teacher rates each token without seeing where the episode ends, so its scores let through the safe, delaying moves that outcome-based RL would punish with zero reward, a penalty agentic OPD lacks. We call this difference between what the teacher does and what it scores the rollout-scorer gap. We propose Rollout-to-Scorer Distillation (R2S), which repairs the scorer before distillation by mining high-disagreement states from the untrained student, running teacher takeovers from them, keeping the states where enough takeovers pass a reward-free check such as consensus or an LLM judge, and fine-tuning the teacher to give its takeover-informed scores on every assistant token from the divergence turn onward, without the takeover in context. The repaired teacher is frozen as the sole scorer for standard OPD, requiring no task reward during repair or training. Across three environments, two teachers and four students, R2S outperforms three state-of-the-art agentic-OPD methods in all twelve teacher-student-environment cells, by 12.8 success points on average over the best of them in each cell. Project page: https://tree-shu-zhao.github.io/r2s/.
