AOPD 提出按优势加权教师动作，改进在线策略蒸馏
When to Follow the Teacher: Advantage-Based On-Policy Distillation
一篇蒸馏方法论文：AOPD 不让学生照抄老师每一步，只学真正有帮助的动作，实验覆盖数学、编程和 Medqa，效果超过蒸馏和 RL 基线。
论文针对在线策略蒸馏（OPD）中"每一步都跟随教师"的假设，提出 Advantage-Based On-Policy Distillation（AOPD）。该方法用反事实优势给教师动作加权，只强化能提升学生预期结果的动作。优势估计只需标准学生 rollout 和终止奖励，不需要额外 rollout 或奖励评估。实验覆盖数学、编程和 Medqa，AOPD 在外部教师和自蒸馏两种设置下都超过蒸馏与 RL 基线。
When to Follow the Teacher: Advantage-Based On-Policy Distillation
Online Policy Distillation (OPD) often outperforms offline methods such as Supervised Fine-Tuning (SFT) by training on trajectories sampled directly from the student. However, standard OPD trains the student to follow the teacher at every state of the trajectory, assuming that teacher guidance is always beneficial. We instead distinguish between what the teacher prefers and what actually helps the current student succeed. We introduce Advantage-Based On-Policy Distillation (AOPD), which weights teacher actions by their counterfactual advantage under the student. Teacher actions that improve the student's expected outcome are reinforced, while those that do not are downweighted. Directly computing these advantages requires trying alternative teacher actions at each step and running new rollouts from each. Instead, we show that the AOPD objective can be estimated using only standard student rollouts and their terminal rewards. Our estimator uses teacher importance weights on the student-sampled actions to provide unbiased advantage-weighted supervision, without any additional rollouts or reward evaluations. This allows AOPD to focus teacher supervision on actions that help the current student while preserving the rollout efficiency of standard OPD. Experiments on math, coding, and Medqa show gains over distillation and RL baselines in both external-teacher and self-distillation settings.
