Waypoint-OPD：用门控在策略蒸馏迁移 RL 引起的策略变化
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
一篇蒸馏论文：用教师模型 RL 中途检查点的方向一致性做门控，Qwen3 系列学生六个设置全拿下最高分，成本只多一次打分。
论文提出 Waypoint-guided On-Policy Distillation（Waypoint-OPD），在端点策略偏移之外引入同一 RL 运行的中间教师检查点，比较早期与晚期策略偏移的方向一致性。当两个方向的偏移相反时，门控机制会抑制端点偏移的迁移。方法只需额外一次冻结教师打分。以 RL 训练的 Qwen3-4B 为教师，在 Qwen3-8B、Qwen3-14B、Qwen3-30B-A3B 学生模型上的数学与代码推理评测中，六个领域-学生设置全部取得对比蒸馏方法中的最高平均分。Qwen3-8B 学生的消融实验显示，移除或打乱门控会降低数学推理成绩。
Waypoint-OPD: Transferring RL-Induced Policy Shifts via Gated On-Policy Distillation
Policy-shift distillation reuses the difference between a teacher's log probabilities before and after reinforcement learning (RL) as supervision on student-generated trajectories. However, this endpoint policy shift cannot reveal whether local action preferences evolved consistently during teacher training. The same endpoint shift can result from early and late shifts that reinforce or partially cancel each other. We investigate whether agreement between early and late teacher policy shifts can guide selective transfer. We introduce Waypoint-guided On-Policy Distillation (Waypoint-OPD), which uses an intermediate teacher checkpoint from the same RL run to compare early and late shifts in relative action preferences at student-visited prefixes. We gate the endpoint shift according to the directional agreement between the early and late shifts, suppressing it when they point in opposing directions. Compared with endpoint-only transfer, our method requires only one additional frozen-teacher scoring pass. Using RL-trained Qwen3-4B teachers, we evaluate our method with Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B as student models in mathematical and code reasoning. In all six domain--student settings, Waypoint-OPD achieves the highest domain-average score among the compared distillation methods. Ablations with the Qwen3-8B student show that removing or shuffling gates reduces mathematical reasoning performance. These results support the use of directional agreement between early and late teacher policy shifts to weight the endpoint shift during student training.
