BranchOPD：分支式在策略蒸馏提升数学与智能体任务表现
Branching On-Policy Distillation
训练小模型的同学可以看看这篇：用分支采样蒸馏，token 成本降三成，数学和 agent 任务成绩都比普通蒸馏高不少。
论文提出 BranchOPD（Branching On-Policy Distillation），在学生模型生成轨迹上利用学生不确定性与师生分歧选取关键决策点，从共享前缀重采样分支并施加 token 级教师监督。共享前缀复用使 ALFWorld 上学生生成 token 成本降低约 19%，WebShop 上降低约 31%。该方法还引入协议边界修复（PBR）机制，抑制重复生成与无效输出。实验显示 BranchOPD 在全部评测基准上超过 Vanilla OPD 与 Multi-Rollout OPD（n=7），数学 avg@16 平均准确率比 Vanilla OPD 高 18.55 个百分点，ALFWorld ID/OOD 平均成功率提升 13.8 点。
Branching On-Policy Distillation
On-policy distillation (OPD) provides teacher supervision on student-generated trajectories, but a single trajectory realizes only one continuation at each decision, limiting supervision coverage under a finite sampling budget. We propose BranchOPD (Branching On-Policy Distillation), which selects informative decision points using student uncertainty and teacher-student disagreement, resamples student-policy continuations from shared prefixes, and applies token-level teacher supervision to both main trajectories and branches. BranchOPD operates at token-level decisions in mathematics and within-turn decisions in agent tasks. Shared-prefix reuse reduces student-generated token costs by approximately 19% on ALFWorld and 31% on WebShop relative to seven independent trajectories. We also introduce protocol-boundary repair (PBR), which uses targeted supervision under protocol constraints to mitigate repetition and wasted generation. BranchOPD outperforms Vanilla OPD and Multi-Rollout OPD (n=7) across all evaluated benchmarks, with gains over Vanilla OPD of 18.55 percentage points in mean mathematics accuracy over 16 sampled answers (avg@16) and 13.8 points in mean ALFWorld ID/OOD success. Mechanistic analyses and selection ablations show that branching at uncertain decisions expands subsequent behavior coverage and outperforms random selection and turn-initial branching.
