Stanford 团队提出 Speculative Self-Distillation，减少 38–69% 训练 token 并缓解遗忘
斯坦福组这篇 NeurIPS 2026 论文思路很巧：老师只在学生写偏时才接管，训练 token 直接省 38–69%，遗忘也从掉 4.5 分降到 1.4 分。做企业文档微调、怕把基座模型练废的开发者可以认真看看这套自蒸馏流程。
Stanford AILab 的 Shayan Talaei 等人发布 NeurIPS 2026 论文 Speculative Self-Distillation（SSD），用于在不遗忘旧知识的前提下向 LLM 注入新文档。核心机制是同一模型的学生与教师（教师持有额外信息）共同生成 rollout，教师像家教一样只在学生偏移过大时接管，从而保持训练数据尽可能接近学生自身的 on-policy 分布。
实验数据上，SSD 比 on-policy 自蒸馏少用 38–69% 的训练 token 且准确率持平；Science QA 任务中约 130 分钟达到 on-policy 约 414 分钟的准确率。在遗忘压力测试中，SSD 掉分仅 1.4 分，而 on-policy 掉 4.5 分，说明学生在训练新知识时旧能力保留更好。
对需要在私有文档上做持续微调的团队来说，SSD 提供了训练成本与灾难性遗忘之间的折中方案，尤其适合算力预算有限但又要保住基座模型通用能力的场景。
How do you teach an LLM a new document without it forgetting what it already knows? 📚 Check out Speculative Self-Distillation, a #NeurIPS2026 6 paper from @talaei_shayan , @chewing_a_gum , @ckodser and @Azaliamirh 's group. The student writes, and the teacher steps in only where it matters. Shayan Talaei @talaei_shayan Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation. In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track. 📉 38–69% fewer training tokens than on-policy, matching its accuracy ⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A) 🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：Speculative Self-Distillation、NeurIPS 2026、微调、自蒸馏，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「Stanford AI Lab」（@StanfordAILab），经 AITOP 多维评分管线结构化提炼。
