研究量化训练-推理失配对 On-Policy 蒸馏的容忍度
How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
做 LLM 训练系统的工程师应该看这篇：它量化了 OPD 在多大失配下不掉质量，FP8 加异步 rollout 能比同步 BF16 快 1.62 倍达到同样效果，还附了能直接复用的校准脚本和代码。
一篇 arXiv 论文系统测量了 On-policy distillation（OPD）对训练-推理失配的容忍度，覆盖三个 student-teacher 对的量化实验，并在其中一对上额外测试权重陈旧和注入噪声。结论是 OPD 能容忍相当大的失配而无明显训练质量损失，超过阈值后才退化，呈现出可校准的失配预算。
机制层面，失配的质量代价和量化引入的失配都依赖具体模型，某些层对精度下降特别敏感；在对照实验中，OPD 在会让 RL 方法失稳的 rollout 配置下仍保持稳定。权重噪声还会改变生成时到达的 prefix，产生逐 token 修正无法解决的错误。
实践上，作者提出一个轻量 admission check，无需每次完整训练即可筛选拟定的 rollout 配置。FP8 量化在保持训练质量的同时提供至少 1.93 倍于 BF16 的 KV-cache token 容量，FP8 结合异步 rollout 比同步 BF16 训练提前 1.62 倍达到同等质量。代码开源于 GitHub。
How Much Training-Inference Mismatch Can On-Policy Distillation Tolerate?
On-policy distillation (OPD) trains a student language model on its own responses, guided by a fixed teacher. Generating these responses is expensive, so systems use quantized inference and asynchronous rollouts to reduce the cost. Both, however, introduce training-inference mismatch between the rollout engine and the model being trained. Such mismatch can destabilize reinforcement learning (RL), but its effects on OPD remain unclear. This paper investigates how much mismatch OPD tolerates, where its effects arise, and how training systems can use this tolerance. Measuring all mismatch sources on one scale, we examine quantization across three student-teacher pairs, with additional tests of weight staleness and injected noise on one pair. OPD tolerates substantial mismatch without meaningful loss of training quality but degrades at higher levels, revealing an empirical mismatch budget. Both the quality cost of mismatch and the mismatch introduced by quantization depend on the model, with some layers particularly sensitive to reduced precision. In controlled comparisons, OPD remains stable under rollout configurations that destabilize RL methods. Under weight noise, mismatch also shifts the prefixes reached during generation, introducing errors that per-token corrections cannot address. To put these findings into practice, we introduce a lightweight admission check that screens rollout configurations against a calibrated budget without requiring a full training run for each candidate. Model-specific FP8 recipes preserve training quality while providing capacity for at least $1.93\times$ as many KV-cache tokens as BF16. Combining FP8 with asynchronous rollouts reaches the same training quality $1.62\times$ sooner than synchronous BF16 training. Code is available at https://github.com/benyucong/train-inference-mismatch-opd
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：On-policy distillation、FP8、量化、RL，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Cong Yu, Ziting Wang, Songlin Jiang, Sitong Zhang, Chengliang Chai, Youhe Jiang, Eiko Yoneki, Tuo Shi），经 AITOP 多维评分管线结构化提炼。