arXiv 论文解析后训练策略优化的覆盖率坍缩与参考重置机制
A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
做 RLHF 或 GRPO 训练的工程师应该看看这篇。它用理论解释了一个常见现象：为什么训练到后期最优响应概率会掉到接近零，以及为什么定期重置 KL 参考点能救回来。连 Kimi 用的训练算法都被纳入了收敛证明。
arXiv 上发布一篇后训练策略优化理论研究，提出 coverage-preserving（保持覆盖）算法框架，用基础模型的响应覆盖率刻画样本与梯度步复杂度。论文指出 REINFORCE——PPO 和 GRPO 的基础——会让最优响应的概率趋近于零，即使奖励仍在提升；这种覆盖率坍缩在梯度精确计算和 KL 正则化下依然发生，收敛可能需要指数级步数。
针对这一问题，作者提出 Approximate Policy Mirror Descent（APMD）元算法，通过反复把 KL 参考点重置到最近策略来缓解覆盖率坍缩，将优化分解为一系列更易求解的 KL 正则化 RL 子问题。基于预训练特征上的 softmax 线性策略，论文给出首个全局收敛保证，证明带显式 KL 正则与参考重置的 PPO、GRPO 变体以及隐式实现这些机制的平方损失算法（包括训练 Kimi 模型所用方法）都是 coverage-preserving 的。
这项工作把正则化强度、clipping 和目标函数选择与后训练的采样成本、优化成本直接关联起来。对做 RLHF/RLVR 的工程师来说，它解释了参考点重置为何在实践中常见且必要，也为在 GRPO 类算法中调整 KL 项提供了理论依据。
A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets
Policy optimization is central to language model post-training, yet principled guidance remains limited. We revisit its theoretical foundations to identify algorithms that learn near-optimal policies using total samples and gradient steps governed by the base model's response coverage, a measure of how readily it generates optimal responses. We call such algorithms coverage-preserving. This guarantee captures the goal of amplifying capabilities accessible through sampling from the base model. (1) Coverage collapse. REINFORCE, the basis for PPO and GRPO, can drive the probability of optimal responses to near zero, even as reward improves. This coverage collapse stalls optimization, and convergence can require exponentially many steps despite good initial coverage. The failure arises even with exact gradients and persists under KL regularization to the base model. (2) Reference resets. We show that repeatedly resetting the KL reference to the most recent policy mitigates coverage collapse. We formalize this intervention in Approximate Policy Mirror Descent (APMD), a meta-algorithm that approximately solves a sequence of KL-regularized RL subproblems. APMD enables stronger regularization, making each subproblem easier to solve, while reference resets allow convergence to global optimality. (3) Coverage-preserving algorithms. For softmax linear policies over pretrained features, we use APMD to establish the first global convergence guarantees showing that widely used families of post-training algorithms are coverage-preserving. These include stylized PPO and GRPO variants with explicit KL regularization and reference resets, and squared-loss algorithms that implement them implicitly, including those used to train the Kimi models. These results clarify how regularization, clipping, and objective choice affect the sampling and optimization costs of post-training.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：RLHF、GRPO、PPO、Kimi，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Audrey Huang, Nan Jiang, Sham M. Kakade, Sadhika Malladi, Akshay Krishnamurthy, Dylan J. Foster），经 AITOP 多维评分管线结构化提炼。
