LGPO：去掉重要性比率，也能稳定训练 LLM 强化学习
Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
做 RL 后训练的值得看：LGPO 用似然门控替掉 GRPO 里重复算 old-policy 似然那步，训练时间少约 18%，全 on-policy 时 GRPO 会崩它还稳着。省显存又省算力，自己跑 RL 的团队可以直接对照论文改实现。
一篇新论文提出 Likelihood-Gated Policy Optimization（LGPO），用基于似然的门控机制实现软信任域约束，省去了 old-policy 似然的重复计算。GRPO 等方法需要同时计算 old-policy 和 current-policy 似然来构造重要性采样比率，带来不小开销。实验显示，去掉重要性采样修正项不损害训练稳定性，而去掉信任域机制会导致训练崩溃；比率裁剪在全 on-policy 训练中还会失效，因为重要性比率始终为 1，约束从未激活。在 GRPO 本身稳定的标准设定下，LGPO 达到相当的稳定性和峰值性能，训练时间平均减少约 18%。在全 on-policy 训练中 GRPO 会失效，LGPO 仍保持稳定。
这一结果指向一个重要判断：训练稳定性主要来自信任域约束而非重要性比率本身。对做 LLM 后训练（RL post-training）的团队来说，省掉 old-policy 似然意味着显存和计算开销直接下降，全 on-policy 场景也因此可行。相比 GRPO 及其各种裁剪变体，LGPO 在不同训练模式下表现更稳健。
Is the Importance Ratio Necessary for Stable Reinforcement Learning in LLMs?
Reinforcement learning (RL) has become central to post-training large language models (LLMs). However, popular RL methods like GRPO incur non-negligible overhead by computing both old-policy and current-policy likelihoods to form importance sampling ratios. In this work, we propose Likelihood-Gated Policy Optimization (LGPO), which enforces a soft trust region constraint via likelihood-based gating, eliminating the need to compute old-policy likelihoods. Empirically, we show that removing the importance sampling correction term does not harm training stability, whereas removing the trust region mechanism can lead to collapse. Moreover, ratio-based clipping can fail in fully on-policy training: the importance ratio stays at 1, so the ratio-based trust region constraint never activates. Under standard training settings where GRPO is stable, LGPO achieves comparable training stability and peak performance while reducing training time by ~18% on average relative to a standard implementation that recomputes old-policy likelihoods. In fully on-policy training, where GRPO fails, LGPO remains stable, enabling more efficient and robust LLM RL post-training across training regimes.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：LGPO、GRPO、强化学习、LLM后训练，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Shuibai Zhang, Junhyuck Kim, Gyeongman Kim, Jaewoong Cho），经 AITOP 多维评分管线结构化提炼。