PACT 论文：对齐 critic 与策略，提升 LLM 强化学习训练效果

基于这一理论，作者指出 On-Policy Distillation 的教师模型相当于隐式 critic，RLOO 信号在期望意义上与 token 级 credit 一致。由此提出 PACT 训练方法，采用 Actor-then-Critic 更新顺序并对 critic 训练施加重要性采样校正。

在四个数学推理基准上平均准确率 72.87%，比 GRPO 和 PPO 分别高 8.

80 和 13.16 个百分点，SWE-bench Verified 通过率 67.

4%。

PACT: From Credit Assignment to Critic Alignment

Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：PACT、GRPO、PPO、SWE-bench Verified，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan），经 AITOP 多维评分管线结构化提炼。截至目前，全网已有 2 家独立信源对该事件进行交叉印证与跟进报道（含 elvis 等），属于近期高关注度行业信号。