BRAT 训练方法解决多轮复用数据后训练崩溃问题
BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training
做 RL 后训练的人可以看看这篇：作者先解释了为什么同一批 rollout 多轮复用容易训崩，再给出 BRAT 这套更新方式，四次复用下不掉精度、种子间波动更小，比 PPO clipping 温和。
一篇arXiv论文提出 Bounded Residual-Aligned Training（BRAT），针对 LLM 强化学习后训练中同一批 rollout 被多次梯度复用导致的崩溃问题。研究测量了小规模 rollout 组上估计的更新与 KL 正则化目标在对同一 prompt 的 held-out rollout 上的对齐程度，发现从样本内到样本外对齐下降近一个数量级，且跨三个基座模型均如此，说明重复 pass 主要在过拟合采样的 rollout 组。
现有方法各有缺陷：不裁剪的更新会因策略漂移而崩溃，加负 baseline 虽避免崩溃但会强化所有采样响应，PPO 式 ratio clipping 会突然切断梯度。BRAT 给每个 token 分配随概率比线性递减、在信任域边界处归零的梯度权重，并将组内中心化奖励按最大幅值归一化，使 advantage 保持零均值且有界，无需额外超参数。
在四次复用 pass 的设置下，BRAT 避免了训练崩溃、限制了策略漂移，并在多数基准上取得更高平均准确率和更低的种子间方差。对需要摊薄生成成本、反复复用 rollout 数据的训练团队，这提供了一种比 PPO clipping 更平滑的更新方案。
BRAT: Bounded Residual-Aligned Training for Stable Multi-Pass LLM Post-Training
Reinforcement learning for large language model post-training often reuses each batch of sampled rollouts for several gradient passes to amortize the cost of generation. Under such multi-pass reuse, existing objectives fail in different ways: unclipped updates collapse as the policy drifts away from the sampling policy, adding a negative baseline avoids collapse but reinforces every sampled response, and PPO-style ratio clipping cuts off gradients abruptly. We first ask why reuse is fragile. Measuring how well an update estimated from a small rollout group aligns with the KL-regularized target on held-out rollouts of the same prompts, we find that this alignment drops by nearly an order of magnitude from in-sample to held-out rollouts across three base models, suggesting that repeated passes largely overfit the sampled group. Motivated by this finding, we propose Bounded Residual-Aligned Training (BRAT), motivated by a proximal update toward a bounded reweighting of the sampling policy. BRAT assigns each token a gradient weight that decreases linearly with its probability ratio and vanishes at a trust-region boundary, and normalizes group-centered rewards by their maximum magnitude, keeping advantages zero-mean and bounded without extra hyperparameters. With four reuse passes, BRAT avoids training collapse, limits policy drift, and achieves higher average accuracy with lower seed variance on most benchmarks.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：BRAT、PPO、RLHF、强化学习，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Xunlan Zhou, Yuzhe Zhao, Xuhong Huang, Feng Chen, Xuanlin Chen, Lang Feng, Bo An），经 AITOP 多维评分管线结构化提炼。