一种新方法稳定大语言模型强化学习
Score Centering Stabilizes Off-policy Reinforcement Learning
这是篇关于如何解决大语言模型强化学习不稳定问题的论文，作者提出了一个叫“分数中心化”的新方法，在实验中效果不错，对做相关研究的人可能有参考价值。
这篇论文提出了一种叫“分数中心化”的新方法，用于解决大语言模型在训练和推理阶段不一致时强化学习不稳定的问题。作者发现这种不稳定主要由训练和推理引擎之间的偏差积累导致，他们推导出一个加法修正项来抵消这种偏差。在参数规模从0.6B到30B的模型上测试，该方法单独使用就能达到或超过基于重要性采样的量化方法，且在时序实验中与重要性采样的组合效果更好。
Score Centering Stabilizes Off-policy Reinforcement Learning
Reinforcement learning (RL) of large language models is notoriously sensitive to small differences between training and inference engines, often referred to as the training-inference mismatch (TIM). However, completely eliminating TIM is impractical, as it would come at a major cost to rollout efficiency. In this paper, we show that the instability of RL under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates with every training step. We derive an additive "score centering" correction term that stabilizes RL under TIM by canceling drift. When training models from 0.6B to 30B parameters, score centering alone matches or outperforms methods based on importance sampling under quantization, with the gap growing as the mismatch becomes more severe. Because the correction is additive, score centering also composes with importance sampling -- their composition outperforms pure importance-sampling baselines in our staleness experiments.
