研究如何用 Monitor 干预缓解 RL 训练中的 reward hacking
Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL
Anthropic 之外少见的 reward hacking 实证研究，开源了三个 Qwen3-4B 可钻空子的环境，还给出 monitor 设计的几条可落地结论，做 RL 训练的可以看看。
一篇 arXiv 研究开源了三个 Qwen3-4B 会出现 reward hacking 的环境：测试覆写型编程环境、谄媚型医疗聊天环境和幻觉型传记生成环境。实验发现模型能利用探针和 LLM judge 的系统性缺陷学会规避高准确率的 monitor。泄漏更多学习信号的 monitor 能抑制 reward hacking，但更容易被规避。研究团队将这些发现应用到医疗聊天和传记生成环境，部分干预下能把 reward hacking 率降到训练前水平。
Designing Effective Monitor-Based Interventions for Mitigating Reward Hacking During RL
Reinforcement learning (RL) rewards are notoriously difficult to design and control, often leading to the model learning unintended behaviors such as reward hacking. One potential solution is to monitor for reward hacking and penalize it when detected; however, training against a monitor could lead to evasive behavior, and our general understanding of how to apply monitors effectively during training is limited. To study how best to use monitors to mitigate reward hacking, we introduce and open source three realistic environments where Qwen3-4B reward hacks: a coding environment hackable via test overwriting, a medical chat environment hackable via sycophancy, and a biography generation environment hackable via hallucination. We first focus on the coding environment, where we find that: (1) models can learn to evade highly accurate monitors by exploiting systemic flaws in probes and LLM judges; (2) monitors that leak more learning signal during RL suppress reward hacking but are more often evaded; and (3) including easier problems in training can decrease reward hacking. We apply our findings to build better reward hacking monitors for the medical chat and biography generation environments that improve upon naive baselines to reduce reward hacking rates across seeds to before-training levels in some interventions. Our results demonstrate that our takeaways translate to new settings and that better monitor intervention designs are possible.