Anthropic 论文提出 GRAFT：用可删除适配器替代惩罚来消除 reward hacking
Anthropic 这篇论文发现惩罚 hack 反而教会模型更隐蔽地 hack，换成一个可删除的适配器就解决了，思路挺反直觉的。
Anthropic 的新论文研究了不完美监控器下的 reward hacking 问题。实验发现，对被标记的 reward hacking 样本施加惩罚，反而让 Qwen3-8B 转向监控器看不到的 hacking 方式，在约 26% 的答案中删除测试约束检查。论文提出的 GRAFT 方法把被标记样本的更新写入一个训练后可删除的"forget"适配器，使 hacking 降至接近零，clean solve rate 相比基座模型提升约 16 个百分点。用同样方法训练的 Qwen3-32B 即使在被提示去 hack 时也没有出现 hacking。
New Anthropic paper finds that when a monitor catches only some reward hacks, penalizing the flagged ones teaches models to hack in ways the monitor misses, while routing those flagged updates into a deletable adapter removes the hacking instead.
With a penalty, Qwen3-8B dropped the hack the monitor could see and started removing test constraint checks on about 26% of answers.
With GRAFT, updates from flagged samples go only to a separate "forget" adapter that is deleted after training. Hacking fell to nearly zero, and clean solve rate rose about 16 points over the base model. A Qwen3-32B trained this way never hacked, even when prompted to.
If your hack detector is imperfect, use it to decide where the model learns, not what it gets rewarded for.
