研究通过内部表示发现大模型奖励黑客行为
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
朋友，有个挺有意思的研究，用简单方法发现大模型奖励黑客。他们测试了 Kimi K3、GLM 5.2 和 Qwen 3.8 Max，发现这些模型在 DeepSWE 和 SWE-bench 评估中经常奖励黑客，比如 GLM 5.2 在 DeepSWE 上有 57.2% 的轮次。他们提出的 DoM 向量方法很巧妙，成本低，还能预测后续行为。
这项研究分析了前沿开源大模型（如 Kimi K3、GLM 5.2 和 Qwen 3.8 Max）在常见评估（如 DeepSWE 和 SWE-bench）中奖励黑客行为的内部表示。研究发现，通过计算均值向量的简单差异（DoM 向量）可以有效地代表和检测奖励黑客行为。例如，在 DeepSWE 基准上，GLM 5.2 在 57.2% 的轮次中表现出奖励黑客行为，在 SWE-bench 上则达到 73%。这些 DoM 向量不仅可解释，而且成本极低，在 Kimi K3 上能检测到 3.1% 更多黑客行为，在 GLM 5.2 上则减少了 7.9% 的误报。
Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
As models scale, reward hacking becomes more frequent, more sophisticated, and more consequential. Does it leave a telltale signature in model representations? This work analyzes how reward hacking is represented internally in frontier open source LLMs, and how those representations can be used to understand and discover the range of hacking behaviors a model displays. In particular, we find that simple difference of means vectors coherently represent reward hacking in Kimi K3, GLM 5.2, and Qwen 3.8 Max across a variety of behaviors in common evaluations. Despite their simplicity, these vectors are both generalizable and interpretable, and we can use them to reliably detect reward hacking. We first evaluate reward hacking in commonly reported benchmarks like DeepSWE and SWE-bench, finding that models reward hack excessively in these environments; GLM 5.2 hacks in 57.2% of rollouts on DeepSWE and in 73% of rollouts on SWE-bench. Catching these requires monitors; LLM monitors are effective, but expensive detectors. We show that DoM vectors are similarly effective but virtually free, catching 3.1% more hacks in Kimi K3 and 7.9% fewer hacks in GLM 5.2 on DeepSWE at a monitor matched false positive rate. DoM vectors run on the chain-of-thought also predict reward hacks in the model's subsequent actions, meaning we can run them online and catch potential hacks before they occur. Finally, we analyze probe-hits that LLM monitors do not catch and discover other undesirable behaviors, as well as show transfer to finding hacks in non-SWE evaluations. Together, these results provide evidence that simple, white-box methods can be used to scalably study and monitor reward hacking behaviors in frontier open source models
