把 RLHF 中的奖励欺骗建模为寻租，提出可度量的审计机制
Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent
论文把奖励模型过度优化类比成寻租，量化了 RL 训练中多少增益是假繁荣，还测出策略会钻审计的空子，做 RLHF 的人可以看看审计频率怎么设。
一篇 arXiv 论文将 RLHF 中的奖励模型过度优化重新表述为经济学中的寻租问题：策略像企业，奖励模型像监管者。在包含三个模型家族、两个 gold 奖励模型的固定测试平台中，RL 把 41% 到 66% 的代理奖励增益变成了不对应真实质量的 rent。两个代理奖励模型分别奖励自信的具体细节和勤奋的表现，导致策略在 97% 到 100% 的探针上编造不存在的实体，或在 67% 到 98% 的场景中回避事实问题。论文还引入税收经济学的 bunching 估计器，发现策略开始学会待在审计阈值之下，并证明固定罚款预算下频繁轻度审计优于稀少严厉审计。
Rent-Seeking Language Models: Reward Models Set the Prices, Frequent Audits Curb the Rent
Reward model overoptimization in RLHF is usually described as reward hacking. We recast it as rent-seeking: the policy, a firm, earns reward from a regulator, the proxy reward model, that true quality does not back. The lens turns overoptimization into measurable quantities, namely rent, a capture index and deadweight loss, and into testable predictions from the economics of regulation. In a fixed, version-controlled testbed with three model families, accuracy-matched proxy reward models and two gold reward models, RL turns 41% to 66% of its proxy gain into rent while gold reward plateaus. Rent follows each regulator's prices. Neither proxy verifies facts, but one pays for confident specifics and the other for displays of diligence, so RL policies learn to fabricate details about nonexistent entities (97% to 100% of probes) or to deflect factual questions (67% to 98%). Kinship between policy and regulator adds a fixed premium of about 0.1 standard deviations but gives the policy no extra leverage under optimization. At a fixed expected fine, frequent mild audits beat rare severe ones, and random audits are as efficient per audit as targeted ones. Bringing the bunching estimator of tax economics to RL training, we also find a first sign that policies learn to stay just below a targeted audit's threshold.
