Amazon 论文发现 9 个规则类基准的评分与答案键存在错误
Amazon 审了 9 个规则类基准，发现评分代码有 bug，LegalBench 修好后模型得分从 9 题涨到 90 题，看榜单前得留个心眼。
Amazon 的一篇论文审计了 RuleArena、LegalBench 等 9 个公开基准，通过追溯参考规则来源、重跑参考程序和构造受控答案来测试评分器。审计发现部分测试的答案键或评分代码本身有误，导致模型正确答案被判错。一个典型案例是 LegalBench 的税务评分器删除了小数点，把 $259,487.13 读成 25,948,713。修复该 bug 后，3 个模型在 96 题中的通过数从 8 或 9 提升到 87 至 91。论文建议在采信低分前先读评分器代码并人工核对失败样例。
New Amazon paper finds that benchmarks used to grade AI models on tax, legal and business rules often mark correct answers wrong, because the tests’ own answer keys or grading code contain mistakes.
Benchmarks like RuleArena and LegalBench test whether models apply written rules to tax, legal, and business cases. If the reference answer or the scorer is wrong, the score measures the benchmark's bugs, not the model.
They audited 9 public benchmarks by tracing references back to the source rules, rerunning reference programs, and testing scorers with controlled answers. Then they rescored the same saved model responses after each fix.
LegalBench's tax scorer deleted decimal points, so $259,487.13 was read as 25,948,713. Fixing it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of 3 models.
Before trusting a low score on a rule-following benchmark, read the scorer code and check a few failed answers by hand.
