论文

arXiv 论文提出 benchleaking 概念:测试题进入预训练数据,ERNIE-5.5-Flash-Base 泄露 96% HumanEval 题目

Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

精选理由

用 20 个基准实测了 4 个开源 base 模型的“背题”程度,ERNIE-5.5-Flash-Base 直接把 96% 的 HumanEval 连答案背出来了,DeepSeek 只漏 9%。做模型评测或选底座的人应该看看 Anti-Hack 这个分数怎么算。

arXiv 论文(编号 2611.21315)将基准测试题直接进入训练数据、使分数反映记忆而非能力的现象命名为 benchleaking,并与不接触测试集、单纯向基准优化的 benchmaxxing 区分开。论文记录了过去一个月的两个升级:后训练阶段,审计发现某个 agent 在 23% 的尝试中复述了从未被告知的隐藏测试要求,其前代模型为 0;预训练阶段,通过 20 个基准、4 个公开 base 模型的 few-shot 续写实验,部分 base 模型会在输入中不含测试题的情况下完整写出题目、选项和答案。

正文

量化结果差异明显:ERNIE-5.5-Flash-Base 有 96% 的 HumanEval 题目连参考实现一起泄露,85% 的 OpenBookQA 题目连答案一起泄露,而 DeepSeek-V4-Flash-Base 对应为 9% 和 0。论文提出 Anti-Hack 分数——把所有泄露题一律判错后重算基准分数——全套基准平均后,DeepSeek-V4-Flash-Base 损失 1.5 分、Hy3-preview-Base 损失 1.3 分、MiMo-V2.5-Base 损失 5.0 分、ERNIE-5.5-Flash-Base 损失 16.6 分。作者呼吁把去污染报告和污染调整后的分数作为行业标配,因为开源 base 模型是大量下游研究的底座,污染影响的不只是排行榜。

原文 · arXiv cs.AI

Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base models

Benchmark contamination is not new, but recent model releases show it in a sharper form. We use the term benchleaking for the case in which a benchmark's own test examples enter training data, so that a score reflects recall rather than capability. This differs from benchmaxxing, which optimizes toward a benchmark without seeing its test set. We document two escalations over the past month. First, in post-training, the traces have become explicit: recent audits show an agent stating hidden test requirements it was never given in 23% of its attempts, where its predecessor did so in none, and continuing to follow them after the task is changed to contradict them (Schaeffer and Räuker, 2026; Lo and Lo, 2026). Second, the problem has reached pre-training. Using few-shot continuation on 20 benchmarks across four public base models, we find that some base models write out other test examples---complete questions, options, and answers---that appear nowhere in their input. For the recently released ERNIE-5.5-Flash-Base, 96% of HumanEval problems leak with their reference implementations and 85% of OpenBookQA questions leak with their answers, against 9% and none for DeepSeek-V4-Flash-Base. To quantify what a reported score owes to such examples, we define an Anti-Hack score: the benchmark score recomputed with every leaked example marked wrong and the number of examples unchanged. Averaged over the suite, it costs DeepSeek-V4-Flash-Base 1.5 points and Hy3-preview-Base 1.3, against 5.0 for MiMo-V2.5-Base and 16.6 for ERNIE-5.5-Flash-Base. Open base models are the foundation of much downstream research, so benchleaking now affects more than leaderboard rankings. We call for decontamination reports and contamination-adjusted scores to become standard practice.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:benchleaking、ERNIE-5.5-Flash-Base、DeepSeek-V4-Flash-Base、HumanEval,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng Guo, Pengfei Liu),经 AITOP 多维评分管线结构化提炼。