Terminal-Bench 审计发现约 30% 任务有缺陷，修复后发布 2.1 版
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
这篇论文把 Terminal-Bench 2.0 翻出来查了一遍，三成任务有问题，修完之后各模型排名都会动。做评测的人应该看看他们的持续验证流程。
arXiv 论文提出基准维护债务概念：模型进步后，原有验证器和评估设置可能失效，导致基准分数偏离真实能力。作者对 Terminal-Bench 2.0 做审计，发现约 30% 的任务存在问题，并将修复结果发布为 Terminal-Bench 2.1。修复后基准准确率提升超过 10 个百分点，排行榜位置最多变动三位，但整体排名相关性仍达 Spearman rho=0.958。论文还展示资源预算会影响结论：延长时限后，GPT 5.4 上的 20pp 工具差距消失，GPT 5.4 mini 的 30pp 差距缩小到 5pp。
Benchmarks as Software: Terminal-Bench and the Case for Continuous Benchmarks
Coding agents are among the most economically consequential deployments of LLMs to date, making reliable evaluation increasingly important. Yet even benchmarks that are carefully validated at release can become stale. As models improve, they can discover valid solutions that existing verifiers reject, while changes to evaluation infrastructure can invalidate assumptions embedded in tasks and tests. We call the resulting accumulation of defects benchmark maintenance debt: a growing gap between benchmark scores and the capabilities they are intended to measure. We introduce a continuous-validation workflow for agentic benchmarks that combines community reports, task artifacts, agent traces, automated oracle checks, and human-gated repairs. We apply this workflow to Terminal-Bench 2.0, a benchmark that underwent extensive human and LM-assisted review and red-teaming before release. Our audit identifies issues in roughly 30% of the benchmark, and we release the resulting repairs as Terminal-Bench 2.1. These repairs change evaluation results: accuracy increased by over 10 percentage points and leaderboard positions move by up to three ranks, even as the overall ranking remains highly correlated (Spearman rho = 0.958). We evaluate how resource allocation shapes benchmark conclusions. We show that fixed wall-clock can materially change agent performance: For example, extending the time limit closes a 20pp harness gap on GPT 5.4 and shrinks a 30 pp gap on GPT 5.4 mini to 5 pp. Benchmark maintenance is therefore not only about repairing task artifacts: resource budgets are part of the measurement specification and must be validated alongside the tasks themselves. More broadly, benchmark maintenance debt is not just accumulated implementation bugs: our results show that as models improve, both evaluation assumptions and resource settings may need to be revalidated.
