编程代理排名失去区分度，SWE-bench需新评估标准
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
这是篇关于编程代理排名的深度研究，作者分析了SWE-bench数据，发现现有排名标准可能不准确，推荐给对编程助手性能评估感兴趣的朋友。
研究审计了254个SWE-bench提交结果，发现前两名在Verified测试集上各解决396个实例，但十强共285个成功和51个失败，164个实例无法区分优劣。
Frontier解决方案的中位嵌套深度为0.935，远高于基于分数的基线0.
774，表明成功共享。分数还取决于模型-框架对，同一模型内框架差异达29.
8个百分点，而前十名仅8.8个百分点。
六项单元格均值交互测试在Holm校正后仍显著，但精确配对McNemar检验无法区分29个相邻的前三十名对。这表明小规模分数差异不足以建立排名顺序，需要报告比较集特定的解决率和模型-框架来源。
Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Small differences on coding-agent leaderboards are often read as an ordering of systems. We audit whether the published verdicts support this reading, using 254 SWE-bench submissions across four splits without running models. On Verified, the leading two entries each resolve 396 of 500 instances. The top ten share 285 successes and 51 failures, leaving 164 instances that distinguish their outcomes. Frontier solution sets have median nesting 0.935 against a score-implied baseline of 0.774, indicating strongly shared successes. Scores also depend on the evaluated model-scaffold pair: observed within-model scaffold ranges reach 29.8 percentage points, compared with the 8.8-point spread of the top thirty. Six of nine cell-mean interaction tests remain significant after Holm correction, although this observational design does not identify causal scaffold effects. Exact paired McNemar tests separate none of the 29 adjacent Verified top-thirty pairs at alpha=0.05, while the larger Test split separates 14 of 23. A stated leader-based rule yields three descriptive tiers, or two after Holm correction; non-rejection does not establish equivalence. We release the partition and a five-step audit protocol that profiles shared outcomes, tests paired differences, reports grouping sensitivity, and estimates the instance budget needed for resolution. The results motivate reporting comparison-set-specific resolution and model-scaffold provenance instead of interpreting small aggregate gaps as established rank differences.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：SWE-bench、编程助手、推理模型、评估标准，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo），经 AITOP 多维评分管线结构化提炼。
