arXiv 研究:编码模型基准分数更高,实际合并率并未提升
Higher-Scoring Coding Models Did Not Get More Work Done
研究借出口管制造成的 19 天模型切换,追踪了近千开发者的真实 PR 数据。结果发现基准分数差 11 分的两个模型,合并率几乎没差别。给用基准榜选模型的工程负责人提了个醒。
一项 2026 年发布于 arXiv 的研究检验了一个此前无人验证的问题:SWE-Bench Pro 上得分更高的编码模型,是否真的能完成更多实际工作。2026 年 6 月 12 日起,美国一项出口管制指令使 Claude Fable 5 停止可用 19 天,迫使原用户切换模型,研究者将此视为自然实验。研究追踪了 297 名 Fable 5 用户在暂停前、中、后的 GitHub pull request,并以 8,908 名始终使用 Opus 4.8 的开发者作为对照组——Opus 4.8 在 SWE-Bench Pro 上比 Fable 5 低 11 分。
基于开发者与日期固定效应的差异中差异模型显示,使用 Fable 5 仅使 pull request 的 7 天合并率提高 1.4 个百分点(区间 [-1.4, +4.3]),不具统计显著性;每周合并数量增加 2.4%(区间 [-4.9%, +10.2%]),同样不显著。研究还在暂停前后对 2026 年的 5 次模型升级重复了对比(1,208 名开发者、15,442 个 pull request),加权平均效应为 -0.9 个百分点,5 个估计值全部为负且 95% 区间均包含 0。
结论指向编码基准分数与真实生产力之间的脱节:SWE-Bench Pro 上 11 分的差距在受控的真实开发场景中未转化为可测量的产出差异。对依赖基准排名选型的团队而言,这份数据提示基准成绩不宜作为唯一采购依据。
Higher-Scoring Coding Models Did Not Get More Work Done
New coding models advertise themselves on score improvements of coding benchmarks. Yet no study has tested if a model that scores higher on these benchmarks gets more real work done. On June 12, 2026, a U.S. export control directive banned access to Claude Fable 5 for 19 days, which forced users to switch to other models. During this time, most former Fable 5 users switched to Opus 4.8 (11 points lower on SWE-Bench Pro compared to Fable 5). We treat this suspension as a natural experiment, where the people and projects stayed the same and only the model changed, so we could isolate the effect of Fable 5 on user productivity. We tracked GitHub pull requests of 297 Fable 5 users before, during, and after the suspension. We used 8,908 developers who stayed on Opus 4.8 all the time as the control group. We also repeated the comparison across five model upgrades before and after the suspension in 2026 (1,208 developers, 15,442 pull requests). Fitting a difference-in-differences model with developer and date fixed effects, we found that having access to Fable 5 increased the 7-day merge rate of pull requests by only 1.4 percentage points [-1.4; +4.3], which was not statistically significant. Similarly, no significant increases of merge rates were found with the upgrades (weighted mean: -0.9 percentage points [-1.9; +0.2]); all five estimates were negative (95% intervals included 0, did not differ from each other). We also did not observe a significant increase in the number of pull requests merged per week (+2.4% [-4.9%; +10.2%]).
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Claude Fable 5、SWE-Bench Pro、Opus 4.8、编码基准,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Yining Hua),经 AITOP 多维评分管线结构化提炼。