SWE-bench 作者 Ofir Press 谈新基准该从 0% 起步
SWE-bench 作者 Ofir Press 分享基准设计心得:上线时最强模型应接近 0 分。他举了 SWE-bench 从 2% 到接近满分的历程,还提到 Cognition 的 Devin 是引爆点。做评测或想看懂榜单的开发者都该听听这段对话。
SWE-bench 共同作者 Ofir Press 在 Kaggle 的 Evaluated 播客中指出新基准构建者的两个常见错误:起点太容易,以及任务没人真正关心。
他建议基准发布当天最强模型得分应接近 0%,并选择易于解释、易于评判但极难执行的自然任务。SWE-bench 发布时最强模型只解决了 2% 的任务,前三四个月被认为几乎不可能完成,直到 Cognition 在 Devin 发布会上引用该基准后才引发团队竞逐。
如今模型在 SWE-bench Verified 上已能解决大多数任务。Ofir 认为下一代基准需要寻找当前完全无法完成的任务,设计时要"以超人标准思考"。
"If you're launching a benchmark that starts out at 50% accuracy, that objective has almost already been reached," says @OfirPress, co-author of SWE-bench.
On the Evaluated podcast, Kaggle's Gina Lee asked him about the mistakes new benchmark builders make:
1️⃣ Starting too easy. The best models should score around 0% on launch day, so there's room to climb. 2️⃣ Choosing tasks nobody cares about. Go with natural tasks, the ones we really want models to do well.
To avoid these mistakes, Ofir recommends tasks that are "easy to explain, easy to judge, but also extremely hard to execute on."
When SWE-bench launched, the best model solved 2% of its tasks, which were based on bugs from GitHub. For the first three or four months, Ofir says, people thought it was too hard, maybe impossible. Then Cognition cited SWE-bench in its Devin launch, and teams started competing. Today, models solve most tasks on SWE-bench Verified.
The next benchmarks must find even more difficult tasks. "We need north stars for things that are totally impossible right now," Ofir says. To build a strong benchmark today, "you need to think superhuman."
What would you build a 0% benchmark for?
本条情报归属 AITOP「行业」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:SWE-bench、Devin、Cognition、Kaggle,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「Kaggle」(@kaggle),经 AITOP 多维评分管线结构化提炼。