论文

SWE-CoWork:把编程智能体放进模拟组织里考核的 101 项任务基准

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

精选理由

这个新基准不是让智能体改几个文件,而是扔进一个有 26 个虚拟同事的模拟公司里干完整活。最强的模型能通过 63% 的测试,但完整交付不到 8%,差距非常直观。做 Agent 工作流或编程助手的人可以拿它来诊断自己系统卡在哪一环。

SWE-CoWork 是一个新的编程智能体基准,包含 101 项开发任务,覆盖 101 个真实仓库、11 种编程语言和 12 个应用领域。每项任务把智能体放进一个模拟的工程组织,面对 26 名职责不同、知识受限的虚拟同事,需要从简短的交接信息出发,恢复散落在同事和工单记录中的需求、处理矛盾信息,并用私有可执行测试来验收交付。对 11 个模型的评估显示,部分实现与完整交付之间存在巨大差距:排名前二的模型能通过约 63% 的 fail-to-pass(F2P)测试,但完整完成的任务不到 8%。提前给出完整需求能把 Seed Evolving 和 GPT-6 Astra 的 F2P 提高约 16 个百分点,但完整任务成功率不变。轨迹分析显示失败贯穿整个交付流程:Claude-Opus-5.5 平均能把 93% 的首次提问发给对应的负责同事,却只拿到 64% 的同事持有需求;被搁置的追问无人跟进,甚至有 31% 被标记为完成的需求最终没通过测试。这说明需求的全流程跟踪——澄清、实现、验证、交接——是自主软件工程可靠性的关键瓶颈。

正文

与 SWE-bench 这类只考单仓库改码的基准不同,SWE-CoWork 考的是“数字员工”在组织协作场景下的完整交付能力,更贴近真实工作中的工程师角色。对做编程智能体和 Agent 工作流的开发者来说,它提供了一个可复现的诊断框架,能定位需求追踪、协作沟通和验证环节的具体失败模式。

原文 · arXiv cs.AI

SWE-CoWork: Benchmarking Coding Agents as Digital Employees in Organizations

A coding agent working within a software organization must establish what to build, collaborate with colleagues, and verify what it delivers. We introduce SWE-CoWork, a benchmark comprising 101 development assignments across 101 real repositories, 11 programming languages, and 12 application domains. Each assignment places an agent in a simulated engineering organization with 26 colleagues who have distinct responsibilities, bounded knowledge, and controlled collaboration behavior. Starting from a brief handoff, agents must recover requirements distributed across colleagues and workplace records, reconcile conflicting evidence, and implement changes evaluated by private executable tests. We evaluate eleven models and find a substantial gap between partial implementation and complete task delivery. The two leading models satisfy approximately 63% of fail-to-pass (F2P) tests but complete fewer than 8% of assignments. Providing complete requirements upfront improves F2P by approximately 16 percentage points for Seed Evolving and GPT-6 Astra, yet leaves complete task success unchanged. Requirement-level trajectory analysis reveals failures throughout the delivery process: Claude-Opus-5.5 directs a mean of 93% of first questions about queried cards to their accountable owners across runs but obtains only 64% of colleague-held requirements. Deferred questions go unfollowed, implementations remain incomplete even after requirements are acquired, and 31% of requirements reported as completed fail their tests. These findings highlight sustained requirement tracking, from clarification through implementation, verification, and handoff, as a key challenge for reliable autonomous software engineering.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:SWE-CoWork、Claude-Opus-5.5、GPT-6、编程助手,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Xinping Lei, Minghao Liu, Ruijie Zhu, Xiaohui Hu, Xingqiao Li, Ao Zhang, Wenhao Chai, Han Li, Tao Wang, Jingyuan Zhang, Enduo Zhao, Yunfeng Shi, Yiming Liang, Zetao Yang, Xingjian Wang, Yifan Yao, Xinkai Ma, Yingru Li, Tianhan Yang, Huan Zhou, Weiran Shi, Shen Yan, Jiaheng Liu, Stephen Huang, Eli Cheung, Weihao Xuan),经 AITOP 多维评分管线结构化提炼。