研究:5个AI编程智能体的合并PR仍需更频繁的后续修复
Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests
追踪了 6774 个智能体 PR 的后续命运,发现 Codex、Claude Code 这些智能体合并代码后还得自己返工,几率比人类高 62%。
一项研究追踪了来自 AIDev-pop 数据集的 6,774 个合并的智能体 PR,覆盖 OpenAI Codex、GitHub Copilot、Devin、Cursor 和 Claude Code 五个工具,并与同期 5,044 个人类 PR 对比。
结果显示,合并后的智能体 PR 被后续修复的几率是人类 PR 的 1.62 倍。
修复者中,69.6% 的验证修复来自同一个智能体,76.
4% 的修复 PR 全部提交均由智能体完成。研究还用 LLM 评审与人类标注对照,Direct-fix 精度达 90%,二分类 Cohen's Kappa 为 0.
78。
Who Finishes the Job? A Study of Follow-Up Fixes and Commit Authorship on AI Coding Agent Pull Requests
AI coding agents now author a large share of pull requests (PRs) merged into popular open-source projects. A merged agent PR is usually considered finished work; yet, prior studies have reported issues in agent code after the merge (e.g., code smells and static-analysis issues). However, little is known about how often a merged agent PR is fixed afterward, and who actually authors the fixing. In this paper, we follow 6,774 merged agent PRs across five AI coding agents (OpenAI Codex, GitHub Copilot, Devin, Cursor, and Claude Code) from the AIDev-pop dataset (open-source repositories with at least 500 stars) into their follow-up fixes, against a baseline of 5,044 contemporaneous human PRs from the same repositories. We link each merge to its candidate fixes, verify every candidate with human annotators and an LLM judge that matches human-level agreement (binary Cohen's Kappa=0.78$ against a human-human K=0.77$, Direct-fix precision 90%), and attribute the fixing work at the PR and the commit level. Our findings show that (1) merged agent PRs attract verified fixes at 1.62 times the odds of merged human PRs in the same repositories over the same period of time; (2) 69.6% of verified fixes in agent merges come from the same agent; and (3) 76.4% of the verified fix PRs are agent-authored throughout all commits. These results show that agents currently largely finish their own job, but their merges still require fixing more often than human merges.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Claude Code、Cursor、Devin、OpenAI Codex,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: OpenAI」(Tier-1 一手发布),具备高确定性参考价值。截至目前,全网已有 11 家独立信源对该事件进行交叉印证与跟进报道(含 GitHub、shao__meng、Fish Audio 等),属于近期高关注度行业信号。