arXiv 论文分析 21000 行全由 Claude 编写的代码库可靠性
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
有人让 Claude 独立写了两万多行 Python 代码，论文统计了错误率，还发现四五个回复里就有一个事实错误。
一篇 arXiv 论文发布了一个 21000 行 Python 工具的完整开发历史数据集，代码与测试全部由 Claude 编写，没有人工参与。作者还开源了两个代码溯源追踪工具，以及针对指令意图、提交溯源、响应可靠性的三套分类体系。分析显示 14.3% 的 AI 代码生成事件包含真实错误，后被 AI 编写的测试套件捕获；AI 的交互回复中约四分之一到五分之一包含至少一处事实性错误。研究还发现编程智能体 CLI 指令与 IDE 聊天指令在类型上不同，前者更侧重理解、规划和咨询。
Between the Commits: Process, Error, and Claim Reliability in a Wholly AI-Authored Codebase
We present: (i) a new dataset consisting of the full development history of a 21,000-line Python tool built entirely by Claude AI, with no human-authored code or tests, (ii) two code-provenance tracing tools, (iii) three taxonomies for instruction intent, commit provenance, and response reliability, (iv) application of these to analyse the dataset. We find that: (i) user coding agent CLI instructions differ in kind from IDE-chat instructions, with a greater focus on comprehension, planning and consultation, (ii) code development is mainly proactive, (iii) 14.3% of AI code-generation events contain a real error later caught by the AI-authored test suite, (iv) roughly 1 in 4-5 of the AI's interactive responses contains one or more factual errors.
