CTXGAP研究：编程智能体的报告仅能恢复一半任务状态
What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
用 DeepSeek 跑的测量研究：编程智能体的汇报只能恢复一半任务状态，失败的尝试可能压根没写进报告。
arXiv 论文 CTXGAP 量化了编程智能体面向用户的报告与完整执行记录之间的任务状态差距。在 37 个编程任务、4,799 条有效探针上，仅读报告者能恢复 52.0% 的任务状态，完整上下文读者为 92.7%，差距达 40.7 个百分点；在 30 个已验证修复的最终检查点，差距仍有 34.8 个百分点。提供完整终端滚动记录可把恢复率提升到 73.6%，基于上下文重写报告可提升约 8 个百分点。人工标注发现"提及但缺乏支撑细节"是未恢复信息中占比最大的类别，结果在 DeepSeek 独立执行与测量下保持一致。
What Coding Agents Leave Unsaid: Measuring Task-State Gaps Between Execution and Reports
Human supervisors rely on coding agents' reports to assess progress, verify repairs, and decide when to intervene. But a supervisor cannot investigate a failure they were never told occurred. Failed attempts, workarounds, and unresolved risks may disappear from routine reports, leaving supervisors unaware of issues that warrant scrutiny. We conduct CTXGAP, a measurement study quantifying the gap in task-state recoverability between agents' accumulated user-facing reports and their full recorded execution contexts. Across 37 coding tasks and 4,799 valid probes, agents answer 93.6% of task-state questions on demand; full-context readers recover 92.7%, while report-only readers recover 52.0%, a 40.7-point gap. At the final checkpoints of 30 verified fixes, the gap remains 34.8 points. Human annotation identifies states mentioned without enough supporting detail as the largest category of unrecovered information, and model-assessed supervision-relevant states retain substantial gaps. Full terminal scrollback improves recovery to 73.6% but still trails full context. Under matched report lengths, context-informed rewrites improve recovery by about eight points, without a detectable targeted-over-natural advantage. Positive gaps persist across harnesses and independent DeepSeek execution and measurement. CTXGAP establishes report-based task-state recoverability as a complementary evaluation dimension.