JetBrains 研究称编码智能体基准应贴近真实用户任务流
JetBrains 研究员看了 4,782 个真实会话，发现 SWE-Bench Pro 这种基准和实际用法差太远，还做了个新基准 SWE-TaskFlow，做评测的应该看看。
JetBrains 研究团队分析了 4,782 个真实的编码智能体会话，发现功能开发和修 Bug 不足一半请求，用户会在会话中频繁切换提问、规划、审查和编辑。现有基准如 SWE-Bench Pro 只给智能体 1 个 issue 并评判 1 个最终补丁，其中 93.8% 是新功能和修 Bug 任务。团队据此构建了 SWE-TaskFlow，将每个 issue 拆成多轮对话并加入可校验的代码问题。在 700 个 SWE-Bench Pro 任务上测试，拆分后成本约翻倍，成功率没有稳定变化。
New JetBrains Research paper argues that coding-agent benchmarks should copy how their real users actually work, because real sessions mix and switch task types across many messages.
Real developers using coding agents ask questions, plan, review, and switch tasks mid-session, but benchmarks like SWE-Bench Pro hand the agent 1 detailed issue and grade 1 final patch. In that benchmark, new features and bug fixes make up 93.8% of tasks.
JetBrains researchers studied 4,782 real agent sessions, where features and bug fixes were well under half of requests. In longer sessions, users kept switching between questions, planning, reviews, and edits.
They built SWE-TaskFlow, which splits each benchmark issue into several turns and adds checkable questions about the code. On 700 SWE-Bench Pro tasks, splitting roughly doubled agent cost with no stable change in success rate.
If you evaluate coding agents, first measure what your own users ask for, then shape the benchmark's conversation to match it.
– arxiv. org/abs/2610.09633
Title: "Coding-Agent Benchmarks Should Match Their Users' Task Flows"
