SWE-DIVE：用用户模拟器评测多轮编程智能体
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
一群研究者做了个 SWE-DIVE 框架，用模拟用户跟编程智能体多轮对话来测真实能力，比 SWE-bench 那种单轮出题更贴近实际，代码开源可跑。
arXiv 论文 SWE-DIVE 指出 SWE-bench 等基准只测一次性、格式规范的 GitHub issue 输入，与真实使用不符。该框架引入一个可模拟不同行为模式的用户模拟器，让编程智能体在多轮对话中接收追加任务和需求。系统还包含一个对话验证器，用来检查用户模拟器的输出是否满足初始要求。整个框架运行在 Harbor 环境中，由它编排智能体、模拟器和验证器并判定任务是否完成，源代码已在 GitHub 开源。
SWE-DIVE: Multi-Turn Coding Agent Evaluation with Dynamic User Simulation
Coding agents are typically evaluated using one-shot, well-formulated prompts. Benchmarks such as SWE-bench use existing GitHub issues as input for a coding agent and evaluate whether the agent successfully solves the task. However, in real-world use cases, prompts are often not as well-formulated or structured as existing GitHub issues. Additionally, a coding agent session is not limited to a single prompt; it typically involves a multi-turn interaction in which the user iteratively adds new tasks and requirements that the agent must fulfill. To benchmark these abilities, we developed SWE-DIVE, a dynamic, interactive validation and evaluation framework for benchmarking coding agents in a multi-turn environment. To achieve this, we introduce a user simulator that can simulate different behavioral patterns and communicate with the coding agent. We also introduce a conversation validator that evaluates the user simulator's outputs to determine whether they fulfill the initial requirements and guide the agent toward successfully solving the task. The entire framework is embedded in a Harbor environment, which orchestrates the coding agent, user simulator, and conversation validator, and executes the test environment to determine whether the agent successfully solves the task. Our results and source code are available in our GitHub repository (https://github.com/lorenzbrehme/swe-dive_usersim).