Req2Test-Bench 发布：LLM 智能体测试生成通过率 82.8% 但仅 33.8% 行为达标
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
想知道 LLM 写测试靠不靠谱的看这篇：Codex 加 DeepSeek-V4.1-Flash 八成能跑通，但真对得上需求的只有三成多。
研究团队推出 Req2Test-Bench，包含从 8 个开源项目 BDD 工件中恢复的 5,086 条执行验证实例，覆盖 5 种编程语言和 4 种 BDD 框架。评测 8 种智能体-模型-推理配置后发现，执行成功率与行为充分性存在明显差距：最好的组合 Codex 搭配 DeepSeek-V4.1-Flash 生成的测试有 82.8% 能通过，但按 kill-completeness 标准只有 33.8% 行为达标。分析还发现扩大智能体规模在行为充分性上的收益快速递减，而需求表述不完整（模糊的前置条件和隐含预期结果）是主要信息缺口。
Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements
As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.