AgentFaultBench:验证 LangGraph 与 CrewAI 智能体在工具故障后的恢复能力
AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery
做智能体开发的人可以看看这个基准:往 GitLab 里注入工具故障,测 LangGraph 和 CrewAI 能不能恢复。结论有点反直觉——给恢复建议不一定有用,主实验里甚至有一组掉了 8.33 个百分点。如果你在给智能体加错误处理提示词,这篇能帮你避免想当然。
一篇新论文提出 AgentFaultBench,通过向隔离部署的 GitLab 环境注入工具错误,评测 LangGraph 和 CrewAI 智能体在故障后能否完成任务。主对比实验包含 1,944 次计划运行,结果显示恢复建议相对中性提示的成功率变化为 -8.33、0.00 和 +1.06 个百分点,部分缺失结果使后两组对比无法下结论。该基准不仅检查最终文件与 issue 的状态,还核查必要的读取、验证步骤与最终回复,提示词中预先给出期望的 issue 和报告文本,因此测试的是执行能力而非内容生成。
后续实验测试修订后的恢复策略:GPT-4o mini 有改进,而 Sol 与 Claude 已能完成可恢复的用例。在另一项 624 次运行、覆盖六个新任务的研究中,引导条件在 192 次计划运行中取得 161 个正确终态(中性反馈为 142/192),包含缺失结果时增益为 +9.90 到 +10.42 个百分点。移除 issue ID 后成功率为 47/48,对比带 ID 的 46/48。第二名独立校验者在 1,879 个主实验结果和 241 个构造测试用例上达成一致,论文未收集人工验证。结论是:恢复建议的效果应通过显式恢复对照实验来检验,不能默认有效。
对开发者而言,这项工作提醒依赖 LangGraph 或 CrewAI 等框架构建智能体时,工具故障恢复环节需要实际基准测试而非凭直觉叠加提示词。论文区分了不同模型(GPT-4o mini、Sol、Claude)在恢复能力上的差异,为选型提供了参考。
AgentFaultBench: State-Verified Evaluation of Tool-Fault Recovery
Does recovery advice help an agent finish a task after a tool fails? AgentFaultBench evaluates LangGraph and CrewAI agents performing specified tasks in an isolated GitLab installation. We inject tool errors and check both the resulting files and issues and the required reads, verification and final response. The prompt supplies the expected issue and report text, so the task tests execution rather than content generation. In the primary comparison of 1,944 planned runs, advice changes success by -8.33, 0.00 and +1.06 percentage points relative to a neutral reminder; missing outcomes leave the latter two comparisons inconclusive. Later experiments test a revised recovery policy. GPT-4o mini improves, while Sol and Claude already complete the recoverable cases. In a separate 624-run study on six related tasks not previously used by that policy, the guided condition records 161 correct final states and one unknown outcome in 192 planned runs, versus 142/192 with neutral feedback. The gain lies between +9.90 and +10.42 pp when missing outcomes are included. Removing issue IDs gives 47/48 successes versus 46/48 with IDs; other execution information remains. A second checker agrees on 1,879 observed primary outcomes and 241 constructed test cases. No human validation was collected. The findings support testing advice against explicit recovery controls rather than assuming it helps.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:AgentFaultBench、LangGraph、CrewAI、GPT-4o mini,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Sachin Gupta),经 AITOP 多维评分管线结构化提炼。