arXiv 论文对比三种智能体评估框架：trace 评估 F1 达 0.90
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
做智能体评估的同学该看看这篇。它用同一批 trace 实测了三种评估框架：trace-based F1 到 0.90，纯文本方案误报多 5 倍，还有 24 个功能测试和安全红队零重叠的新失败。结论很实在：功能评估用 trace，安全验证用红队，两条腿走路。
arXiv 论文 2611.14493 对三种 LLM 智能体评估框架做了三阶段对比研究，覆盖基于 trace 的 LLM 评判、纯文本 LLM 评判加红队测试、确定性启发式规则三种范式，实验对象是两个企业分析智能体。Phase 1 通过重复测量量化评估框架的稳定性，97% 的配置达到稳定性阈值，且稳定性模式与具体智能体无关。Phase 2 在 72 个 ground-truth 测试用例上测量已知风险检测能力：trace-based 框架取得最佳精确率-召回率平衡（F1=0.90），纯文本评估召回率相当但误报多 5 倍。
Phase 3 评估未知风险发现，共挖出 24 个新失败案例，功能测试与安全红队测试之间零重叠，说明两者互补。论文还发现基于 LLM 的评估框架能发现人工审查遗漏的细微错误，评估框架可获得的信息量决定评估质量。
作者给出的落地建议是双框架策略：用 trace-based 评估做功能验证，用对抗性红队测试做安全验证。对需要在多工具工作流中部署分析智能体的企业工程团队，这是选型评估方案的直接参考。
Comparative Analysis of Agent Evaluation Frameworks: Stability, Detection, and Discovery in Enterprise Analytics Agents
Deploying LLM-based analytics agents in enterprise settings requires evaluation frameworks that can reliably detect failures across complex, multi-tool workflows. We present a three-phase comparative study of three evaluation frameworks, each representing a distinct evaluation paradigm (trace-based LLM judging, text-only LLM judging with red-teaming, and deterministic heuristics), applied to two analytics agents in a controlled research setting using frozen execution traces. Phase 1 quantifies evaluation harness stability through repeated measurements, finding that 97% of configurations meet the stability threshold and that stability patterns are agent-independent. Phase 2 measures known risk detection on 72 ground-truth test cases, where the trace-based framework achieves the best precision-recall balance (F1=0.90) while text-only evaluation matches on recall but generates 5x more false alarms. Phase 3 evaluates unknown risk discovery, uncovering 24 novel failures with zero overlap between functional testing and security red-teaming. Our results demonstrate that (1) the information available to the evaluation harness determines evaluation quality, (2) functional and security evaluation are complementary, and (3) LLM-based evaluation harnesses surface subtle errors that human reviewers miss. We recommend a two-framework strategy: trace-based evaluation for functional assessment and adversarial red-teaming for security validation.
本条情报归属 AITOP「论文」核心栏目，重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：智能体评估、trace-based evaluation、red-teaming、LLM judging，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Howard Zhang, Chuck Arvin, Logan Brown, Mihir Bhatia, Neelu Choudhary, Lily Miao），经 AITOP 多维评分管线结构化提炼。
