arXiv 论文提出分层统计框架量化智能体系统可靠性
A Hierarchical Statistical Framework for Evaluating Agent Reliability
做智能体评估的朋友该看看这篇:它把 pass@k 这类指标配上了有限样本方差,还能把执行噪声和任务本身的难度差异分开。如果你在纠结跑多少次采样才算数、或者 Thinkingbox 上模型忽好忽坏怎么解释,这篇文章给了统计上的答案。
一篇 arXiv 论文提出用于评估智能体系统不确定性与可靠性的分层统计框架,将评估对象组织为场景、实例、样本三个层级。论文在该框架下为文献中常见的 mean pass rate、pass@k、passhat^k 等指标给出不确定性估计,并独立推导出无偏 pass@k 与 passhat^k 估计量的精确有限样本方差表达式。
方法上的关键贡献是对分层不确定性传播的构造性推导:论文将层级化指标聚合与物理及统计文献中的粗粒化过程对应起来,并明确区分了执行噪声与任务异质性两类变异来源。在应用层面,论文引用近期发布的 Thinkingbox 基准作为例证——即便最强模型在多样化业务场景下也难以保持一致表现,这凸显了系统性量化性能波动对生产环境部署的意义。
A Hierarchical Statistical Framework for Evaluating Agent Reliability
Agents operating in complex scenarios often face uncertainty due to incomplete information about the environment they operate in, ambiguous user queries, complex business rules and the inherently stochastic dynamics of the governing policy. Several new benchmarks have been proposed to evaluate LLM-based agents on complex tasks in multi-tool environments, including settings with policy constraints and varying levels of intent ambiguity. These conditions make it important to evaluate agentic systems systematically and quantify their performance under different conditions to assess their reliability in production. This need is evident in the recently introduced Thinkingbox benchmark, where even the strongest models struggle to perform consistently across diverse business scenarios. This paper presents a statistical framework for evaluating uncertainty and reliability in agentic systems. We define a hierarchy of scenarios, instances, and samples to assess model performance and variability systematically. We apply this framework to common metrics found in the literature: mean pass rate, $pass@k$, and $passhat^k$, with uncertainty estimates for each metric. Aggregating metrics at different levels of the hierarchy reveals different structures and properties of the policy model; we connect this hierarchical aggregation with the known process of \emph{coarse-graining} as intended in the physics and statistical literature. We independently derive exact finite-sample expressions for the sampling variances of the unbiased $pass@k$ and $passhatk$ estimators; related results also appear in recent submissions. Our central contribution is a constructive derivation of hierarchical uncertainty propagation, making explicit its correspondence with coarse-graining and separating execution noise from task heterogeneity.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:智能体评估、pass@k、Thinkingbox、统计框架,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Mirco Milletarì, Tuhin Kundu),经 AITOP 多维评分管线结构化提炼。