论文

新评估套件可检测用户模拟器全部 10 类行为失败

On Evaluating User Simulators for Interactive Agents

精选理由

如果你在做智能体评测,这篇很实用:它实测了现有用户模拟器评估指标的盲区,三类老方法只能抓出 10 类失败里的 5-6 类,新套件能全抓,还开源了基于 351 段真实会话的评测数据。

一篇 arXiv 论文提出针对交互式智能体评估中用户模拟器的元评估方法:先对模拟对话回放注入 10 类扰动(目标改变、约束遗漏、提前停止、接受未完成工作等),再检验各评估指标能否惩罚这些失败。结果显示三项已有研究的评估方法只能检测出其中 5-6 类,其综合得分甚至会偏好那些被指示接受破损工作的模拟器人群,优于未修改的对照组。

正文

作者随后提出新评估套件,从三个维度衡量:模拟器对智能体评估的效用、人群级保真度(整体交互模式)、轨迹级保真度(个体用户的目标、约束与人设是否保留)。在同一检测标准下,该套件可检测全部 10 类扰动。

研究团队在 351 段真实多轮会话上评测了六个用户模拟器,这些会话覆盖 348 名用户、23 种语言,任务包括全栈 Web 应用、游戏、设计资产和自动化工作流。结果显示人群级指标可能漏检模拟用户追求错误目标、忽略约束或行为出格的情况,说明需要在多个粒度上评估保真度。

原文 · arXiv cs.LG

On Evaluating User Simulators for Interactive Agents

User simulators offer a scalable way to evaluate interactive agents, but how to assess their quality remains underexplored. In particular, it is unclear whether existing user simulator evaluation frameworks reliably detect meaningful failures in simulated user behavior. This uncertainty makes it difficult to interpret simulator evaluation. To understand what a high benchmark score tells us about simulator quality, we (meta-)evaluate the metrics themselves. To instantiate this meta-evaluation, we deliberately perturb simulated replays to induce failures such as altered goals, omitted constraints, premature stopping, and acceptance of broken work, then test whether the metrics penalize these failures. Evaluation methods from three prior studies detect only 5-6 of the 10 perturbations; their combined scores even favor populations instructed to accept broken work over unmodified controls. Motivated by these gaps, we propose a new evaluation suite that combines utility of user simulators for evaluating agents and two measures of fidelity to real users: population-level fidelity assesses collective interaction patterns, while trace-level fidelity assesses the preservation of individual users' goals, constraints, and persona. Under the same detection criterion, our evaluation suite detects all 10 perturbations. Evaluating six user simulators on 351 real multi-turn sessions in which 348 distinct users across 23 languages and agents work together to create and refine artifacts such as full-stack web apps, games, design assets, and automated workflows, we find that population-level measures can miss when a simulated user pursues the wrong goal, ignores a constraint, or behaves out of character, highlighting the need to assess fidelity at comprehensive granularities.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:用户模拟器、智能体评估、benchmark、arXiv,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Hengguang Zhou, I-Hung Hsu, Li Chen, Priyanshi Shrivastava, Anastasios Angelopoulos, Wei-Lin Chiang, Ion Stoica, Cho-Jui Hsieh),经 AITOP 多维评分管线结构化提炼。