ServiceNow与McGill研究：AI裁判难以发现智能体任务的细微错误
想让LLM给智能体打分？GPT-5.5在困难集也只有80.9%，Qwen加工具反而掉分。
ServiceNow 与 McGill 的论文指出，LLM 裁判在评估上百步的 computer-use 智能体运行时，常漏掉画错形状、发错 Slack 频道这类细微错误。团队据此构建 AgentHorizon 基准，从人类屏幕录像整理出 1373 个评判任务，通过交换近似任务的指令构造出干净的失败样本。最强的 GPT-5.5（Codex 环境）在困难分集上拿到 80.9% 平衡准确率，而全部判对的基线只有 50%。Qwen 3.6 27B 直接读完整运行记录得 77.6%，改用工具方式判断却降到 70.4%。
New ServiceNow and McGill paper finds that AI judges still miss subtle mistakes in long computer-use tasks, and giving them tools does not reliably help.
Teams use LLMs to grade agent runs for training and evaluation. A run of 100+ steps can look finished while it drew the wrong shape or posted to the wrong Slack channel.
They built AgentHorizon, 1,373 judging tasks from human screen recordings. People recorded pairs of near-identical tasks and swapped the instructions, so each failure is a clean run of a slightly different request.
The best judge, GPT-5.5 in Codex, scored 80.9% balanced accuracy on the hard split, where always saying yes gets 50%. Qwen 3.6 27B did worse with tools, scoring 77.6% reading the whole run directly but 70.4% as an agent.
Before you trust an LLM judge with agent runs, test it on near-miss failures inside the exact agent setup you will use.
