arXiv 论文提出 Verification Transport:检验 LLM 验证器判断能否跨抽象边界保持稳定
Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent Evaluation
做 agent 评估的人该看看这篇:同一轨迹换个表示,验证器判断可能就变了,宏抽象下失败率到 31%。文章还给了修法——补回执行溯源能救回 85.5% 的失败,实操参考价值很高。
arXiv 论文(编号 2611.18471)提出 Verification Transport 测试,衡量 LLM 验证器在执行轨迹经过工具组合、委托边界、历史压缩等抽象化处理后,判断是否仍然可靠。
实验覆盖 4 个 agent 评估基准、7 个验证器骨干、4 种验证协议和 3 类抽象边界,结果显示传输失败率最高约达 31%,出现在宏抽象与委托场景下。不同协议的失败模式不同:直接验证更容易给出错误判定,而基于证据和审慎式协议在可观测性受限时倾向于弃权,说明低失败率并不等于更可靠。
恢复对应的执行溯源可修复 85.5% 的传输失败,但溯源信息有损或缺失时恢复率下降。
结论是仅看原生视图上的准确率不足以刻画验证器的可靠性,跨抽象边界的稳定性需要单独测量。
Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent Evaluation
LLM-based verifiers are increasingly used to evaluate agent executions, but their reliability is typically measured under a single representation of the trajectory. In deployed agent systems, the same execution may be exposed through composed tools, delegation boundaries, or compacted histories, changing what execution evidence is directly observable to the verifier. It therefore remains unclear whether correct verifier judgments persist across such abstraction boundaries. We introduce Verification Transport, a reliability test that measures judgment stability under abstraction-induced changes in execution observability. We construct matched native, abstracted, and restored views of the same frozen executions while holding the task, realized trajectory, outcome, ground truth, evaluation criterion, and verifier fixed. Across four agent-evaluation benchmarks, seven verifier backbones, four verification protocols, and three abstraction boundaries, we find substantial transport failure, reaching about 31% under macro abstraction and delegation. Verification protocols exhibit distinct failure modes: direct verification more often produces incorrect verdicts, whereas evidence-grounded and deliberative protocols tend to abstain under limited observability. This suggests that lower transport failure rates do not necessarily imply greater reliability. Restoring the corresponding execution provenance recovers 85.5% of transport failures, while recovery is lower when provenance is lossy or unavailable. These results show that native-view accuracy alone is insufficient to characterize verifier reliability when execution evidence crosses abstraction boundaries.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LLM验证器、智能体评估、arXiv、Agent,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Disha Sheshanarayana, Sai Linisha Reddy Goli),经 AITOP 多维评分管线结构化提炼。