新研究提出决策审计方法:模型执行成功不代表产品决策正确
The Delegation Blind Spot: Auditing Product Decisions from Agent Choices
这篇论文用 4,800 次请求证明模型跑得准不等于选对了产品方向,还给出不用调用模型的提取器,做 Agent 评估的可以看看。
arXiv 论文提出一种针对 Agent 决策的审计方法,将声明的观察通道与产品价值对比映射为兼容区间和见证人群。
冻结实验向两个固定版本的模型快照发出 4,800 次请求,36 个保守主区间全部未获解决。后续 2,400 次调用仅解决每模型九项对比中的三项,而确定性提取器在不调用模型的情况下解决了九项中的七项。
论文还提供带来源标签的决策回执和离线查看器,建议在收集更多遥测数据前先诊断决策为何无法确定。
The Delegation Blind Spot: Auditing Product Decisions from Agent Choices
Successful agent execution need not identify which future product improvement its user would value. We present a decision-specific audit that maps a declared observation channel and product-value contrast to compatible intervals and witness populations. Its foundations are established identification and decision theory; the contribution is an executable measurement workflow and a controlled study of its limits. A frozen experiment makes 4,800 requests to two pinned model snapshots on shared synthetic tasks. All 36 conservative primary intervals remain unresolved despite different execution accuracy. An exploratory 2,400-call follow-up records supplied preferences and resolves three of nine comparisons per model. A deterministic extractor resolves seven of nine without model calls or calibration observations, exposing unnecessary uncertainty introduced by model-generated reports. A further 14,400 controlled multinomial simulations distinguish structural ambiguity from weak identification and finite calibration precision. We propose a source-labeled decision receipt and provide an offline viewer for inspecting the audit. These results motivate preserving decision-relevant structured input and diagnosing why a decision is unresolved before collecting more telemetry. The study contains no human participants or real customer outcomes. Full proofs, raw model provenance, controlled experiments, and reproducible analyses accompany the report.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:智能体、决策审计、论文、评估方法,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Shivam Gupta),经 AITOP 多维评分管线结构化提炼。