NumTrace 基准：审计金融分析师智能体的数字来源
NumTrace: Auditing the Number Provenance of Financial Analyst Agents
论文测了三个主流模型在 SEC 文件上算数字的表现，发现数字本身往往算对，但挂错了公司或科目，这个问题之前没人单独量化过。
arXiv 论文 NumTrace 针对金融分析师智能体生成的数字论断，审计其来源与财务语义。该基准包含 540 道题目，覆盖 2,163 份 SEC EDGAR 文件，首次覆盖 Form 13F 持股和 Item 402(i) 高管薪酬数据。基准对每个论断给出 value、binding、support 三项独立判定，用来识别“数字对但归属错”（RNWA）错误。审计 Claude Opus 5.5、GPT-6 Astra 和 DeepSeek-V4.1-Flash 三个智能体后发现，RNWA 错误占各自失败论断的 67.5%-81.8%，在 13F 联合申报组问题上约 91% 的论断通过数值检查，但通过 binding 检查的不足一半。
NumTrace: Auditing the Number Provenance of Financial Analyst Agents
A numerical claim generated by a financial analyst agent can match a number in a company filing but refer to the wrong entity, concept, period, unit or scope. A check based on reference answers can mark such an answer as wrong but does not explain why it fails. In this paper, we present NumTrace, a new benchmark that audits the provenance of each numerical claim made by financial analyst agents: its source and financial semantics. NumTrace currently has 540 questions over 2,163 SEC EDGAR documents. To the best of our knowledge, it is the first numerical benchmark about who holds a company's shares (Form 13F) and what its executives are owed (Item 402(i)). On a frozen snapshot of these documents, we build the reference answers according to accounting rules. NumTrace then gives each claim three separate verdicts: value, binding and support. By retaining separate value and binding verdicts, NumTrace detects right number, wrong account (RNWA) errors: a claim has an RNWA error when it passes the value check but fails the binding check. The RNWA rate quantifies the gap between value-only and joint value-and-binding pass rates. Finally, we audit three agents, Claude Opus 5.5, GPT-6 Astra and DeepSeek-V4.1-Flash. RNWA errors make up 67.5-81.8% of each agent's failing claims. In particular, on 13F co-filing group questions, about 91% of claims pass the value check, but fewer than half pass the binding check.