论文精选

论文揭示企业文档摄取丢失格式线索,导致智能体自信出错

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

精选理由

作者团队测试了 docling 等主流转换器,发现高亮和颜色信息在文档转 token 时全丢了,六个模型都因此答错。做企业智能体的可以看看自己的摄取层。

一篇 arXiv 论文构建了基于 76 份真实政策文档的任务,要求智能体恢复作者用格式标出的需求,并用确定性匹配器对照 OOXML 属性评分。格式盲摄取使 micro F1 从 0.690 跌至 0.150(docling)和 0.075(纯文本),该差距在五个 Claude 模型和 GPT-5.6 上均复现。留一法定位到高亮(+0.302)和颜色(+0.262)两个线索是主要损失源,而加粗、斜体等影响接近零。6,460 份文档中 32.8% 携带刻意的颜色或高亮,颜色则是所有主流转换器都不保留、解析基准也不计分的线索。论文结论是企业智能体的可靠性瓶颈在摄取层而非推理层,且修复成本低、可量化。

原文 · arXiv: OpenAI

What the Author Wrote, What the Agent Reads: Silent Cue Loss in Enterprise Document Ingestion

Enterprise agents act on documents. Before an agent can follow a policy, answer a compliance question or file a ticket, some ingestion layer must turn a Word file or PDF into tokens - and that layer is where an agent deployed in the wild silently loses information. We show that one layer is lost by every mainstream converter - the meaning an author encodes in how text looks - and that the loss makes agents confidently wrong rather than visibly incomplete. We build a task from 76 real policy documents: recover the requirements the author singled out, scored against the file's own OOXML properties by a deterministic matcher with no LLM judge. Formatting-blind ingestion collapses from micro F1 0.690 to 0.150 (docling) and 0.075 (plain text), a gap that replicates across six answering models from two vendors - five Claude models spanning the Opus, Sonnet and Haiku tiers, and OpenAI's GPT-5.6. An agent that cannot see the cue guesses, and docling's false-positive rate on requirements the author did not flag is four times ours. A span-level leave-one-out localises the loss to two cues - highlight (+0.302) and colour (+0.262), both Holm-adjusted p < 1e-4 - while bold, italic, monospace and strikethrough each sit within +/-0.003 of zero. The same asymmetry appears in PDF. Colour is also the cue no mainstream converter preserves and no parsing benchmark scores, and it is common: 32.8% of 6,460 documents carry a deliberate colour or highlight, 10.3% on an obligation. A pre-registered test shows that typing a cue for the agent adds +0.085 beyond merely preserving it (Holm-adjusted p = 0.049). We validate the labels against rendered page images through an independent modality (kappa = 0.72) and bound what wording alone can achieve (AUC 0.664). For agent builders the practical finding is that ingestion, not reasoning, is where this reliability is won or lost, and that the fix is cheap and measurable.