LlamaIndex 发布 ExtractBenchPro：面向真实长尾文档抽取的基准
LlamaIndex 做了个专挑复杂文档难题的抽取基准，Codex 才拿 58% 还要每页 1 美元，搞 RAG 和文档处理的可以拿来自测。
LlamaIndex 团队推出 ExtractBenchPro，聚焦试算平衡表、测井曲线、现场图审核等复杂真实文档抽取场景。现有基准已被前沿模型刷到饱和，其最便宜模式在旧基准 ExtractBench 上得分已超过 93%。新基准考核文档图例约定、内部术语交叉引用、多来源聚合推理、图表读点等难点，核心指标为整份文档完美率。当前最强的 Codex with Astra 在 Value F1 上约 58%，但成本接近每页 1 美元；LlamaParse agentic/agentic plus 模式为 20-40%，价格更适合生产环境。基准已在官网、GitHub 和 HuggingFace 开放。
Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introduci… Website: extractbenchpro.ai Hugging Face: huggingface.co/datasets/llama… Github (is ExtractBench): github.com/run-llama/Extr… Your browser does not support the video tag. 🔗 View on Twitter 💬 11 🔄 8 ❤️ 26 👀 1156 📊 18 ⚡
