LlamaIndex 发布 ExtractBenchPro 文档提取基准
LlamaIndex 搞了个专挑复杂 PDF 毛病的提取基准，最强的 Codex 才 58% 分，做文档处理的朋友可以拿去测测自家方案。
LlamaIndex 发布 ExtractBenchPro，针对现实世界中长尾复杂文档（如试算平衡表、测井曲线、场地图审核）的提取任务。现有提取基准已被前沿模型打满，LlamaIndex 自家最便宜的模型在旧基准上已达 93%+。新基准考察文档内约定、内部术语交叉引用、跨来源聚合推理、图表读数等能力。目前表现最好的 Codex with Astra 在 Value F1 上约 58%，但成本接近每页 1 美元；LlamaParse 的 agentic/agentic plus 模式处于价格可行区间，得分为 20-40%。基准已在网站、GitHub 和 HuggingFace 上开放。
There’s a crazy number of complicated PDFs out there. Each PDF is a self-contained environment for knowledge work, and it can take humans hours to read, analyze, and synthesize a document to solve their task. @llama_index is a research lab dedicated to pushing the frontier of document work. ExtractBench Pro represents our latest effort to extract from the hardest documents at the frontier of accuracy and cost. Even as frontier intelligence starts to do anything with enough compute, there are massive opportunities to solve any document work extremely quickly and quickly. If you haven’t already, check out ExtractBench Pro: llamaindex.ai/blog/introduci… If you’re looking to migrate off of legacy solutions, come talk to us! llamaindex.ai/contact Your browser does not support the video tag. 🔗 View on Twitter Jerry Liu @jerryjliu0 Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction. We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more. Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work. We spent a long time curating the most complex elements that we saw from real world extraction use cases: ✅ Document conventions: containing legends or rules where the rest of the document should be interpreted ✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc ✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information ✅ Drawn relations: complex diagrams like procedures and circuits ✅ Charts inside documents: read precise points within a chart ✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score. (Besides this we also measure value f1 and grounded f1) It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use. ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out! Blog: llamaindex.ai/blog/introduci… Website: extractbenchpro.ai Hugging Face: huggingface.co/datasets/llama… Github (is ExtractBench): github.com/run-llama/Extr… Your browser does not support the video tag. 🔗 View on Twitter 🔗 View Quoted Tweet 💬 1 🔄 0 ❤️ 3 👀 358 📊 1 ⚡
