LlamaIndex 发布 ParseBench-2:覆盖 6 大维度的文档解析基准
做 RAG 或文档处理的人应该看看这个。LlamaIndex 的 ParseBench-2 把 6 个维度的解析错误拆开量化,还给了 Metric Lab 让你按自己的业务调评分权重,比传统确定性指标和 LLM 评委都更贴近真实判断。
LlamaIndex 创始人 Jerry Liu 发布博客,阐述构建文档解析基准的 6 项设计标准:代表性数据、粒度、可量化、可重复、可控、与人类判断对齐,并回顾了从静态确定性指标到 LLM-as-a-judge 两类历史方案的缺陷。
团队上周推出 ParseBench-2,包含 4,317 页测试数据,覆盖表格、图表、版式、格式、文本和表单 6 个维度,其中 1,025 张图表配有完整参考表。评分方式改为先确定各类错误应占的权重,再通过 330 种扰动类型测试和调校评分,使其与人类判断一致。基准还提供 Metric Lab,用户可针对自身工作负载设置评分优先级。
博客坦承 ParseBench-1 的失误:曾在一份 McDonald's 财务报表上,给把金额数字断行的解析器打出高于完全解析正确的解析器的分数。该基准此前已被 NVIDIA、Cohere 和 Perplexity 引用,团队表示会快速迭代标注和指标质量。
Building a great benchmark is challenging, building a perfect benchmark is near impossible. This especially holds true in the world of document parsing. I wrote a blog post that details the core design criteria needed for a document parsing benchmark, and the historical attempts to get there - from those based on static deterministic metrics to LLM as a judge approaches. Building a “great” benchmark requires at least the following: 1. Representative data 2. Granularity 3. Quantifiable 4. Repeatable 5. Controllable 6. Aligned with human judgment Existing benchmarks based on deterministic metrics are oftentimes granular/repeatable, but not controllable or aligned with human judgment. Those based on LLM as a judge are not granular, quantifiable, repeatable, and can obscure how human judgment is encoded. Last week we launched ParseBench-2 in an attempt to tackle all of these dimensions. It contains a taxonomy of metrics across 6 dimensions (tables, charts, layout, formatting, text, and forms) that precisely measure document parsing failures and quantifies their impact. It’s certainly not perfect, but does make an honest attempt towards advancing every design criteria. It’s also designed to be customizable by the user (see our metric lab!) We’re rapidly shipping new iterations of this benchmark to improve annotations, metric quality, and customizability. If you do check it out, would love to hear your thoughts. Blog: llamaindex.ai/blog/building-… ParseBench-2: parsebench.ai LlamaIndex 🦙 @llama_index ParseBench has been cited by NVIDIA, Cohere and Perplexity. It still isn’t a perfect benchmark. Part of getting closer meant owning where ParseBench-1 got things wrong. On one McDonald’s financial statement, it scored a parser that broke a dollar amount above one that got every number right. So for ParseBench-2, we changed how we build the metric. We decide how much each kind of error should matter first, then test and tune the scoring until it agrees. → 330 perturbation types to check that scores move by the right amount → 4,317 pages across six dimensions, now including forms → Full reference tables for all 1,025 charts → A Metric Lab to set scoring priorities for your own workload There’s still more to do, and we wanted to be open about that too. Read more about how we’re building toward a perfect document parsing benchmark in the reply below. 🔗 View Quoted Tweet📊 9 ⚡
本条情报归属 AITOP「产品」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LlamaIndex、ParseBench、文档解析、RAG,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「Jerry Liu」(@jerryjliu0),经 AITOP 多维评分管线结构化提炼。截至目前,全网已有 11 家独立信源对该事件进行交叉印证与跟进报道(含 TestingCatalog、Replit、kimmonismus 等),属于近期高关注度行业信号。