Jerry Liu 分享文档解析基准设计:ParseBench-2 与 DPP 的颗粒度实践
LlamaIndex 的 Jerry Liu 把文档解析评测拆成 6 维 330 场景,还配了 516 条 LLM 判分规则。正在做 RAG 或解析管道的人可以直接抄这套加权评估思路,比只看一个总分靠谱多了。
LlamaIndex 创始人 Jerry Liu 发文讨论文档解析基准的核心难题:如何让评估同时具备颗粒度和人类偏好对齐。他给出两条互补路径:确定性指标方面,ParseBench-2 将解析拆成 6 个维度、330 个测试场景,按人工定义的严重度加权——数字错误扣 10 分以上,多余 HTML 标记扣 0 分。LLM as a judge 方面,其 Document Parsing Protocol(DPP)定义了 140 条规则、516 项书面检查,每项由专门的 judge prompt 执行。
颗粒度带来的价值在于可以精确编码人类偏好,并按需对图表错误、表格错误、表单错误、版式错误等维度加权。Jerry Liu 也坦承 ParseBench-2 和 DPP 都不完美,表示后续会持续迭代。对正在选型文档解析方案或自建评估体系的开发者来说,这套分维度加权的思路可直接借鉴。
One of the hardest parts about building any benchmark that measures high-dimensional output is making it granular, while also being aligned with human preferences. This especially holds true for document parsing. Granularity is helpful because it lets you more precisely encode human preferences. Within document parsing, there are many categories of errors across various document dimensions: chart errors, table errors, form errors, layout errors, and more. Granularity also enables customization - it lets you better choose the dimensions you care about the most and weigh them more heavily. You can have a granular benchmark using either deterministic metrics or LLM as a judge; they’re not mutually exclusive: 1️⃣ Deterministic: ParseBench-2 splits parsing into 6 dimensions and 330 test scenarios. Each dimension is weighed by human-defined severity (a wrong digit should cost 10+ pts, extra HTML markup costs 0). 2️⃣ LLM as a judge: You can also define granular metrics where each one is measured by LLM as a judge. Our Document Parsing Protocol (DPP) has a comprehensive taxonomy of 140 rules / 516 written checks, each codified by a judge prompt! Building a benchmark that is granular but aligned is challenging. There’s no free lunch; neither ParseBench-2 nor DPP are ‘perfect’, but they do make an honest effort in advancing every criteria. We’re shipping a lot more iterations here. Keep your eyes peeled! ParseBench-2: parsebench.ai Document Parsing Protocol: parseprotocol.com
本条情报归属 AITOP「技巧」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ParseBench-2、LlamaIndex、文档解析、RAG,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「Jerry Liu」(@jerryjliu0),经 AITOP 多维评分管线结构化提炼。