产品精选

Jerry Liu 详解 ParseBench 2:让用户自定义评测指标与惩罚策略

精选理由

LlamaIndex 的 Jerry Liu 回应了解析基准 ParseBench 2 的争议,还开放了自定义惩罚策略的试用入口。做 RAG 或文档处理的开发者可以亲自调整评测规则,不用被动接受厂商偏好的排名。对比各家自评方案,这套机制透明多了,值得上手试试。

LlamaIndex 创始人 Jerry Liu 在 X 上回应 ParseBench 2 引发的争议,核心观点是:好的评测基准应允许用户根据自己关心的维度调整权重,而这在确定性指标体系下通常难以实现。团队因此把 ParseBench 2 的指标设计当成一个目标导向的优化问题:用户先为不同场景指定严重性策略(severity policy),系统再据此优化评分指标。据团队介绍,这是首个支持用户自定义惩罚策略的解析(parsing)基准,可在 parsebench.ai 上试用。

正文

论文合作者 Pierre-Loic Doulcet 同时回应了竞品的批评。他举例称,有厂商抱怨数据集评估下划线保留、漏掉下划线会被扣分“不合理”,但明确测下划线保留时扣分正是设计意图。他还指出,某些呼吁终结基准测试的厂商几天后自己发布了两个闭源基准,并推广自家“LLM-as-a-judge”评测方案,而这种方式同样可以在评估提示词里嵌入厂商偏好。

Doulcet 承认包括 ParseBench 2 在内的所有基准都有缺陷与偏见,因为创建者只会测自己在意的东西,但他认为这应通过技术讨论解决,而非人身攻击或恶意解读。对开发者而言,自定义评测策略的价值在于不必盲从基准创建者的偏好,能按业务场景衡量文档解析模型的真实质量。

原文 · Jerry Liu

I do think any good benchmark should be customizable by the user to weigh towards what they care about the most. This is typically hard to do for benchmarks measured with deterministic metrics. That's why in ParseBench-2, we treated metric design as a goal-oriented optimization problem! 1. Let the user specify a severity policy for different scenarios 2. Optimize metrics to fit that objective. More details in the paper: arxiv.org/abs/2611.17400 Pierre-Loic Doulcet @hexapode I agree with @jerryjliu0 's sentiment. [Rant on] Some of the reactions to ParseBench 2 from competitors have been pretty disappointing. We've seen personal attacks, bad-faith interpretations of the benchmark, and criticisms that suggest some people haven't actually taken the time to understand what we built. One example that stood out: "Look, there's underlined text in a table cell, and if our parser misses the underline, the benchmark penalizes us. This is scandalous!" Well... yes? If the dataset evaluates underline preservation, missing underlines gets penalized. That's kind of the point. Another interesting one: calling for an end to benchmarks, only to release two new closed benchmarks a few days later, while promoting a proprietary "LLM-as-a-judge" evaluation that conveniently ranks your own parser first. (Surprise: vendor preferences can be embedded in evaluation prompts too!) There have been dozens of takes like these, often from people who clearly haven't looked into how ParseBench 2 actually works. At times, it's felt more like a coordinated effort to discredit the benchmark than a genuine technical discussion. [/Rant off] Here's the thing: every vendor will continue publishing benchmarks. That's how this industry works. And every benchmark, including ours, will have flaws and biases. We naturally measure the things we care about. That's fine. It's worth discussing. It doesn't justify personal attacks. I'll continue taking valid feedback seriously and improving and expanding ParseBench. I genuinely want us to get closer to a fairer, more useful benchmark. Because ultimately, I care about the quality of what we ship, not winning a marketing argument. And with ParseBench 2, we actually built some pretty cool things that I think deserve more attention: 1. User-defined evaluation policies To our knowledge, ParseBench 2 is the first parsing benchmark that lets users define their own penalty policies, instead of blindly accepting the benchmark creator's preferences (including ours). Try it: parsebench.ai/lab/index.html We're working on making this easier to use, customize, and export. 2. Metrics designed to reflect what actually matters We spent a lot of effort making sure different models get a fair evaluation, without unnecessarily penalizing formatting differences or model-specific quirks. We also developed a technique to better align scoring metrics with the user's chosen evaluation policy. This means we can distinguish between errors based on their actual severity, rather than relying purely on traditional edit-distance metrics that may treat very different mistakes similarly. You can explore the metrics, different severity policies, and our analysis of dozens of other benchmarks here: parsebench.ai/compare/index.… I'm genuinely impressed by the technical work coming out of some of our competitors. Some teams are doing incredible things, and I have a lot of respect for that. I'd love to see more technical discussions, more reproducible evaluations, and less high-school-level drama. Anyway, back to building. 🔗 View Quoted Tweet📊 5 ⚡

产业坐标与信源核验

本条情报归属 AITOP「产品」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ParseBench 2、LlamaIndex、文档解析、评测基准,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「Jerry Liu」(@jerryjliu0),经 AITOP 多维评分管线结构化提炼。