Vik Paruchuri 演讲：为什么选 OCR 模型不能只看公开基准分数
Vik Paruchuri 拆了 ParseBench、olmOCR-bench 这些 OCR 基准的判分 bug，正确答案都被判错。选模型前先看这个，学会自己搭评测。
Vik Paruchuri 在演讲中逐一指出主流 OCR 基准的具体缺陷。ParseBench 里一道测试把太阳能装机容量 303 吉瓦错标成 310，73% 的图表测试要求读取图上并未印出的数值。olmOCR-bench 会惩罚正确保留换行的模型，大小写抄对反而判错，一段 700 行的后处理脚本就能涨 1-2 分，超过许多模型间的差距。OmniDocBench 用编辑距离打分，导致 HTML 格式的正确表格比 Markdown 里的错误数字扣分更多，格式权重盖过了答案本身。演讲最后介绍了如何按自己的任务切片评测、用自建验证答案的私有测试来选模型。
Should you rely on public benchmark scores when choosing a model? @VikParuchuri argues "no" in this talk.
To illustrate this, he shows specific problems found in popular OCR benchmarks:
ParseBench:
• A test expects 310 gigawatts of solar-panel capacity when the correct value is 303. Additionally, 73% of its chart tests ask for values that aren't printed on the chart. (4:51)
• A test requires "Parish Priest:" to be bold even though those words aren't bold in the source document. (15:33)
• The scorer fails to recognize heading tags placed on separate lines. (16:15)
• dots.mocr ranks last among six models even though it beats the first-ranked model on content and tables. This is an example of how overall benchmark score can hide important differences in model capabilities. (14:48)
olmOCR-bench:
• A model correctly preserves a line break in a document and gets penalized for it. (10:30)
• A test expects "Republican Party" when a document says "Republican party". Copying the capitalization correctly makes the model fail. (10:49)
• A 700-line postprocessor that rewrites model output adds 1 to 2 points, more than many gaps between models. (12:12)
OmniDocBench:
• This benchmark uses edit distance, which penalizes a correct table written in HTML more heavily than a wrong number in Markdown. So the format matters more than the answer! (6:58)
This is just the tip of the iceberg, the video contains more examples.
Chapters:
0:00 Choosing a model when every vendor claims to be #1 1:54 How OCR benchmark scores improve without better OCR 5:30 Metric design and tests that reject correct answers 11:33 Postprocessing, saturation, and misleading model rankings 12:40 How to slice benchmarks by your task and check for bias 16:53 How to build fair tests with verified answers and consistent scoring 19:05 How to test OCR models on your own documents 24:10 Benchmark marketing, private evals, and how to choose a model
