论文

dhAIBench:48 个模型在九项语言学标注任务上的基准测试

Benchmarking Large Language Models for Linguistic Annotation

精选理由

做语言学标注、文本规范化的研究者可以直接用这个开源工具跑自己模型,不用自己搭评估流程。它连置信度校准和成本都记录,比通用排行榜更贴近实际标注场景。数字人文团队值得上手试试。

arXiv 发布论文介绍 dhAIBench,一个面向语言学标注的 Python 基准测试与标注工具,配有可浏览和共享结果的网页界面。

正文

该框架记录每次运行的提示词、配置和预测,可比较标注质量、置信度校准、可重复性和估算成本。研究团队通过 293 次运行、48 个模型、九项语言学标注任务(涵盖形态学、句法、语义学、语篇-语用、拼写规范化和学习者语言纠正)展示了该框架。

结果显示,部分模型与参考标注一致性较高,包括非标准化的历史语言变体和需要语境解释的任务。结论支持按具体语言学任务评估模型,而非依赖通用排行榜。

基准已在 dhaibench.silent3.

ff.cuni.

cz 公开。

原文 · arXiv cs.AI

Benchmarking Large Language Models for Linguistic Annotation

Assessing large language models for linguistic annotation requires task-specific evaluation under documented conditions. We present dhAIBench, a Python benchmarking and annotation tool with a web interface for exploring and sharing results. It records prompts, configurations and predictions, enabling comparisons of annotation quality, confidence calibration, repeatability and estimated cost. We demonstrate the framework and the associated methodology through 293 runs of 48 models on nine linguistic annotation tasks spanning morphology, syntax, semantics, discourse-pragmatics, spelling normalization and learner-language correction. Models achieve high agreement with reference annotations on several tasks, including those on non-standardized, historical varieties as well as requiring contextual interpretation. The results support evaluating models against specific linguistic tasks, rather than relying on general rankings. To this end, we make the current and future benchmarks publicly available at dhaibench.silent3.ff.cuni.cz.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦欧洲人工智能治理与技术生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:dhAIBench、语言学标注、基准测试、LLM评估,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Ondřej Tichý, Anna Marklová, Barbora Bulantová, Magdalena Titlbachová, Jiří Milička),经 AITOP 多维评分管线结构化提炼。