研究:微调后的开源 LLM 在日语临床 NER 上超越 BERT-CRF
Fine-Grained Japanese Clinical Named Entity Recognition: A Controlled Comparison of BERT and Open-Weight Large Language Models
做医疗文本抽取的团队可以看看这篇:同一批开源 LLM 在 iCorpus 上零样本只有 0.337 的 F1,QLoRA 微调加宽 token 上限后直接干到 0.914,反超 BERT-CRF 的 0.841。结论很实用——评估 LLM 抽取任务前先检查输出 token 限制,JSON Schema 约束帮不上忙。
arXiv 上的一项研究在 iCorpus 数据集上对比了 BERT-CRF 基线与九个开源 LLM,该数据集定义了 113 个实体类别,评估使用 36 个测试文档中的 433 个句子,训练集为 1,672 个句子。零样本提示下 LLM 表现很差,最高 relaxed F1 仅 0.337,而 BERT 基线达到 0.836-0.841。但经过 QLoRA 三种子微调后,在 2,600 或 4,096 输出 token 上限下,八个总参数量不低于 13B 的 LLM 全面超过 BERT,relaxed F1 达 0.882-0.914,exact-surface F1 达 0.840-0.880。
机制层面的关键发现是输出 token 上限对结果影响显著:增大上限后 relaxed F1 平均提升 0.052-0.085,召回率更高且被截断的输出更少。而 JSON Schema 结构约束在七个非推理模型上仅带来 -0.014 到 +0.008 的变化,说明强制输出格式无法弥补零样本能力不足。此外,医疗通用模型组合并未表现出一致的专业化优势。
这项工作对医学信息抽取工程实践有直接参考价值:评估生成式模型做 NER 时,输出容量是必须控制的变量,否则容易低估微调 LLM 的真实水平。对于处理日语等低资源临床文本的团队,QLoRA 微调配合充足的 token 预算是比纯提示更可靠的路线。
Fine-Grained Japanese Clinical Named Entity Recognition: A Controlled Comparison of BERT and Open-Weight Large Language Models
Background: Generative large language models (LLMs) offer an alternative to encoder-based clinical named entity recognition (NER), but their performance on fine-grained Japanese extraction tasks and sensitivity to generation settings remain uncertain. Methods: We compared two fine-tuned BERT-CRF baselines with nine open-weight LLMs on iCorpus, which defines 113 entity categories. The evaluation used 433 sentences from 36 test documents; 1,672 sentences were used for training. LLMs were evaluated using zero-shot prompting with and without JSON Schema constraints and three-seed QLoRA fine-tuning. Fine-tuned models were assessed under different output-token limits, including 900 and larger model-specific limits of 2,600 or 4,096 tokens. Performance was measured using sentence-level label-surface matching with relaxed and exact-surface F1. Results: The highest zero-shot relaxed F1 was 0.337, compared with 0.836-0.841 for the BERT baselines. At the larger output limits, all eight fine-tuned LLMs with at least 13B total parameters exceeded both BERT baselines (relaxed F1, 0.882-0.914; exact-surface F1, 0.840-0.880). Across three-seed means, increasing the output limit improved relaxed F1 by 0.052-0.085, alongside higher recall and fewer length-limited outputs. JSON Schema constraints changed relaxed F1 by -0.014 to +0.008 among seven models outside the two reasoning-oriented configurations, while medical-general model pairs showed no consistent advantage for medical specialization. Conclusions: On this historical Japanese case-report benchmark, task-specific fine-tuning combined with adequate generation capacity was associated with stronger LLM extraction performance. Output-token limits materially affected comparisons with BERT, whereas enforcing output structure alone did not overcome low zero-shot accuracy.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:NER、QLoRA、BERT、iCorpus,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Feng Han, Ayako Yagahara, Kenji Hirata, Atsushi Yoshida, Kohsuke Kudo),经 AITOP 多维评分管线结构化提炼。