用 LLM 从刑事判决书抽取结构化数据：DUI 案例抽取准确率从 52% 提升到 70%
Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research
做法律实证研究的朋友可以看看：用 LLM 把刑事判决书变成结构化数据，指令改一改准确率就从 52% 涨到 70%，方法写得很细。
一篇 arXiv 论文提出基于 LLM 的信息抽取流程，包含属性探索、文本片段抽取和数值归一化三个环节，法律专家可通过修订标注指令来调整抽取规则。在酒驾（DUI）这一交通犯罪场景上，修订指令后正确抽取值的比例从 52% 提升到 70%。实验还发现同一模型多次运行彼此一致性高于与人工金标准的一致性，说明输出稳定不等于正确。在另一项禁止驾驶（DDB）场景中，由于指令规定了固定类别和输出格式，大多数属性的一致率较高。
Unlocking Criminal Court Decisions: Flexible Information Extraction for Empirical Legal Research
Criminal verdicts contain descriptions of criminal behavior that can support quantitative legal research, but structuring them requires attributes and extraction rules that cannot always be fully specified in advance. We present an LLM-based pipeline that builds a structured dataset through attribute exploration, text span extraction, and value normalization, guided by annotation instructions that legal experts can refine. We evaluate it on two traffic offenses: driving under the influence (DUI) and driving despite a ban (DDB). For DUI, revising the instructions was associated with higher extraction accuracy (from 52% to 70% of correctly extracted values) and higher agreement between human annotators. Repeated runs of the same model agreed with each other more than with the gold annotations, showing that consistent output is not necessarily correct, while disagreement between annotators and between models helped reveal unclear instructions. For DDB, where the instructions defined fixed categories and output formats, agreement was high for most attributes.
