论文

SG-LegalClarify:新加坡法律推理与歧义识别基准发布

SG-LegalClarify: A Large-Scale Benchmark for Legal Reasoning and Ambiguity Recognition in Singapore Law

精选理由

新加坡法律圈的 LLM 测评来了,1,100 道律师出的题,还专门考模型会不会主动追问隐藏事实。最强模型闭卷也只有 69.8% 达标,想看模型在普通法+本地成文法混合场景下的真实水平,这套题比泛用基准更贴近实际。

新加坡国立法律体系结合了继承自英国的普通法与本地成文法,LLM 需要区分本地法理与通用普通法直觉。研究者推出 SG-LegalClarify,包含 1,100 道律师撰写的问题,覆盖 10 个法律领域,其中 760 道为信息不足问题、340 道为信息完整问题,配合固定隐藏事实设定共构成 1,860 个评测实例。

正文

评分采用律师撰写的 IRAC(Issue、Rule、Application、Conclusion)量表,评测在闭卷和联网搜索两种条件下进行。流程采用多轮对话设计:模型需自行判断是直接作答还是先澄清,用户模拟器只在模型主动提问时才揭示隐藏事实。

共评测 19 个 LLM,结果显示基准远未饱和:最佳模型闭卷仅满足 69.8% 的必达量表项(平均 53.8%),联网搜索后为 77.8%(平均 63.0%)。模型普遍难以识别何时需要向用户澄清,作者将公开基准公共部分、模型输出与评测管线。

原文 · arXiv cs.AI

SG-LegalClarify: A Large-Scale Benchmark for Legal Reasoning and Ambiguity Recognition in Singapore Law

Singapore offers a distinctive testbed for legal reasoning by large language models (LLMs). Its legal system combines inherited English common law with jurisdiction-specific statutes, requiring models to distinguish local doctrine from generic common-law intuition. This setting also highlights the challenge of recognizing underspecification in lay legal queries, because the facts material to a legal outcome depend on the governing doctrine. Yet legal benchmarks rarely cover Singapore law or test this form of ambiguity recognition. To address these gaps, we introduce SG-LegalClarify, a large-scale benchmark for open-ended legal reasoning and ambiguity recognition in Singapore law. SG-LegalClarify contains 1,100 lawyer-authored source questions spanning 10 legal domains, including 340 fully specified and 760 underspecified questions. Each underspecified question has a fixed hidden-fact assignment and a fully specified counterpart, yielding 1,860 evaluation instances. Responses are graded with lawyer-authored Issue, Rule, Application, and Conclusion (IRAC) rubrics. We evaluate 19 LLMs, closed-book and with web search, using a multi-turn pipeline in which models decide whether to answer or clarify, and a user simulator reveals hidden facts only when asked. We find the benchmark far from saturated: the best models satisfy only 69.8% of mandatory rubric items closed-book (average 53.8%) and 77.8% with web search (average 63.0%). Models also struggle to recognize when clarification is necessary. We will release the public split of our benchmark, the model outputs on it, and the evaluation pipeline to support future research.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦东南亚与东盟六国(新加坡、马来西亚、印尼、泰国、越南、菲律宾)数字经济生态。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LLM、SG-LegalClarify、法律AI、基准测试,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Ziwei Gu, Michael S. Lee, Drew Rein, Yash Maurya, Vikneswaran Kumaran, Darshini Ramiah, Chenguang Wang, Udari Madhushani Sehwag),经 AITOP 多维评分管线结构化提炼。

SG-LegalClarify:新加坡法律推理与歧义识别基准发布|AI热报