实验研究：混合检索与开源 LLM 结合做科学论断验证
An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification
一条 arXiv 实验论文：用 BM25 加语义检索喂证据，Qwen3 8B 做 SciFact 论断验证拿了 0.79 的 Macro-F1，而且发现 NLI 加 LLM 融合反而掉分，做 RAG 事实核查的可以看看。
该研究构建了一个包含 2,035 条论断的基准，数据来自 SciFact 和 CliniFact（858 条 Support、152 条 Contradict、1,025 条 Not Enough Evidence）。检索环节用语义检索加 BM25 的混合方法为每条论断取最相关的五句话，效果略优于单一方法。对比 DeBERTa-v3 NLI 基线与 Llama 3.1 8B、Qwen3 8B、Mistral 7B 三个模型，Qwen3 的 Macro-F1 最高，达 0.7937，NLI 基线仅 0.3157。与假设相反，两者一致才下结论的保守融合规则反而降低表现，Qwen3 的 Macro-F1 从 0.7937 降至 0.7392，错误分析显示分歧时常把 LLM 本来正确的判断改错。
An Experimental Study of Hybrid Retrieval, NLI and Open-source LLMs for Scientific Claim Verification
Claim verification is the task of deciding whether a source text supports a claim, contradicts it, or lacks sufficient evidence. Natural Language Inference (NLI) models can judge whether a single sentence supports or contradicts a claim, but they struggle when the evidence is spread across several sentences. Conversely, Large Language Models (LLMs) can reason across sentences, but may draw conclusions the text does not support. In this experimental study, we test whether a conservative fusion rule, labeling a claim Support or Contradict only when both models agree and defaulting to Not Enough Evidence otherwise, combines their strengths. We construct a benchmark of 2,035 claims from SciFact and CliniFact (858 Support, 152 Contradict, 1,025 Not Enough Evidence). We retrieve the top five most relevant sentences per claim using a hybrid of semantic and lexical (BM25) retrieval, which slightly outperforms either method alone. These retrieved sentences are then provided to every verification model. We then compare a DeBERTa-v3-base-mnli-fever-anli NLI baseline against Llama 3.1 8B, Qwen3 8B, and Mistral 7B. Qwen3 achieves the highest Macro-F1 (0.7937), followed by Llama 3.1 (0.7204) and Mistral (0.6388), while the NLI baseline reaches only 0.3157. Contrary to our hypothesis, fusion degrades performance, dropping Macro-F1 to 0.7392, 0.6730, and 0.5991, respectively. To understand this decline, we analyze agreements, disagreements, and common error patterns between the NLI and LLM verifiers. Our error analysis shows that NLI and LLM disagreements often result in correct LLM predictions being overturned.
- DAIR.AI11-02 15:15原文
