研究指出主流LLM诚实度基准存在表面特征泄露问题
Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage
这是篇关于如何提升LLM诚实度评估准确性的技术论文，作者团队通过具体实验指出了现有基准测试的缺陷，并提供了可复用的解决方案，对做相关研究或评估工作的人很有参考价值。
论文发现，在TruthfulQA等基准测试中，模型仅通过识别正确答案和错误答案之间的表面特征差异（如长度、标点符号等）就能超过随机水平，而非真正理解事实。研究团队开发了Audit-Prune机制来清理这些数据，并发布了修正后的TruthfulQA版本。
Judging by the Cover: Cleaning LLM Truthfulness Benchmarks to Avoid Surface-Level Feature Leakage
Binary-choice truth benchmarks ask models to choose between a correct and an incorrect answer, but if the two answers differ systematically in surface-level features, models can exceed chance without performing the intended reasoning. We show that this failure mode is detectable and can be exploited by downstream classifiers. In TruthfulQA, a simple six-feature logistic classifier achieves substantial accuracy in separating correct from incorrect answers. We further show that similar surface-level artifacts are present in additional benchmarks. To counteract this, we developed a general mechanism to clean them by removing the most leakage-reinforcing pairs. We release a version of TruthfulQA with surface-feature leakage reduced close to chance and provide a mechanism, Audit-Prune, so that the datasets can be cleaned before release.
