分解LLM评估器不确定性以定位专家标注
Decomposing LLM-Judge Uncertainty to Target Expert Labels
这篇论文教你如何让LLM评估器更聪明地知道何时需要专家帮助，而不是盲目地寻求所有答案。
研究人员提出了一种小型贝叶斯模型，可将LLM评估器的不确定性分解为两类：偶然性不确定性和认知不确定性。该模型通过已收集标注的回归分析，学习如何信任黑盒评估器的预测。在ChaosNLI数据集上，认知不确定性排名比总不确定性减少了83%的误差。研究团队展示了如何评估评估器的无知程度而非专家的真实分歧。
Decomposing LLM-Judge Uncertainty to Target Expert Labels
An LLM judge evaluates outputs at scale. Experts should label only where it is least sure. Its natural escalation signal conflates two uncertainties: aleatoric, real disagreement in the expert pool, which labels cannot reduce, and epistemic, the judge's ignorance, which labels do reduce. A small Bayesian model separates them: a regression on labels already collected learns how far to trust a black-box judge's prediction. Both components follow as simple formulas, with no sampling or further judge calls. The components isolate on a real LLM judge against exactly known truth, and stated confidence is no guide to its actual error. On real human disagreement (ChaosNLI) the epistemic ranking removes 83% more error than total uncertainty for the same expert labels, though simply escalating the least-labelled items does as well there. We demonstrate we can estimate where a judge is ignorant rather than where experts genuinely disagree, and propose using this to direct expert labelling.