论文

研究发现扰动不一致性与语义熵可作为题目难度的代理指标

When Does Confidence Reveal Problem Difficulty?

精选理由

搞评测和推理优化的可以看看这篇。它用数据证明语义熵和解题长度能反映题目难度,而且模型答对了也成立,说明难度信息独立于正确率信号存在。

arXiv 上的一项研究(编号 2611.19173)系统考察了置信度相关信号与题目难度之间的关系。结果显示,扰动不一致性、语义熵和解题长度这三类信号与总体难度的关联,明显强于 token 级不确定性。

正文

更有意思的是,即便只分析模型自己答对的那些题目,这些信号与难度的关联依然存在,说明它们捕捉到的难度信息超出了预测模型自身正确率的范畴。论文还测试了用这些信号指导推理时决策的应用效果,同时指出了其局限。

作者的结论是:模型特定信号可以作为总体题目难度的不完美但有信息量的代理指标。对做评测集构建、数据筛选或推理时资源分配的开发者来说,这是一个值得借鉴的实证结论。

原文 · arXiv cs.AI

When Does Confidence Reveal Problem Difficulty?

While confidence signals are commonly evaluated for their ability to predict a model's own correctness, their relationship with problem difficulty remains less understood. In this work, we investigate a wide range of confidence-related signals and whether they can serve as indicators of problem difficulty across various task domains. We find that perturbation inconsistency, semantic entropy, and solution length exhibit strong associations with population difficulty than token-level uncertainty. Notably, these relationships persist even when restricting the analysis to problems correctly answered by the focal model, suggesting that such signals capture information about difficulty beyond their ability to predict the model's own correctness. We further explore applications of these signals, illustrating the potential and limitations of using confidence-related signals to guide inference-time decisions. Our findings highlight the utility of model-specific signals as imperfect but informative proxies for population-level problem difficulty.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:语义熵、置信度校准、推理时计算、评测基准,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Xiyu Bao, Langyuan Cui, Hwee Tou Ng, Chun Kai Ling),经 AITOP 多维评分管线结构化提炼。