推理表示对人类评估LLM输出的影响研究
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
这篇论文研究了六种推理格式如何影响人类对LLM输出的评估，发现偏好与实际效果存在差异。
该研究通过六种推理格式的对照实验，探究不同表示方式对人类评估大语言模型输出的影响。研究在任务复杂度各异的场景下收集了结构理解、错误检测和信任校准的精细判断。
结果显示，参与者偏好规划和分解式表示，但简单的思维链轨迹更有利于验证、信任和可解释性。
Do Reasoning Representations Help Humans Evaluate LLM Outputs?
Reasoning representations are increasingly used as explanations for large language model outputs. Yet they are typically evaluated with model-centric criteria, such as answer accuracy and faithfulness, leaving it unclear whether they help people evaluate model responses. In this work, we study reasoning representations as human-facing interfaces rather than proxies for model reasoning ability. We conduct a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order. The study collects fine-grained judgments of structural understanding, error detection and localization, and trust calibration. Our study shows a mismatch between perceived preference and support for human evaluation. Participants prefer planning- and decomposition-based representations, but simpler chain-of-thought traces better support verification, trust, and interpretability. Preferred representations also introduce calibration risks, with more false alarms on correct traces and high trust despite low willingness to verify.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：LLM、推理表示、思维链、可解释性，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Jaewoo Lim, Sungbok Shin, Sanghyun Hong），经 AITOP 多维评分管线结构化提炼。