MuSeToM 基准测试评估 LLM 基于多感官线索的信念修正能力
MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses
想测模型的心智理论能力，可以看看这个 MuSeToM，2400 多个案例，覆盖六种感官线索。它发现模型能察觉信念要变，却经常推错变化后的内容，还容易被表面线索带偏。做 agent 和对话系统评测的值得一试。
arXiv 上发布的 MuSeToM 是一个评估心智理论（ToM）中认知性信念修正能力的文本基准，包含 2,415 个实例，改编自 BigToM 和 OpenToM，文本线索覆盖六种感官类别。该基准测试模型能否判断主角何时应该修正信念，并推断出修正后的信念状态。
此前对 LLM 的 ToM 评估主要停留在静态信念归因和错误信念推理，MuSeToM 补上了“感知证据是否足以支持信念修正”这一环节。团队用直接提问和思维链（chain-of-thought）两种提示方式系统评测了多个 SOTA 模型。
结果显示模型存在明显失衡：模型往往能察觉信念可能发生变化，却难以推断出更新后的具体信念内容，且容易被感知上显著但认知上不充分的线索诱导而过度更新信念。该基准经过人工验证，可用于诊断模型的社交推理短板。
MuSeToM: Multi-Sensory Evaluation of Belief Revision with Textual Cues Across Six Senses
Theory of Mind (ToM), the ability to infer mental states in others, is pivotal for human social cognition. Existing evaluations of ToM in large language models (LLMs) mainly focus on static belief attribution and false-belief reasoning, leaving a critical aspect of social reasoning underexplored: can LLMs decide whether perceptual evidence is sufficient to justify another person's belief revision? To address this gap, we present MuSeToM, a human-validated, text-based benchmark for evaluating epistemic belief revision in ToM reasoning. MuSeToM contains 2,415 instances adapted from BigToM and OpenToM, with textual cues spanning six sensory categories, and tests whether models can determine when a protagonist should revise a belief and infer the resulting updated belief state. Using MuSeToM, we systematically evaluate state-of-the-art LLMs under direct and chain-of-thought prompting, revealing a pronounced dissonance: while models often detect that a belief may change, they struggle to identify the updated belief and frequently over-update on perceptually salient but epistemically insufficient cues.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：MuSeToM、Theory of Mind、LLM评测、思维链，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Chunkit Chan, Kawai Chung, Sze Heng Douglas Kwok, Yuxuan Liu, Yauwai Yim, Tianshi Zheng, Haochen Shi, Qing Zong, Wong Kai Chung, Pak Hin Lau, Calvin Yang, Yifan Gao, Xiaoman Pan, Ginny Wong, Simon See, Yangqiu Song），经 AITOP 多维评分管线结构化提炼。