MM-SystemOneBias基准:评测13个多模态决策模型的六大偏差
MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models
如果你在做审核或质量判断类多模态系统,这个基准直接回答了"决策封装值不值":7 组配对全部显示决策模型更抗干扰,但 6 组也从正确外部信息里学得更少,是稳和灵之间的取舍。想看自家 backbone 是否值得套决策层,可以照它的六种干预设计自测。
MM-SystemOneBias 是一个新的多模态决策模型偏差评测基准,覆盖 13 个模型、9 个公开数据集的 5,323 条原始实例和 11 个任务场景,通过六种成对干预进行测试:选项顺序、选项标签、外部判断、图文描述、无关信息和评分量表。该基准首次提供了统一协议,可在实际任务上比较决策模型与其骨干模型的表现。
核心发现是:在 7 组决策模型—骨干模型配对中,决策模型在选项标签、无关信息和评分量表变化下全部 7 组都更稳定,6 组还表现出更低的选项顺序翻转率;面对错误外部判断,7 组的准确率损失都更小,6 组在冲突描述下损失更小。但代价是:6 组从正确外部判断中获得的准确率提升更少,6 组在一致描述下准确率变化更不利。
此外,在语义决策不变的样本中,答案概率的平均绝对变化在 4.56 到 11.55 个百分点之间,说明仅靠准确率无法捕捉输入扰动对模型置信度的影响。这一结果提示决策模型以牺牲对外部信息的利用能力换取抗扰稳定性,对部署审核、质量判断类系统的团队有直接参考价值。
MM-SystemOneBias: Evaluating Bias in Multimodal Jev-Style System One Decision Models
When multimodal decision models determine actions, moderation outcomes, or quality judgments, their reliability depends on preserving semantic decisions across presentation changes, ignoring irrelevant additions, and distinguishing information consistent with task evidence from information that conflicts with it. Existing evaluations study decision-interface biases and multimodal model behavior in separate settings, leaving no common protocol to compare multimodal decision models with their backbones on practical tasks. We introduce \textbf{MM-SystemOneBias}, which evaluates 13 models on 5,323 original instances from nine public datasets and 11 task scenarios using six paired interventions: option order, option label, external judgment, image--text description, irrelevant information, and rating scale. Across seven matched decision-model--backbone pairs, decision models are more stable under option-label, irrelevant-information, and rating-scale changes in all seven pairs; six pairs also show lower option-order flip rates. They incur smaller accuracy losses under incorrect external judgments in all seven pairs and conflicting descriptions in six. In six pairs, they gain less accuracy from correct external judgments and show less favorable accuracy changes under consistent descriptions. Among pairs with unchanged semantic decisions, mean absolute changes in answer probabilities range from 4.56 to 11.55 percentage points, revealing input effects that accuracy alone misses.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:多模态、模型评测、MM-SystemOneBias,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Jinzhe Li, Qi Wang, Dandan Guo, Li Liu, Yuan Wu, Yi Chang),经 AITOP 多维评分管线结构化提炼。